You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Niels Basjes <Ni...@basjes.nl> on 2018/08/16 12:05:37 UTC
How to handle tests that need docker?
Hi,
I'm working together with Richard Deurwaarder on the PubSub Source and Sink
https://issues.apache.org/jira/browse/FLINK-9311
We currently have most of the code done and we also have several tests that
verify the behavior by actually reading and writing messages to and from
PubSub.
These tests use a docker image provided by Google that includes the pubsub
emulator.
Starting and stopping this thing is really fast.
Starting up, creating topic, running a few tests and cleaning up takes
about 5 seconds.
Although the complete set of tests run on my machine in about 20-25 seconds
it does mean that anyone doing that MUST have docker installed locally.
So my question to the committers: Do we simply leave these tests "always
on" or do you want them to be "off by default" or something else.
I case you do not want the "always on" then please indicate how we should
implement the "on"/"off" switch for these tests.
Thanks.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: How to handle tests that need docker?
Posted by Till Rohrmann <tr...@apache.org>.
I agree with Dominik. We could add this test to
`flink-end-to-end-test/run-pre-commit-tests.sh` or
`flink-end-to-end-test/run-nightly-tests.sh` like we did for the
`flink-end-to-end-test/test-scripts/test_docker_embedded_job.sh` and
`flinke-end-to-end-test/test-scripts/test_yarn_kerberos_docker.sh`.
Cheers,
Till
On Thu, Aug 16, 2018 at 2:10 PM Dominik Wosiński <wo...@gmail.com> wrote:
> Imho,
> Such tests should be off by default as they impose some requirements on
> the people trying to build Flink from source. And one would have to install
> docker only for the tests to pass. But of course they should be enabled for
> CI builds.
>
> Best Regards,
> Dominik.
>
> Wysłane z aplikacji Poczta dla Windows 10
>
> Od: Niels Basjes
> Wysłano: czwartek, 16 sierpnia 2018 14:05
> Do: dev@flink.apache.org
> Temat: How to handle tests that need docker?
>
> Hi,
>
> I'm working together with Richard Deurwaarder on the PubSub Source and Sink
> https://issues.apache.org/jira/browse/FLINK-9311
>
> We currently have most of the code done and we also have several tests that
> verify the behavior by actually reading and writing messages to and from
> PubSub.
> These tests use a docker image provided by Google that includes the pubsub
> emulator.
>
> Starting and stopping this thing is really fast.
> Starting up, creating topic, running a few tests and cleaning up takes
> about 5 seconds.
>
> Although the complete set of tests run on my machine in about 20-25 seconds
> it does mean that anyone doing that MUST have docker installed locally.
>
> So my question to the committers: Do we simply leave these tests "always
> on" or do you want them to be "off by default" or something else.
> I case you do not want the "always on" then please indicate how we should
> implement the "on"/"off" switch for these tests.
>
> Thanks.
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
>
ODP: How to handle tests that need docker?
Posted by Dominik Wosiński <wo...@gmail.com>.
Imho,
Such tests should be off by default as they impose some requirements on the people trying to build Flink from source. And one would have to install docker only for the tests to pass. But of course they should be enabled for CI builds.
Best Regards,
Dominik.
Wysłane z aplikacji Poczta dla Windows 10
Od: Niels Basjes
Wysłano: czwartek, 16 sierpnia 2018 14:05
Do: dev@flink.apache.org
Temat: How to handle tests that need docker?
Hi,
I'm working together with Richard Deurwaarder on the PubSub Source and Sink
https://issues.apache.org/jira/browse/FLINK-9311
We currently have most of the code done and we also have several tests that
verify the behavior by actually reading and writing messages to and from
PubSub.
These tests use a docker image provided by Google that includes the pubsub
emulator.
Starting and stopping this thing is really fast.
Starting up, creating topic, running a few tests and cleaning up takes
about 5 seconds.
Although the complete set of tests run on my machine in about 20-25 seconds
it does mean that anyone doing that MUST have docker installed locally.
So my question to the committers: Do we simply leave these tests "always
on" or do you want them to be "off by default" or something else.
I case you do not want the "always on" then please indicate how we should
implement the "on"/"off" switch for these tests.
Thanks.
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: How to handle tests that need docker?
Posted by Gary Yao <ga...@data-artisans.com>.
Hi,
If you are writing JUnit tests, you could use testcontainers [1] to start
the
PubSub container. Then you could introduce a build profile [2] that enables
certain JUnit tests by using Categories [3] or the maven-failsafe-plugin
[4].
Best,
Gary
[1] https://www.testcontainers.org/usage/generic_containers.html#benefits
[2]
https://maven.apache.org/guides/introduction/introduction-to-profiles.html
[3] https://github.com/junit-team/junit4/wiki/categories
[4]
https://maven.apache.org/surefire/maven-failsafe-plugin/examples/inclusion-exclusion.html
On Thu, Aug 16, 2018 at 2:05 PM, Niels Basjes <Ni...@basjes.nl> wrote:
> Hi,
>
> I'm working together with Richard Deurwaarder on the PubSub Source and Sink
> https://issues.apache.org/jira/browse/FLINK-9311
>
> We currently have most of the code done and we also have several tests that
> verify the behavior by actually reading and writing messages to and from
> PubSub.
> These tests use a docker image provided by Google that includes the pubsub
> emulator.
>
> Starting and stopping this thing is really fast.
> Starting up, creating topic, running a few tests and cleaning up takes
> about 5 seconds.
>
> Although the complete set of tests run on my machine in about 20-25 seconds
> it does mean that anyone doing that MUST have docker installed locally.
>
> So my question to the committers: Do we simply leave these tests "always
> on" or do you want them to be "off by default" or something else.
> I case you do not want the "always on" then please indicate how we should
> implement the "on"/"off" switch for these tests.
>
> Thanks.
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>