You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Robert Metzger <rm...@apache.org> on 2020/11/17 15:35:38 UTC
[DISCUSS] Stop adding new bash-based e2e tests to Flink
Hi all,
Since we are currently testing the 1.12 release, and potentially adding
more automated e2e tests, I would like to bring up our end to end tests for
discussion.
Some time ago, we introduced a Java-based testing framework, with the idea
of replacing the current bash-based end to end tests.
Since the introduction of the java-based framework, more bash tests were
actually added, making a future migration even harder.
*For that reason, I would like to propose that we are stopping to add any
new bash end to end tests to Flink. All new end to end tests must be
written in Java and rely on the existing testing framework.*
For the 1.13 release, I'm trying to find some time to revisit potential
improvements for the existing java e2e framework (such as using Docker
images everywhere), as well as a migration plan for the existing bash
tests. We have a large number of bash e2e tests that are just parameterized
differently. If we would start migrating them to Java, we could move a
larger proportion of tests over to the new Java framework, and tackle the
more involved bash tests later (kerberized yarn, kubernetes, ...).
Let me know what you think!
Best,
Robert
PS: If you are wondering why I'm bringing this up now: I'm spending quite a
lot of time trying to figure out really hard to debug issues with our bash
testing infra.
Also, it is very difficult to introduce something generic for all tests
(such as a test-timeout, using docker as the preferred deployment method
etc.) since the tests often don't share common tooling.
Speaking about tooling: there are a lot of utilities everywhere, sometimes
duplicated, with different features / stability etc.
I believe bash is not the right tool for a project this size (in terms of
developers and lines of code)
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Till Rohrmann <tr...@apache.org>.
Big +1 for your proposal Robert. I think the decision to use bash for our
e2e tests already fired back on us several times. Hence, I believe it is
time for a serious effort to get away from it.
The e2e test framework can be found here [1].
When using the e2e test framework, it should be possible to use Flink's
programmatic tools to submit a job (e.g. the RestClusterClient).
For killing K8s processes one could write a wrapper for kubectl which can
be called from the Java code. I think we already have it for some other cli
tools [2]. Alternatively, one could use Fabric8 to communicate with the K8s
cluster.
[1]
https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common
[2]
https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/CommandLineWrapper.java
Cheers,
Till
On Wed, Nov 18, 2020 at 7:42 AM Yang Wang <da...@gmail.com> wrote:
> Thanks for starting this discussion.
>
> In general, I agree with you that a java-based testing framework is better
> than the bash-based. It will
> help a lot for the commons and utilities.
>
> Since I am trying to add a new bash-based Kubernetes HA test, I have some
> quick questions.
> * I am not sure where the java-based framework is. Do you mean
> "flink-jepsen" module or sth else?
> * Maybe it will be harder to run a cli command(e.g. flink run /
> run-application) to submit a Flink job in the java-based framework.
> * It will be harder to inject some operations. For example, kill the
> JobManager in Kubernetes. Currently, I
> am trying to use "kubectl exec" to do this.
>
>
> Best,
> Yang
>
> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
>
> > Hi all,
> >
> > Since we are currently testing the 1.12 release, and potentially adding
> > more automated e2e tests, I would like to bring up our end to end tests
> for
> > discussion.
> >
> > Some time ago, we introduced a Java-based testing framework, with the
> idea
> > of replacing the current bash-based end to end tests.
> > Since the introduction of the java-based framework, more bash tests were
> > actually added, making a future migration even harder.
> >
> > *For that reason, I would like to propose that we are stopping to add any
> > new bash end to end tests to Flink. All new end to end tests must be
> > written in Java and rely on the existing testing framework.*
> >
> > For the 1.13 release, I'm trying to find some time to revisit potential
> > improvements for the existing java e2e framework (such as using Docker
> > images everywhere), as well as a migration plan for the existing bash
> > tests. We have a large number of bash e2e tests that are just
> parameterized
> > differently. If we would start migrating them to Java, we could move a
> > larger proportion of tests over to the new Java framework, and tackle the
> > more involved bash tests later (kerberized yarn, kubernetes, ...).
> >
> > Let me know what you think!
> >
> > Best,
> > Robert
> >
> >
> > PS: If you are wondering why I'm bringing this up now: I'm spending
> quite a
> > lot of time trying to figure out really hard to debug issues with our
> bash
> > testing infra.
> > Also, it is very difficult to introduce something generic for all tests
> > (such as a test-timeout, using docker as the preferred deployment method
> > etc.) since the tests often don't share common tooling.
> > Speaking about tooling: there are a lot of utilities everywhere,
> sometimes
> > duplicated, with different features / stability etc.
> > I believe bash is not the right tool for a project this size (in terms of
> > developers and lines of code)
> >
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Arvid Heise <ar...@ververica.com>.
Also +1 for java-based testing framework; bash just disrupts development
flow too much.
+1 to use docker as much as possible, ideally with testcontainers [1].
[1] https://www.testcontainers.org/
On Wed, Nov 18, 2020 at 10:23 AM Jark Wu <im...@gmail.com> wrote:
> +1 to use the Java-based testing framework and +1 for using docker images
> in the future.
> IIUC, the Java-based testing framework refers to the
> `flink-end-to-end-tests-common` module.
> The java-based framework helped us a lot when debugging the unstable e2e
> tests.
>
> Best,
> Jark
>
> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com> wrote:
>
> > Thanks for starting this discussion.
> >
> > In general, I agree with you that a java-based testing framework is
> better
> > than the bash-based. It will
> > help a lot for the commons and utilities.
> >
> > Since I am trying to add a new bash-based Kubernetes HA test, I have some
> > quick questions.
> > * I am not sure where the java-based framework is. Do you mean
> > "flink-jepsen" module or sth else?
> > * Maybe it will be harder to run a cli command(e.g. flink run /
> > run-application) to submit a Flink job in the java-based framework.
> > * It will be harder to inject some operations. For example, kill the
> > JobManager in Kubernetes. Currently, I
> > am trying to use "kubectl exec" to do this.
> >
> >
> > Best,
> > Yang
> >
> > Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> >
> > > Hi all,
> > >
> > > Since we are currently testing the 1.12 release, and potentially adding
> > > more automated e2e tests, I would like to bring up our end to end tests
> > for
> > > discussion.
> > >
> > > Some time ago, we introduced a Java-based testing framework, with the
> > idea
> > > of replacing the current bash-based end to end tests.
> > > Since the introduction of the java-based framework, more bash tests
> were
> > > actually added, making a future migration even harder.
> > >
> > > *For that reason, I would like to propose that we are stopping to add
> any
> > > new bash end to end tests to Flink. All new end to end tests must be
> > > written in Java and rely on the existing testing framework.*
> > >
> > > For the 1.13 release, I'm trying to find some time to revisit potential
> > > improvements for the existing java e2e framework (such as using Docker
> > > images everywhere), as well as a migration plan for the existing bash
> > > tests. We have a large number of bash e2e tests that are just
> > parameterized
> > > differently. If we would start migrating them to Java, we could move a
> > > larger proportion of tests over to the new Java framework, and tackle
> the
> > > more involved bash tests later (kerberized yarn, kubernetes, ...).
> > >
> > > Let me know what you think!
> > >
> > > Best,
> > > Robert
> > >
> > >
> > > PS: If you are wondering why I'm bringing this up now: I'm spending
> > quite a
> > > lot of time trying to figure out really hard to debug issues with our
> > bash
> > > testing infra.
> > > Also, it is very difficult to introduce something generic for all tests
> > > (such as a test-timeout, using docker as the preferred deployment
> method
> > > etc.) since the tests often don't share common tooling.
> > > Speaking about tooling: there are a lot of utilities everywhere,
> > sometimes
> > > duplicated, with different features / stability etc.
> > > I believe bash is not the right tool for a project this size (in terms
> of
> > > developers and lines of code)
> > >
> >
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Xingbo Huang <hx...@gmail.com>.
Thanks Matthias,
I have created sub-tasks for PyFlink related e2e tests.
Best,
Xingbo
> 2020年11月27日 下午10:53，Jark Wu <im...@gmail.com> 写道：
>
> Thanks Matthias,
>
> I have created sub-tasks for the Table SQL related bash-based e2e tests.
>
> Best,
> Jark
>
> On Fri, 27 Nov 2020 at 21:25, Matthias Pohl <ma...@ververica.com> wrote:
>
>> Thanks Robert for pushing this. +1 for creating java-based e2e tests.
>>
>> In the engine team, we decided to work towards the goal of migrating the
>> bash-based e2e tests to Java/Docker. We plan to migrate the existing
>> bash-based e2e tests located in the Engine team's component space
>> step-by-step. I created an umbrella Jira issue [1] to collect and document
>> the migration efforts. Feel free to do the same by creating subtasks under
>> [1].
>>
>> Best,
>> Matthias
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-20392
>>
>> On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <my...@live.com> wrote:
>>
>>> +1 for java-based E2E test as bash scripts lack of the power to handle
>>> more complicated cases.
>>>
>>> For the docker image improvement, I think we should be more cautious as
>>> developers at China might suffer with the network issue, or at least we
>>> should ensure or find some guides to speed up the image downloading.
>>>
>>> Best
>>> Yun Tang
>>> ________________________________
>>> From: Xingbo Huang <hx...@gmail.com>
>>> Sent: Thursday, November 19, 2020 12:09
>>> To: dev <de...@flink.apache.org>
>>> Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
>>>
>>> Big +1 for java-based e2e tests. Currently PyFlink related tests each
>> take
>>> ~15minutes in bash e2e tests because we are using a secured YARN cluster
>>> which is the only convenient way of starting a YARN cluster in the bash
>> e2e
>>> tests. I think if we migrate these tests to the java-based testing
>>> framework, we will start a Yarn Cluster more conveniently, which will
>>> greatly reduce our testing time.
>>>
>>> Best,
>>> Xingbo
>>>
>>> Rui Li <li...@gmail.com> 于2020年11月19日周四 上午10:47写道：
>>>
>>>> Big +1 to java-based e2e tests. It'll be much easier to write/debug
>> these
>>>> tests.
>>>>
>>>> On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xb...@gmail.com> wrote:
>>>>
>>>>> +1 to stop using bash scripts,
>>>>> and I also have experienced the bash scripts that is really hard to
>>>>> maintain and debug, thanks @Robert for the great work again.
>>>>>
>>>>> I think testcontainers is a nice candidate.
>>>>>
>>>>> Best,
>>>>> Leonard
>>>>>
>>>>>> 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
>>>>>>
>>>>>> +1
>>>>>>
>>>>>> And I want to second Arvid's mention of testcontainers [1].
>>>>>>
>>>>>> [1] https://www.testcontainers.org/
>>>>>>
>>>>>> On 18.11.20 10:43, Yang Wang wrote:
>>>>>>> Thanks till and Jark for sharing the information.
>>>>>>> I am also +1 for this proposal and glad to wire the new introduced
>>> K8s
>>>>> HA
>>>>>>> e2e tests to java based framework.
>>>>>>> Best,
>>>>>>> Yang
>>>>>>> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
>>>>>>>> +1 to use the Java-based testing framework and +1 for using
>> docker
>>>>> images
>>>>>>>> in the future.
>>>>>>>> IIUC, the Java-based testing framework refers to the
>>>>>>>> `flink-end-to-end-tests-common` module.
>>>>>>>> The java-based framework helped us a lot when debugging the
>>> unstable
>>>>> e2e
>>>>>>>> tests.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Jark
>>>>>>>>
>>>>>>>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com>
>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for starting this discussion.
>>>>>>>>>
>>>>>>>>> In general, I agree with you that a java-based testing framework
>>> is
>>>>>>>> better
>>>>>>>>> than the bash-based. It will
>>>>>>>>> help a lot for the commons and utilities.
>>>>>>>>>
>>>>>>>>> Since I am trying to add a new bash-based Kubernetes HA test, I
>>> have
>>>>> some
>>>>>>>>> quick questions.
>>>>>>>>> * I am not sure where the java-based framework is. Do you mean
>>>>>>>>> "flink-jepsen" module or sth else?
>>>>>>>>> * Maybe it will be harder to run a cli command(e.g. flink run /
>>>>>>>>> run-application) to submit a Flink job in the java-based
>>> framework.
>>>>>>>>> * It will be harder to inject some operations. For example, kill
>>> the
>>>>>>>>> JobManager in Kubernetes. Currently, I
>>>>>>>>> am trying to use "kubectl exec" to do this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Yang
>>>>>>>>>
>>>>>>>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Since we are currently testing the 1.12 release, and
>> potentially
>>>>> adding
>>>>>>>>>> more automated e2e tests, I would like to bring up our end to
>> end
>>>>> tests
>>>>>>>>> for
>>>>>>>>>> discussion.
>>>>>>>>>>
>>>>>>>>>> Some time ago, we introduced a Java-based testing framework,
>> with
>>>> the
>>>>>>>>> idea
>>>>>>>>>> of replacing the current bash-based end to end tests.
>>>>>>>>>> Since the introduction of the java-based framework, more bash
>>> tests
>>>>>>>> were
>>>>>>>>>> actually added, making a future migration even harder.
>>>>>>>>>>
>>>>>>>>>> *For that reason, I would like to propose that we are stopping
>> to
>>>> add
>>>>>>>> any
>>>>>>>>>> new bash end to end tests to Flink. All new end to end tests
>> must
>>>> be
>>>>>>>>>> written in Java and rely on the existing testing framework.*
>>>>>>>>>>
>>>>>>>>>> For the 1.13 release, I'm trying to find some time to revisit
>>>>> potential
>>>>>>>>>> improvements for the existing java e2e framework (such as using
>>>>> Docker
>>>>>>>>>> images everywhere), as well as a migration plan for the
>> existing
>>>> bash
>>>>>>>>>> tests. We have a large number of bash e2e tests that are just
>>>>>>>>> parameterized
>>>>>>>>>> differently. If we would start migrating them to Java, we could
>>>> move
>>>>> a
>>>>>>>>>> larger proportion of tests over to the new Java framework, and
>>>> tackle
>>>>>>>> the
>>>>>>>>>> more involved bash tests later (kerberized yarn, kubernetes,
>>> ...).
>>>>>>>>>>
>>>>>>>>>> Let me know what you think!
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Robert
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> PS: If you are wondering why I'm bringing this up now: I'm
>>> spending
>>>>>>>>> quite a
>>>>>>>>>> lot of time trying to figure out really hard to debug issues
>> with
>>>> our
>>>>>>>>> bash
>>>>>>>>>> testing infra.
>>>>>>>>>> Also, it is very difficult to introduce something generic for
>> all
>>>>> tests
>>>>>>>>>> (such as a test-timeout, using docker as the preferred
>> deployment
>>>>>>>> method
>>>>>>>>>> etc.) since the tests often don't share common tooling.
>>>>>>>>>> Speaking about tooling: there are a lot of utilities
>> everywhere,
>>>>>>>>> sometimes
>>>>>>>>>> duplicated, with different features / stability etc.
>>>>>>>>>> I believe bash is not the right tool for a project this size
>> (in
>>>>> terms
>>>>>>>> of
>>>>>>>>>> developers and lines of code)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Best regards!
>>>> Rui Li
>>>>
>>>
>>
>>
>> --
>>
>> Matthias Pohl | Engineer
>>
>> Follow us @VervericaData Ververica <https://www.ververica.com/>
>>
>> --
>>
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
>> Wehner
>>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Jark Wu <im...@gmail.com>.
Thanks Matthias,
I have created sub-tasks for the Table SQL related bash-based e2e tests.
Best,
Jark
On Fri, 27 Nov 2020 at 21:25, Matthias Pohl <ma...@ververica.com> wrote:
> Thanks Robert for pushing this. +1 for creating java-based e2e tests.
>
> In the engine team, we decided to work towards the goal of migrating the
> bash-based e2e tests to Java/Docker. We plan to migrate the existing
> bash-based e2e tests located in the Engine team's component space
> step-by-step. I created an umbrella Jira issue [1] to collect and document
> the migration efforts. Feel free to do the same by creating subtasks under
> [1].
>
> Best,
> Matthias
>
> [1] https://issues.apache.org/jira/browse/FLINK-20392
>
> On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <my...@live.com> wrote:
>
> > +1 for java-based E2E test as bash scripts lack of the power to handle
> > more complicated cases.
> >
> > For the docker image improvement, I think we should be more cautious as
> > developers at China might suffer with the network issue, or at least we
> > should ensure or find some guides to speed up the image downloading.
> >
> > Best
> > Yun Tang
> > ________________________________
> > From: Xingbo Huang <hx...@gmail.com>
> > Sent: Thursday, November 19, 2020 12:09
> > To: dev <de...@flink.apache.org>
> > Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
> >
> > Big +1 for java-based e2e tests. Currently PyFlink related tests each
> take
> > ~15minutes in bash e2e tests because we are using a secured YARN cluster
> > which is the only convenient way of starting a YARN cluster in the bash
> e2e
> > tests. I think if we migrate these tests to the java-based testing
> > framework, we will start a Yarn Cluster more conveniently, which will
> > greatly reduce our testing time.
> >
> > Best,
> > Xingbo
> >
> > Rui Li <li...@gmail.com> 于2020年11月19日周四 上午10:47写道：
> >
> > > Big +1 to java-based e2e tests. It'll be much easier to write/debug
> these
> > > tests.
> > >
> > > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xb...@gmail.com> wrote:
> > >
> > > > +1 to stop using bash scripts,
> > > > and I also have experienced the bash scripts that is really hard to
> > > > maintain and debug, thanks @Robert for the great work again.
> > > >
> > > > I think testcontainers is a nice candidate.
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
> > > > >
> > > > > +1
> > > > >
> > > > > And I want to second Arvid's mention of testcontainers [1].
> > > > >
> > > > > [1] https://www.testcontainers.org/
> > > > >
> > > > > On 18.11.20 10:43, Yang Wang wrote:
> > > > >> Thanks till and Jark for sharing the information.
> > > > >> I am also +1 for this proposal and glad to wire the new introduced
> > K8s
> > > > HA
> > > > >> e2e tests to java based framework.
> > > > >> Best,
> > > > >> Yang
> > > > >> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
> > > > >>> +1 to use the Java-based testing framework and +1 for using
> docker
> > > > images
> > > > >>> in the future.
> > > > >>> IIUC, the Java-based testing framework refers to the
> > > > >>> `flink-end-to-end-tests-common` module.
> > > > >>> The java-based framework helped us a lot when debugging the
> > unstable
> > > > e2e
> > > > >>> tests.
> > > > >>>
> > > > >>> Best,
> > > > >>> Jark
> > > > >>>
> > > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com>
> > > wrote:
> > > > >>>
> > > > >>>> Thanks for starting this discussion.
> > > > >>>>
> > > > >>>> In general, I agree with you that a java-based testing framework
> > is
> > > > >>> better
> > > > >>>> than the bash-based. It will
> > > > >>>> help a lot for the commons and utilities.
> > > > >>>>
> > > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I
> > have
> > > > some
> > > > >>>> quick questions.
> > > > >>>> * I am not sure where the java-based framework is. Do you mean
> > > > >>>> "flink-jepsen" module or sth else?
> > > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > > > >>>> run-application) to submit a Flink job in the java-based
> > framework.
> > > > >>>> * It will be harder to inject some operations. For example, kill
> > the
> > > > >>>> JobManager in Kubernetes. Currently, I
> > > > >>>> am trying to use "kubectl exec" to do this.
> > > > >>>>
> > > > >>>>
> > > > >>>> Best,
> > > > >>>> Yang
> > > > >>>>
> > > > >>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> > > > >>>>
> > > > >>>>> Hi all,
> > > > >>>>>
> > > > >>>>> Since we are currently testing the 1.12 release, and
> potentially
> > > > adding
> > > > >>>>> more automated e2e tests, I would like to bring up our end to
> end
> > > > tests
> > > > >>>> for
> > > > >>>>> discussion.
> > > > >>>>>
> > > > >>>>> Some time ago, we introduced a Java-based testing framework,
> with
> > > the
> > > > >>>> idea
> > > > >>>>> of replacing the current bash-based end to end tests.
> > > > >>>>> Since the introduction of the java-based framework, more bash
> > tests
> > > > >>> were
> > > > >>>>> actually added, making a future migration even harder.
> > > > >>>>>
> > > > >>>>> *For that reason, I would like to propose that we are stopping
> to
> > > add
> > > > >>> any
> > > > >>>>> new bash end to end tests to Flink. All new end to end tests
> must
> > > be
> > > > >>>>> written in Java and rely on the existing testing framework.*
> > > > >>>>>
> > > > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > > > potential
> > > > >>>>> improvements for the existing java e2e framework (such as using
> > > > Docker
> > > > >>>>> images everywhere), as well as a migration plan for the
> existing
> > > bash
> > > > >>>>> tests. We have a large number of bash e2e tests that are just
> > > > >>>> parameterized
> > > > >>>>> differently. If we would start migrating them to Java, we could
> > > move
> > > > a
> > > > >>>>> larger proportion of tests over to the new Java framework, and
> > > tackle
> > > > >>> the
> > > > >>>>> more involved bash tests later (kerberized yarn, kubernetes,
> > ...).
> > > > >>>>>
> > > > >>>>> Let me know what you think!
> > > > >>>>>
> > > > >>>>> Best,
> > > > >>>>> Robert
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm
> > spending
> > > > >>>> quite a
> > > > >>>>> lot of time trying to figure out really hard to debug issues
> with
> > > our
> > > > >>>> bash
> > > > >>>>> testing infra.
> > > > >>>>> Also, it is very difficult to introduce something generic for
> all
> > > > tests
> > > > >>>>> (such as a test-timeout, using docker as the preferred
> deployment
> > > > >>> method
> > > > >>>>> etc.) since the tests often don't share common tooling.
> > > > >>>>> Speaking about tooling: there are a lot of utilities
> everywhere,
> > > > >>>> sometimes
> > > > >>>>> duplicated, with different features / stability etc.
> > > > >>>>> I believe bash is not the right tool for a project this size
> (in
> > > > terms
> > > > >>> of
> > > > >>>>> developers and lines of code)
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >
> > > >
> > > >
> > >
> > > --
> > > Best regards!
> > > Rui Li
> > >
> >
>
>
> --
>
> Matthias Pohl | Engineer
>
> Follow us @VervericaData Ververica <https://www.ververica.com/>
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
> Wehner
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Matthias Pohl <ma...@ververica.com>.
Thanks Robert for pushing this. +1 for creating java-based e2e tests.
In the engine team, we decided to work towards the goal of migrating the
bash-based e2e tests to Java/Docker. We plan to migrate the existing
bash-based e2e tests located in the Engine team's component space
step-by-step. I created an umbrella Jira issue [1] to collect and document
the migration efforts. Feel free to do the same by creating subtasks under
[1].
Best,
Matthias
[1] https://issues.apache.org/jira/browse/FLINK-20392
On Thu, Nov 19, 2020 at 8:02 AM Yun Tang <my...@live.com> wrote:
> +1 for java-based E2E test as bash scripts lack of the power to handle
> more complicated cases.
>
> For the docker image improvement, I think we should be more cautious as
> developers at China might suffer with the network issue, or at least we
> should ensure or find some guides to speed up the image downloading.
>
> Best
> Yun Tang
> ________________________________
> From: Xingbo Huang <hx...@gmail.com>
> Sent: Thursday, November 19, 2020 12:09
> To: dev <de...@flink.apache.org>
> Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
>
> Big +1 for java-based e2e tests. Currently PyFlink related tests each take
> ~15minutes in bash e2e tests because we are using a secured YARN cluster
> which is the only convenient way of starting a YARN cluster in the bash e2e
> tests. I think if we migrate these tests to the java-based testing
> framework, we will start a Yarn Cluster more conveniently, which will
> greatly reduce our testing time.
>
> Best,
> Xingbo
>
> Rui Li <li...@gmail.com> 于2020年11月19日周四 上午10:47写道：
>
> > Big +1 to java-based e2e tests. It'll be much easier to write/debug these
> > tests.
> >
> > On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xb...@gmail.com> wrote:
> >
> > > +1 to stop using bash scripts,
> > > and I also have experienced the bash scripts that is really hard to
> > > maintain and debug, thanks @Robert for the great work again.
> > >
> > > I think testcontainers is a nice candidate.
> > >
> > > Best,
> > > Leonard
> > >
> > > > 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
> > > >
> > > > +1
> > > >
> > > > And I want to second Arvid's mention of testcontainers [1].
> > > >
> > > > [1] https://www.testcontainers.org/
> > > >
> > > > On 18.11.20 10:43, Yang Wang wrote:
> > > >> Thanks till and Jark for sharing the information.
> > > >> I am also +1 for this proposal and glad to wire the new introduced
> K8s
> > > HA
> > > >> e2e tests to java based framework.
> > > >> Best,
> > > >> Yang
> > > >> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
> > > >>> +1 to use the Java-based testing framework and +1 for using docker
> > > images
> > > >>> in the future.
> > > >>> IIUC, the Java-based testing framework refers to the
> > > >>> `flink-end-to-end-tests-common` module.
> > > >>> The java-based framework helped us a lot when debugging the
> unstable
> > > e2e
> > > >>> tests.
> > > >>>
> > > >>> Best,
> > > >>> Jark
> > > >>>
> > > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com>
> > wrote:
> > > >>>
> > > >>>> Thanks for starting this discussion.
> > > >>>>
> > > >>>> In general, I agree with you that a java-based testing framework
> is
> > > >>> better
> > > >>>> than the bash-based. It will
> > > >>>> help a lot for the commons and utilities.
> > > >>>>
> > > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I
> have
> > > some
> > > >>>> quick questions.
> > > >>>> * I am not sure where the java-based framework is. Do you mean
> > > >>>> "flink-jepsen" module or sth else?
> > > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > > >>>> run-application) to submit a Flink job in the java-based
> framework.
> > > >>>> * It will be harder to inject some operations. For example, kill
> the
> > > >>>> JobManager in Kubernetes. Currently, I
> > > >>>> am trying to use "kubectl exec" to do this.
> > > >>>>
> > > >>>>
> > > >>>> Best,
> > > >>>> Yang
> > > >>>>
> > > >>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> > > >>>>
> > > >>>>> Hi all,
> > > >>>>>
> > > >>>>> Since we are currently testing the 1.12 release, and potentially
> > > adding
> > > >>>>> more automated e2e tests, I would like to bring up our end to end
> > > tests
> > > >>>> for
> > > >>>>> discussion.
> > > >>>>>
> > > >>>>> Some time ago, we introduced a Java-based testing framework, with
> > the
> > > >>>> idea
> > > >>>>> of replacing the current bash-based end to end tests.
> > > >>>>> Since the introduction of the java-based framework, more bash
> tests
> > > >>> were
> > > >>>>> actually added, making a future migration even harder.
> > > >>>>>
> > > >>>>> *For that reason, I would like to propose that we are stopping to
> > add
> > > >>> any
> > > >>>>> new bash end to end tests to Flink. All new end to end tests must
> > be
> > > >>>>> written in Java and rely on the existing testing framework.*
> > > >>>>>
> > > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > > potential
> > > >>>>> improvements for the existing java e2e framework (such as using
> > > Docker
> > > >>>>> images everywhere), as well as a migration plan for the existing
> > bash
> > > >>>>> tests. We have a large number of bash e2e tests that are just
> > > >>>> parameterized
> > > >>>>> differently. If we would start migrating them to Java, we could
> > move
> > > a
> > > >>>>> larger proportion of tests over to the new Java framework, and
> > tackle
> > > >>> the
> > > >>>>> more involved bash tests later (kerberized yarn, kubernetes,
> ...).
> > > >>>>>
> > > >>>>> Let me know what you think!
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Robert
> > > >>>>>
> > > >>>>>
> > > >>>>> PS: If you are wondering why I'm bringing this up now: I'm
> spending
> > > >>>> quite a
> > > >>>>> lot of time trying to figure out really hard to debug issues with
> > our
> > > >>>> bash
> > > >>>>> testing infra.
> > > >>>>> Also, it is very difficult to introduce something generic for all
> > > tests
> > > >>>>> (such as a test-timeout, using docker as the preferred deployment
> > > >>> method
> > > >>>>> etc.) since the tests often don't share common tooling.
> > > >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> > > >>>> sometimes
> > > >>>>> duplicated, with different features / stability etc.
> > > >>>>> I believe bash is not the right tool for a project this size (in
> > > terms
> > > >>> of
> > > >>>>> developers and lines of code)
> > > >>>>>
> > > >>>>
> > > >>>
> > > >
> > >
> > >
> >
> > --
> > Best regards!
> > Rui Li
> >
>
--
Matthias Pohl | Engineer
Follow us @VervericaData Ververica <https://www.ververica.com/>
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Yun Tang <my...@live.com>.
+1 for java-based E2E test as bash scripts lack of the power to handle more complicated cases.
For the docker image improvement, I think we should be more cautious as developers at China might suffer with the network issue, or at least we should ensure or find some guides to speed up the image downloading.
Best
Yun Tang
________________________________
From: Xingbo Huang <hx...@gmail.com>
Sent: Thursday, November 19, 2020 12:09
To: dev <de...@flink.apache.org>
Subject: Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Big +1 for java-based e2e tests. Currently PyFlink related tests each take
~15minutes in bash e2e tests because we are using a secured YARN cluster
which is the only convenient way of starting a YARN cluster in the bash e2e
tests. I think if we migrate these tests to the java-based testing
framework, we will start a Yarn Cluster more conveniently, which will
greatly reduce our testing time.
Best,
Xingbo
Rui Li <li...@gmail.com> 于2020年11月19日周四 上午10:47写道：
> Big +1 to java-based e2e tests. It'll be much easier to write/debug these
> tests.
>
> On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xb...@gmail.com> wrote:
>
> > +1 to stop using bash scripts,
> > and I also have experienced the bash scripts that is really hard to
> > maintain and debug, thanks @Robert for the great work again.
> >
> > I think testcontainers is a nice candidate.
> >
> > Best,
> > Leonard
> >
> > > 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
> > >
> > > +1
> > >
> > > And I want to second Arvid's mention of testcontainers [1].
> > >
> > > [1] https://www.testcontainers.org/
> > >
> > > On 18.11.20 10:43, Yang Wang wrote:
> > >> Thanks till and Jark for sharing the information.
> > >> I am also +1 for this proposal and glad to wire the new introduced K8s
> > HA
> > >> e2e tests to java based framework.
> > >> Best,
> > >> Yang
> > >> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
> > >>> +1 to use the Java-based testing framework and +1 for using docker
> > images
> > >>> in the future.
> > >>> IIUC, the Java-based testing framework refers to the
> > >>> `flink-end-to-end-tests-common` module.
> > >>> The java-based framework helped us a lot when debugging the unstable
> > e2e
> > >>> tests.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com>
> wrote:
> > >>>
> > >>>> Thanks for starting this discussion.
> > >>>>
> > >>>> In general, I agree with you that a java-based testing framework is
> > >>> better
> > >>>> than the bash-based. It will
> > >>>> help a lot for the commons and utilities.
> > >>>>
> > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have
> > some
> > >>>> quick questions.
> > >>>> * I am not sure where the java-based framework is. Do you mean
> > >>>> "flink-jepsen" module or sth else?
> > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > >>>> run-application) to submit a Flink job in the java-based framework.
> > >>>> * It will be harder to inject some operations. For example, kill the
> > >>>> JobManager in Kubernetes. Currently, I
> > >>>> am trying to use "kubectl exec" to do this.
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>> Yang
> > >>>>
> > >>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Since we are currently testing the 1.12 release, and potentially
> > adding
> > >>>>> more automated e2e tests, I would like to bring up our end to end
> > tests
> > >>>> for
> > >>>>> discussion.
> > >>>>>
> > >>>>> Some time ago, we introduced a Java-based testing framework, with
> the
> > >>>> idea
> > >>>>> of replacing the current bash-based end to end tests.
> > >>>>> Since the introduction of the java-based framework, more bash tests
> > >>> were
> > >>>>> actually added, making a future migration even harder.
> > >>>>>
> > >>>>> *For that reason, I would like to propose that we are stopping to
> add
> > >>> any
> > >>>>> new bash end to end tests to Flink. All new end to end tests must
> be
> > >>>>> written in Java and rely on the existing testing framework.*
> > >>>>>
> > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > potential
> > >>>>> improvements for the existing java e2e framework (such as using
> > Docker
> > >>>>> images everywhere), as well as a migration plan for the existing
> bash
> > >>>>> tests. We have a large number of bash e2e tests that are just
> > >>>> parameterized
> > >>>>> differently. If we would start migrating them to Java, we could
> move
> > a
> > >>>>> larger proportion of tests over to the new Java framework, and
> tackle
> > >>> the
> > >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
> > >>>>>
> > >>>>> Let me know what you think!
> > >>>>>
> > >>>>> Best,
> > >>>>> Robert
> > >>>>>
> > >>>>>
> > >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
> > >>>> quite a
> > >>>>> lot of time trying to figure out really hard to debug issues with
> our
> > >>>> bash
> > >>>>> testing infra.
> > >>>>> Also, it is very difficult to introduce something generic for all
> > tests
> > >>>>> (such as a test-timeout, using docker as the preferred deployment
> > >>> method
> > >>>>> etc.) since the tests often don't share common tooling.
> > >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> > >>>> sometimes
> > >>>>> duplicated, with different features / stability etc.
> > >>>>> I believe bash is not the right tool for a project this size (in
> > terms
> > >>> of
> > >>>>> developers and lines of code)
> > >>>>>
> > >>>>
> > >>>
> > >
> >
> >
>
> --
> Best regards!
> Rui Li
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Xingbo Huang <hx...@gmail.com>.
Big +1 for java-based e2e tests. Currently PyFlink related tests each take
~15minutes in bash e2e tests because we are using a secured YARN cluster
which is the only convenient way of starting a YARN cluster in the bash e2e
tests. I think if we migrate these tests to the java-based testing
framework, we will start a Yarn Cluster more conveniently, which will
greatly reduce our testing time.
Best,
Xingbo
Rui Li <li...@gmail.com> 于2020年11月19日周四 上午10:47写道：
> Big +1 to java-based e2e tests. It'll be much easier to write/debug these
> tests.
>
> On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xb...@gmail.com> wrote:
>
> > +1 to stop using bash scripts,
> > and I also have experienced the bash scripts that is really hard to
> > maintain and debug, thanks @Robert for the great work again.
> >
> > I think testcontainers is a nice candidate.
> >
> > Best,
> > Leonard
> >
> > > 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
> > >
> > > +1
> > >
> > > And I want to second Arvid's mention of testcontainers [1].
> > >
> > > [1] https://www.testcontainers.org/
> > >
> > > On 18.11.20 10:43, Yang Wang wrote:
> > >> Thanks till and Jark for sharing the information.
> > >> I am also +1 for this proposal and glad to wire the new introduced K8s
> > HA
> > >> e2e tests to java based framework.
> > >> Best,
> > >> Yang
> > >> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
> > >>> +1 to use the Java-based testing framework and +1 for using docker
> > images
> > >>> in the future.
> > >>> IIUC, the Java-based testing framework refers to the
> > >>> `flink-end-to-end-tests-common` module.
> > >>> The java-based framework helped us a lot when debugging the unstable
> > e2e
> > >>> tests.
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com>
> wrote:
> > >>>
> > >>>> Thanks for starting this discussion.
> > >>>>
> > >>>> In general, I agree with you that a java-based testing framework is
> > >>> better
> > >>>> than the bash-based. It will
> > >>>> help a lot for the commons and utilities.
> > >>>>
> > >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have
> > some
> > >>>> quick questions.
> > >>>> * I am not sure where the java-based framework is. Do you mean
> > >>>> "flink-jepsen" module or sth else?
> > >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> > >>>> run-application) to submit a Flink job in the java-based framework.
> > >>>> * It will be harder to inject some operations. For example, kill the
> > >>>> JobManager in Kubernetes. Currently, I
> > >>>> am trying to use "kubectl exec" to do this.
> > >>>>
> > >>>>
> > >>>> Best,
> > >>>> Yang
> > >>>>
> > >>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> Since we are currently testing the 1.12 release, and potentially
> > adding
> > >>>>> more automated e2e tests, I would like to bring up our end to end
> > tests
> > >>>> for
> > >>>>> discussion.
> > >>>>>
> > >>>>> Some time ago, we introduced a Java-based testing framework, with
> the
> > >>>> idea
> > >>>>> of replacing the current bash-based end to end tests.
> > >>>>> Since the introduction of the java-based framework, more bash tests
> > >>> were
> > >>>>> actually added, making a future migration even harder.
> > >>>>>
> > >>>>> *For that reason, I would like to propose that we are stopping to
> add
> > >>> any
> > >>>>> new bash end to end tests to Flink. All new end to end tests must
> be
> > >>>>> written in Java and rely on the existing testing framework.*
> > >>>>>
> > >>>>> For the 1.13 release, I'm trying to find some time to revisit
> > potential
> > >>>>> improvements for the existing java e2e framework (such as using
> > Docker
> > >>>>> images everywhere), as well as a migration plan for the existing
> bash
> > >>>>> tests. We have a large number of bash e2e tests that are just
> > >>>> parameterized
> > >>>>> differently. If we would start migrating them to Java, we could
> move
> > a
> > >>>>> larger proportion of tests over to the new Java framework, and
> tackle
> > >>> the
> > >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
> > >>>>>
> > >>>>> Let me know what you think!
> > >>>>>
> > >>>>> Best,
> > >>>>> Robert
> > >>>>>
> > >>>>>
> > >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
> > >>>> quite a
> > >>>>> lot of time trying to figure out really hard to debug issues with
> our
> > >>>> bash
> > >>>>> testing infra.
> > >>>>> Also, it is very difficult to introduce something generic for all
> > tests
> > >>>>> (such as a test-timeout, using docker as the preferred deployment
> > >>> method
> > >>>>> etc.) since the tests often don't share common tooling.
> > >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> > >>>> sometimes
> > >>>>> duplicated, with different features / stability etc.
> > >>>>> I believe bash is not the right tool for a project this size (in
> > terms
> > >>> of
> > >>>>> developers and lines of code)
> > >>>>>
> > >>>>
> > >>>
> > >
> >
> >
>
> --
> Best regards!
> Rui Li
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Rui Li <li...@gmail.com>.
Big +1 to java-based e2e tests. It'll be much easier to write/debug these
tests.
On Wed, Nov 18, 2020 at 9:44 PM Leonard Xu <xb...@gmail.com> wrote:
> +1 to stop using bash scripts,
> and I also have experienced the bash scripts that is really hard to
> maintain and debug, thanks @Robert for the great work again.
>
> I think testcontainers is a nice candidate.
>
> Best,
> Leonard
>
> > 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
> >
> > +1
> >
> > And I want to second Arvid's mention of testcontainers [1].
> >
> > [1] https://www.testcontainers.org/
> >
> > On 18.11.20 10:43, Yang Wang wrote:
> >> Thanks till and Jark for sharing the information.
> >> I am also +1 for this proposal and glad to wire the new introduced K8s
> HA
> >> e2e tests to java based framework.
> >> Best,
> >> Yang
> >> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
> >>> +1 to use the Java-based testing framework and +1 for using docker
> images
> >>> in the future.
> >>> IIUC, the Java-based testing framework refers to the
> >>> `flink-end-to-end-tests-common` module.
> >>> The java-based framework helped us a lot when debugging the unstable
> e2e
> >>> tests.
> >>>
> >>> Best,
> >>> Jark
> >>>
> >>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com> wrote:
> >>>
> >>>> Thanks for starting this discussion.
> >>>>
> >>>> In general, I agree with you that a java-based testing framework is
> >>> better
> >>>> than the bash-based. It will
> >>>> help a lot for the commons and utilities.
> >>>>
> >>>> Since I am trying to add a new bash-based Kubernetes HA test, I have
> some
> >>>> quick questions.
> >>>> * I am not sure where the java-based framework is. Do you mean
> >>>> "flink-jepsen" module or sth else?
> >>>> * Maybe it will be harder to run a cli command(e.g. flink run /
> >>>> run-application) to submit a Flink job in the java-based framework.
> >>>> * It will be harder to inject some operations. For example, kill the
> >>>> JobManager in Kubernetes. Currently, I
> >>>> am trying to use "kubectl exec" to do this.
> >>>>
> >>>>
> >>>> Best,
> >>>> Yang
> >>>>
> >>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Since we are currently testing the 1.12 release, and potentially
> adding
> >>>>> more automated e2e tests, I would like to bring up our end to end
> tests
> >>>> for
> >>>>> discussion.
> >>>>>
> >>>>> Some time ago, we introduced a Java-based testing framework, with the
> >>>> idea
> >>>>> of replacing the current bash-based end to end tests.
> >>>>> Since the introduction of the java-based framework, more bash tests
> >>> were
> >>>>> actually added, making a future migration even harder.
> >>>>>
> >>>>> *For that reason, I would like to propose that we are stopping to add
> >>> any
> >>>>> new bash end to end tests to Flink. All new end to end tests must be
> >>>>> written in Java and rely on the existing testing framework.*
> >>>>>
> >>>>> For the 1.13 release, I'm trying to find some time to revisit
> potential
> >>>>> improvements for the existing java e2e framework (such as using
> Docker
> >>>>> images everywhere), as well as a migration plan for the existing bash
> >>>>> tests. We have a large number of bash e2e tests that are just
> >>>> parameterized
> >>>>> differently. If we would start migrating them to Java, we could move
> a
> >>>>> larger proportion of tests over to the new Java framework, and tackle
> >>> the
> >>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
> >>>>>
> >>>>> Let me know what you think!
> >>>>>
> >>>>> Best,
> >>>>> Robert
> >>>>>
> >>>>>
> >>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
> >>>> quite a
> >>>>> lot of time trying to figure out really hard to debug issues with our
> >>>> bash
> >>>>> testing infra.
> >>>>> Also, it is very difficult to introduce something generic for all
> tests
> >>>>> (such as a test-timeout, using docker as the preferred deployment
> >>> method
> >>>>> etc.) since the tests often don't share common tooling.
> >>>>> Speaking about tooling: there are a lot of utilities everywhere,
> >>>> sometimes
> >>>>> duplicated, with different features / stability etc.
> >>>>> I believe bash is not the right tool for a project this size (in
> terms
> >>> of
> >>>>> developers and lines of code)
> >>>>>
> >>>>
> >>>
> >
>
>
--
Best regards!
Rui Li
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Leonard Xu <xb...@gmail.com>.
+1 to stop using bash scripts,
and I also have experienced the bash scripts that is really hard to maintain and debug, thanks @Robert for the great work again.
I think testcontainers is a nice candidate.
Best,
Leonard
> 在 2020年11月18日，19:46，Aljoscha Krettek <al...@apache.org> 写道：
>
> +1
>
> And I want to second Arvid's mention of testcontainers [1].
>
> [1] https://www.testcontainers.org/
>
> On 18.11.20 10:43, Yang Wang wrote:
>> Thanks till and Jark for sharing the information.
>> I am also +1 for this proposal and glad to wire the new introduced K8s HA
>> e2e tests to java based framework.
>> Best,
>> Yang
>> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
>>> +1 to use the Java-based testing framework and +1 for using docker images
>>> in the future.
>>> IIUC, the Java-based testing framework refers to the
>>> `flink-end-to-end-tests-common` module.
>>> The java-based framework helped us a lot when debugging the unstable e2e
>>> tests.
>>>
>>> Best,
>>> Jark
>>>
>>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com> wrote:
>>>
>>>> Thanks for starting this discussion.
>>>>
>>>> In general, I agree with you that a java-based testing framework is
>>> better
>>>> than the bash-based. It will
>>>> help a lot for the commons and utilities.
>>>>
>>>> Since I am trying to add a new bash-based Kubernetes HA test, I have some
>>>> quick questions.
>>>> * I am not sure where the java-based framework is. Do you mean
>>>> "flink-jepsen" module or sth else?
>>>> * Maybe it will be harder to run a cli command(e.g. flink run /
>>>> run-application) to submit a Flink job in the java-based framework.
>>>> * It will be harder to inject some operations. For example, kill the
>>>> JobManager in Kubernetes. Currently, I
>>>> am trying to use "kubectl exec" to do this.
>>>>
>>>>
>>>> Best,
>>>> Yang
>>>>
>>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
>>>>
>>>>> Hi all,
>>>>>
>>>>> Since we are currently testing the 1.12 release, and potentially adding
>>>>> more automated e2e tests, I would like to bring up our end to end tests
>>>> for
>>>>> discussion.
>>>>>
>>>>> Some time ago, we introduced a Java-based testing framework, with the
>>>> idea
>>>>> of replacing the current bash-based end to end tests.
>>>>> Since the introduction of the java-based framework, more bash tests
>>> were
>>>>> actually added, making a future migration even harder.
>>>>>
>>>>> *For that reason, I would like to propose that we are stopping to add
>>> any
>>>>> new bash end to end tests to Flink. All new end to end tests must be
>>>>> written in Java and rely on the existing testing framework.*
>>>>>
>>>>> For the 1.13 release, I'm trying to find some time to revisit potential
>>>>> improvements for the existing java e2e framework (such as using Docker
>>>>> images everywhere), as well as a migration plan for the existing bash
>>>>> tests. We have a large number of bash e2e tests that are just
>>>> parameterized
>>>>> differently. If we would start migrating them to Java, we could move a
>>>>> larger proportion of tests over to the new Java framework, and tackle
>>> the
>>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
>>>>>
>>>>> Let me know what you think!
>>>>>
>>>>> Best,
>>>>> Robert
>>>>>
>>>>>
>>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
>>>> quite a
>>>>> lot of time trying to figure out really hard to debug issues with our
>>>> bash
>>>>> testing infra.
>>>>> Also, it is very difficult to introduce something generic for all tests
>>>>> (such as a test-timeout, using docker as the preferred deployment
>>> method
>>>>> etc.) since the tests often don't share common tooling.
>>>>> Speaking about tooling: there are a lot of utilities everywhere,
>>>> sometimes
>>>>> duplicated, with different features / stability etc.
>>>>> I believe bash is not the right tool for a project this size (in terms
>>> of
>>>>> developers and lines of code)
>>>>>
>>>>
>>>
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Aljoscha Krettek <al...@apache.org>.
+1
And I want to second Arvid's mention of testcontainers [1].
[1] https://www.testcontainers.org/
On 18.11.20 10:43, Yang Wang wrote:
> Thanks till and Jark for sharing the information.
>
> I am also +1 for this proposal and glad to wire the new introduced K8s HA
> e2e tests to java based framework.
>
> Best,
> Yang
>
> Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
>
>> +1 to use the Java-based testing framework and +1 for using docker images
>> in the future.
>> IIUC, the Java-based testing framework refers to the
>> `flink-end-to-end-tests-common` module.
>> The java-based framework helped us a lot when debugging the unstable e2e
>> tests.
>>
>> Best,
>> Jark
>>
>> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com> wrote:
>>
>>> Thanks for starting this discussion.
>>>
>>> In general, I agree with you that a java-based testing framework is
>> better
>>> than the bash-based. It will
>>> help a lot for the commons and utilities.
>>>
>>> Since I am trying to add a new bash-based Kubernetes HA test, I have some
>>> quick questions.
>>> * I am not sure where the java-based framework is. Do you mean
>>> "flink-jepsen" module or sth else?
>>> * Maybe it will be harder to run a cli command(e.g. flink run /
>>> run-application) to submit a Flink job in the java-based framework.
>>> * It will be harder to inject some operations. For example, kill the
>>> JobManager in Kubernetes. Currently, I
>>> am trying to use "kubectl exec" to do this.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
>>>
>>>> Hi all,
>>>>
>>>> Since we are currently testing the 1.12 release, and potentially adding
>>>> more automated e2e tests, I would like to bring up our end to end tests
>>> for
>>>> discussion.
>>>>
>>>> Some time ago, we introduced a Java-based testing framework, with the
>>> idea
>>>> of replacing the current bash-based end to end tests.
>>>> Since the introduction of the java-based framework, more bash tests
>> were
>>>> actually added, making a future migration even harder.
>>>>
>>>> *For that reason, I would like to propose that we are stopping to add
>> any
>>>> new bash end to end tests to Flink. All new end to end tests must be
>>>> written in Java and rely on the existing testing framework.*
>>>>
>>>> For the 1.13 release, I'm trying to find some time to revisit potential
>>>> improvements for the existing java e2e framework (such as using Docker
>>>> images everywhere), as well as a migration plan for the existing bash
>>>> tests. We have a large number of bash e2e tests that are just
>>> parameterized
>>>> differently. If we would start migrating them to Java, we could move a
>>>> larger proportion of tests over to the new Java framework, and tackle
>> the
>>>> more involved bash tests later (kerberized yarn, kubernetes, ...).
>>>>
>>>> Let me know what you think!
>>>>
>>>> Best,
>>>> Robert
>>>>
>>>>
>>>> PS: If you are wondering why I'm bringing this up now: I'm spending
>>> quite a
>>>> lot of time trying to figure out really hard to debug issues with our
>>> bash
>>>> testing infra.
>>>> Also, it is very difficult to introduce something generic for all tests
>>>> (such as a test-timeout, using docker as the preferred deployment
>> method
>>>> etc.) since the tests often don't share common tooling.
>>>> Speaking about tooling: there are a lot of utilities everywhere,
>>> sometimes
>>>> duplicated, with different features / stability etc.
>>>> I believe bash is not the right tool for a project this size (in terms
>> of
>>>> developers and lines of code)
>>>>
>>>
>>
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Yang Wang <da...@gmail.com>.
Thanks till and Jark for sharing the information.
I am also +1 for this proposal and glad to wire the new introduced K8s HA
e2e tests to java based framework.
Best,
Yang
Jark Wu <im...@gmail.com> 于2020年11月18日周三 下午5:23写道：
> +1 to use the Java-based testing framework and +1 for using docker images
> in the future.
> IIUC, the Java-based testing framework refers to the
> `flink-end-to-end-tests-common` module.
> The java-based framework helped us a lot when debugging the unstable e2e
> tests.
>
> Best,
> Jark
>
> On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com> wrote:
>
> > Thanks for starting this discussion.
> >
> > In general, I agree with you that a java-based testing framework is
> better
> > than the bash-based. It will
> > help a lot for the commons and utilities.
> >
> > Since I am trying to add a new bash-based Kubernetes HA test, I have some
> > quick questions.
> > * I am not sure where the java-based framework is. Do you mean
> > "flink-jepsen" module or sth else?
> > * Maybe it will be harder to run a cli command(e.g. flink run /
> > run-application) to submit a Flink job in the java-based framework.
> > * It will be harder to inject some operations. For example, kill the
> > JobManager in Kubernetes. Currently, I
> > am trying to use "kubectl exec" to do this.
> >
> >
> > Best,
> > Yang
> >
> > Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> >
> > > Hi all,
> > >
> > > Since we are currently testing the 1.12 release, and potentially adding
> > > more automated e2e tests, I would like to bring up our end to end tests
> > for
> > > discussion.
> > >
> > > Some time ago, we introduced a Java-based testing framework, with the
> > idea
> > > of replacing the current bash-based end to end tests.
> > > Since the introduction of the java-based framework, more bash tests
> were
> > > actually added, making a future migration even harder.
> > >
> > > *For that reason, I would like to propose that we are stopping to add
> any
> > > new bash end to end tests to Flink. All new end to end tests must be
> > > written in Java and rely on the existing testing framework.*
> > >
> > > For the 1.13 release, I'm trying to find some time to revisit potential
> > > improvements for the existing java e2e framework (such as using Docker
> > > images everywhere), as well as a migration plan for the existing bash
> > > tests. We have a large number of bash e2e tests that are just
> > parameterized
> > > differently. If we would start migrating them to Java, we could move a
> > > larger proportion of tests over to the new Java framework, and tackle
> the
> > > more involved bash tests later (kerberized yarn, kubernetes, ...).
> > >
> > > Let me know what you think!
> > >
> > > Best,
> > > Robert
> > >
> > >
> > > PS: If you are wondering why I'm bringing this up now: I'm spending
> > quite a
> > > lot of time trying to figure out really hard to debug issues with our
> > bash
> > > testing infra.
> > > Also, it is very difficult to introduce something generic for all tests
> > > (such as a test-timeout, using docker as the preferred deployment
> method
> > > etc.) since the tests often don't share common tooling.
> > > Speaking about tooling: there are a lot of utilities everywhere,
> > sometimes
> > > duplicated, with different features / stability etc.
> > > I believe bash is not the right tool for a project this size (in terms
> of
> > > developers and lines of code)
> > >
> >
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Jark Wu <im...@gmail.com>.
+1 to use the Java-based testing framework and +1 for using docker images
in the future.
IIUC, the Java-based testing framework refers to the
`flink-end-to-end-tests-common` module.
The java-based framework helped us a lot when debugging the unstable e2e
tests.
Best,
Jark
On Wed, 18 Nov 2020 at 14:42, Yang Wang <da...@gmail.com> wrote:
> Thanks for starting this discussion.
>
> In general, I agree with you that a java-based testing framework is better
> than the bash-based. It will
> help a lot for the commons and utilities.
>
> Since I am trying to add a new bash-based Kubernetes HA test, I have some
> quick questions.
> * I am not sure where the java-based framework is. Do you mean
> "flink-jepsen" module or sth else?
> * Maybe it will be harder to run a cli command(e.g. flink run /
> run-application) to submit a Flink job in the java-based framework.
> * It will be harder to inject some operations. For example, kill the
> JobManager in Kubernetes. Currently, I
> am trying to use "kubectl exec" to do this.
>
>
> Best,
> Yang
>
> Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
>
> > Hi all,
> >
> > Since we are currently testing the 1.12 release, and potentially adding
> > more automated e2e tests, I would like to bring up our end to end tests
> for
> > discussion.
> >
> > Some time ago, we introduced a Java-based testing framework, with the
> idea
> > of replacing the current bash-based end to end tests.
> > Since the introduction of the java-based framework, more bash tests were
> > actually added, making a future migration even harder.
> >
> > *For that reason, I would like to propose that we are stopping to add any
> > new bash end to end tests to Flink. All new end to end tests must be
> > written in Java and rely on the existing testing framework.*
> >
> > For the 1.13 release, I'm trying to find some time to revisit potential
> > improvements for the existing java e2e framework (such as using Docker
> > images everywhere), as well as a migration plan for the existing bash
> > tests. We have a large number of bash e2e tests that are just
> parameterized
> > differently. If we would start migrating them to Java, we could move a
> > larger proportion of tests over to the new Java framework, and tackle the
> > more involved bash tests later (kerberized yarn, kubernetes, ...).
> >
> > Let me know what you think!
> >
> > Best,
> > Robert
> >
> >
> > PS: If you are wondering why I'm bringing this up now: I'm spending
> quite a
> > lot of time trying to figure out really hard to debug issues with our
> bash
> > testing infra.
> > Also, it is very difficult to introduce something generic for all tests
> > (such as a test-timeout, using docker as the preferred deployment method
> > etc.) since the tests often don't share common tooling.
> > Speaking about tooling: there are a lot of utilities everywhere,
> sometimes
> > duplicated, with different features / stability etc.
> > I believe bash is not the right tool for a project this size (in terms of
> > developers and lines of code)
> >
>
Re: [DISCUSS] Stop adding new bash-based e2e tests to Flink
Posted by Yang Wang <da...@gmail.com>.
Thanks for starting this discussion.
In general, I agree with you that a java-based testing framework is better
than the bash-based. It will
help a lot for the commons and utilities.
Since I am trying to add a new bash-based Kubernetes HA test, I have some
quick questions.
* I am not sure where the java-based framework is. Do you mean
"flink-jepsen" module or sth else?
* Maybe it will be harder to run a cli command(e.g. flink run /
run-application) to submit a Flink job in the java-based framework.
* It will be harder to inject some operations. For example, kill the
JobManager in Kubernetes. Currently, I
am trying to use "kubectl exec" to do this.
Best,
Yang
Robert Metzger <rm...@apache.org> 于2020年11月17日周二 下午11:36写道：
> Hi all,
>
> Since we are currently testing the 1.12 release, and potentially adding
> more automated e2e tests, I would like to bring up our end to end tests for
> discussion.
>
> Some time ago, we introduced a Java-based testing framework, with the idea
> of replacing the current bash-based end to end tests.
> Since the introduction of the java-based framework, more bash tests were
> actually added, making a future migration even harder.
>
> *For that reason, I would like to propose that we are stopping to add any
> new bash end to end tests to Flink. All new end to end tests must be
> written in Java and rely on the existing testing framework.*
>
> For the 1.13 release, I'm trying to find some time to revisit potential
> improvements for the existing java e2e framework (such as using Docker
> images everywhere), as well as a migration plan for the existing bash
> tests. We have a large number of bash e2e tests that are just parameterized
> differently. If we would start migrating them to Java, we could move a
> larger proportion of tests over to the new Java framework, and tackle the
> more involved bash tests later (kerberized yarn, kubernetes, ...).
>
> Let me know what you think!
>
> Best,
> Robert
>
>
> PS: If you are wondering why I'm bringing this up now: I'm spending quite a
> lot of time trying to figure out really hard to debug issues with our bash
> testing infra.
> Also, it is very difficult to introduce something generic for all tests
> (such as a test-timeout, using docker as the preferred deployment method
> etc.) since the tests often don't share common tooling.
> Speaking about tooling: there are a lot of utilities everywhere, sometimes
> duplicated, with different features / stability etc.
> I believe bash is not the right tool for a project this size (in terms of
> developers and lines of code)
>