Reorganizing Travis CI with build stages

[Misagh Moayyed <mi...@tirasa.net>]

Hello all,

In discussing git workflows with Francesco, I have learned that the Travis CI process is not actually running a comprehensive set of tests and checks (such as checkstyle, etc) that would be valuable for pull requests and contributors, possibly because such jobs take a long time, more than what is allowed by Travis CI. I'd like to propose to reorganize the CI build process to take advantage of the build stages feature [1]. This [2] demonstrates an example where a given build is broken down into multiple stages, each of which has numerous jobs.

A key thing to note in this configuration, as I understand, is that a given CI build can have an *unlimited* number of stages, each with an unlimited number of jobs. While each job cannot exceed a certain timeout value [3], the collective overall build stage can go on forever in theory. Also, stages do depend on each other such that failures in one would halt the build. A typical example would be: "Run the build, and if everything compiles, run tests, then publish snapshots, etc".

I think this is quite valuable for contributors who, specially, work on multiple JIRAs at the same time. A sample workflow is to add a change, do some sanity tests and then push to let CI run the full build suite. While it's busy kicking off the build, one could switch to the next JIRA to repeat the same process, and as such hop back and forth between branches and JIRAs to parallelize work activity. Ultimately, the CI is the canonical reference for all changes and this would avoid the problem of "but it works on my machine" and of course, saves quite a bit of time. (A full build for me takes about 20-30 minutes, if the stars align).

How do folks feel about this proposal?

[1] https://docs.travis-ci.com/user/build-stages/
[2] https://travis-ci.org/apereo/cas/builds/581042662
[3] The good folks at Travis CI in the past have been kind enough to increase the timeout from 50 minutes to 180 minutes and they might be willing to make the same change for syncope. Larger tasks need to be broken down to separate jobs, of course.

--Misagh

Re: Reorganizing Travis CI with build stages

[Andrea Patricelli <an...@apache.org>]

Il 09/09/19 16:07, Misagh Moayyed ha scritto:
>> thanks for bringing this item to discussion: improving our CI power is
>> definitely more than welcome.
>>
>> I would say there are several requirements one might want to handle:
> All sounds excellent. Once the proper structure is in, it would be quite easy to add jobs to cover all additional test cases. I'll need some help to figure out how to run the IT tests for different systems and specifically separate them per job to minimize time, but those would be trivial to include and massage once the basics start working.
>
> ...and to answer Andrea's question: yes absolutely! integrations tests can totally be added for all sorts of systems. The IT tests that CAS runs mostly pull down docker images and run tests against predefined addresses and ports to connect to an external system. Syncope can more or less follow the same approach, or we could also take advantage of the test-containers project to simplify the setup; that would be a good experiment. (No such thing as test-containers existed at the time CAS started doing this)
>   
Great! Basically I meant points 3, 4, 5 of Francesco's list. Is quite 
simple to run on different containers or DBMSes, it is all summarized at 
[1]. BTW for any help doing this job don't hesitate to ask.
>> I am not sure about [3] - we should ask ASF Infra to issue such a
>> request, in case, as they are the actual owner of the GitHub repo.
> That's totally possible and in fact, it might simpler than that. One could just send an email to Travis CI, asking them to increase the timeout. They are super quick to respond, and all related changes are done on their end; nothing to modify as far as the GH repo goes for Syncope. Still, ASF Infra might be better to submit that email...and we might not even need it, if individual jobs can run under the current 50-minute timeout.
>
> I'll create a JIRA to track this with possible sub-tasks and will get to work. This will take a number of tries, so apologies beforehand for the noise.
[1] http://syncope.apache.org/building#Debug

-- 
Dott. Andrea Patricelli
Tel. +39 3204524292

Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net

Apache Syncope PMC Member

Re: Reorganizing Travis CI with build stages

[Misagh Moayyed <mi...@tirasa.net>]

> 
> Thanks for this Misagh.
> 
> Maybe, in order to minimize the "noise" and possibly to allow you to run
> faster, what do you think of experimenting with Travis CI on your own
> fork? You'd only have to change the notification address as throw-away
> change.

Yup, sounds like a plan!

Parent JIRA: https://issues.apache.org/jira/browse/SYNCOPE-1489

Re: Reorganizing Travis CI with build stages

[Francesco Chicchiriccò <il...@apache.org>]

On 09/09/19 16:07, Misagh Moayyed wrote:
>> thanks for bringing this item to discussion: improving our CI power is
>> definitely more than welcome.
>>
>> I would say there are several requirements one might want to handle:
> All sounds excellent. Once the proper structure is in, it would be quite easy to add jobs to cover all additional test cases. I'll need some help to figure out how to run the IT tests for different systems and specifically separate them per job to minimize time, but those would be trivial to include and massage once the basics start working.
>
> ...and to answer Andrea's question: yes absolutely! integrations tests can totally be added for all sorts of systems. The IT tests that CAS runs mostly pull down docker images and run tests against predefined addresses and ports to connect to an external system. Syncope can more or less follow the same approach, or we could also take advantage of the test-containers project to simplify the setup; that would be a good experiment. (No such thing as test-containers existed at the time CAS started doing this)
>  
>> I am not sure about [3] - we should ask ASF Infra to issue such a
>> request, in case, as they are the actual owner of the GitHub repo.
> That's totally possible and in fact, it might simpler than that. One could just send an email to Travis CI, asking them to increase the timeout. They are super quick to respond, and all related changes are done on their end; nothing to modify as far as the GH repo goes for Syncope. Still, ASF Infra might be better to submit that email...and we might not even need it, if individual jobs can run under the current 50-minute timeout.
>
> I'll create a JIRA to track this with possible sub-tasks and will get to work. This will take a number of tries, so apologies beforehand for the noise. 

Thanks for this Misagh.

Maybe, in order to minimize the "noise" and possibly to allow you to run
faster, what do you think of experimenting with Travis CI on your own
fork? You'd only have to change the notification address as throw-away
change.

WDYT?

-- 
Francesco Chicchiriccò

Tirasa - Open Source Excellence
http://www.tirasa.net/

Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/

Re: Reorganizing Travis CI with build stages

[Misagh Moayyed <mi...@tirasa.net>]

> thanks for bringing this item to discussion: improving our CI power is
> definitely more than welcome.
> 
> I would say there are several requirements one might want to handle:

All sounds excellent. Once the proper structure is in, it would be quite easy to add jobs to cover all additional test cases. I'll need some help to figure out how to run the IT tests for different systems and specifically separate them per job to minimize time, but those would be trivial to include and massage once the basics start working.

...and to answer Andrea's question: yes absolutely! integrations tests can totally be added for all sorts of systems. The IT tests that CAS runs mostly pull down docker images and run tests against predefined addresses and ports to connect to an external system. Syncope can more or less follow the same approach, or we could also take advantage of the test-containers project to simplify the setup; that would be a good experiment. (No such thing as test-containers existed at the time CAS started doing this)
 
> I am not sure about [3] - we should ask ASF Infra to issue such a
> request, in case, as they are the actual owner of the GitHub repo.

That's totally possible and in fact, it might simpler than that. One could just send an email to Travis CI, asking them to increase the timeout. They are super quick to respond, and all related changes are done on their end; nothing to modify as far as the GH repo goes for Syncope. Still, ASF Infra might be better to submit that email...and we might not even need it, if individual jobs can run under the current 50-minute timeout.

I'll create a JIRA to track this with possible sub-tasks and will get to work. This will take a number of tries, so apologies beforehand for the noise. 

Re: Reorganizing Travis CI with build stages

[Francesco Chicchiriccò <il...@apache.org>]

On 09/09/19 11:37, Misagh Moayyed wrote:
> Hello all,
>
> In discussing git workflows with Francesco, I have learned that the Travis CI process is not actually running a comprehensive set of tests and checks (such as checkstyle, etc) that would be valuable for pull requests and contributors, possibly because such jobs take a long time, more than what is allowed by Travis CI. I'd like to propose to reorganize the CI build process to take advantage of the build stages feature [1]. This [2] demonstrates an example where a given build is broken down into multiple stages, each of which has numerous jobs.
>
> A key thing to note in this configuration, as I understand, is that a given CI build can have an *unlimited* number of stages, each with an unlimited number of jobs. While each job cannot exceed a certain timeout value [3], the collective overall build stage can go on forever in theory. Also, stages do depend on each other such that failures in one would halt the build. A typical example would be: "Run the build, and if everything compiles, run tests, then publish snapshots, etc".
>
> I think this is quite valuable for contributors who, specially, work on multiple JIRAs at the same time. A sample workflow is to add a change, do some sanity tests and then push to let CI run the full build suite. While it's busy kicking off the build, one could switch to the next JIRA to repeat the same process, and as such hop back and forth between branches and JIRAs to parallelize work activity. Ultimately, the CI is the canonical reference for all changes and this would avoid the problem of "but it works on my machine" and of course, saves quite a bit of time. (A full build for me takes about 20-30 minutes, if the stars align).
>
> How do folks feel about this proposal?

Hi Misagh,
thanks for bringing this item to discussion: improving our CI power is
definitely more than welcome.

I would say there are several requirements one might want to handle:

1. run the straight "mvn clean install" one normally performs on his /
her laptop, which involves RAT checks, Checkstyle, modernizer, etc, all
unit and integration tests

1a. run integration tests with different content types [5]

2. run the Admin UI tests [4], currently not run by default under
fit/core-reference

3. run the Enduser UI tests ([4] for master), currently not run by
default under fit/core-reference

4. run integration tests under the other supported Java EE containers
(besides Tomcat) [6]

5. run integration tests with the other supported DBMSes (besides H2) [7]

6. run integration tests with Elasticsearch - not documented but available

7. run SAML 2.0 / OpenID Connect 1.0 login tests - not documented but
available (only master)

I am not saying we should cover all above in one shot, just reporting
the full requirements; I would be happy even with only (1), a great
progress anyway compared to the current situation.

I am not sure about [3] - we should ask ASF Infra to issue such a
request, in case, as they are the actual owner of the GitHub repo.

Regards.

> [1] https://docs.travis-ci.com/user/build-stages/
> [2] https://travis-ci.org/apereo/cas/builds/581042662
> [3] The good folks at Travis CI in the past have been kind enough to increase the timeout from 50 minutes to 180 minutes and they might be willing to make the same change for syncope. Larger tasks need to be broken down to separate jobs, of course.
> [4] http://syncope.apache.org/building#All_integration_tests
> [5] http://syncope.apache.org/building#Other_Content-Type_values
> [6] http://syncope.apache.org/building#Java_EE_containers
> [7] http://syncope.apache.org/building#DBMSes

-- 
Francesco Chicchiriccò

Tirasa - Open Source Excellence
http://www.tirasa.net/

Member at The Apache Software Foundation
Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail
http://home.apache.org/~ilgrosso/

Re: Reorganizing Travis CI with build stages

[Matteo Alessandroni <sk...@apache.org>]

Hi Misagh,

It sounds good to me, so +1!
Thank you!

Best regards,
Matteo

On 09/09/19 11:37, Misagh Moayyed wrote:
> Hello all,
>
> In discussing git workflows with Francesco, I have learned that the Travis CI process is not actually running a comprehensive set of tests and checks (such as checkstyle, etc) that would be valuable for pull requests and contributors, possibly because such jobs take a long time, more than what is allowed by Travis CI. I'd like to propose to reorganize the CI build process to take advantage of the build stages feature [1]. This [2] demonstrates an example where a given build is broken down into multiple stages, each of which has numerous jobs.
>
> A key thing to note in this configuration, as I understand, is that a given CI build can have an *unlimited* number of stages, each with an unlimited number of jobs. While each job cannot exceed a certain timeout value [3], the collective overall build stage can go on forever in theory. Also, stages do depend on each other such that failures in one would halt the build. A typical example would be: "Run the build, and if everything compiles, run tests, then publish snapshots, etc".
>
> I think this is quite valuable for contributors who, specially, work on multiple JIRAs at the same time. A sample workflow is to add a change, do some sanity tests and then push to let CI run the full build suite. While it's busy kicking off the build, one could switch to the next JIRA to repeat the same process, and as such hop back and forth between branches and JIRAs to parallelize work activity. Ultimately, the CI is the canonical reference for all changes and this would avoid the problem of "but it works on my machine" and of course, saves quite a bit of time. (A full build for me takes about 20-30 minutes, if the stars align).
>
> How do folks feel about this proposal?
>
> [1] https://docs.travis-ci.com/user/build-stages/
> [2] https://travis-ci.org/apereo/cas/builds/581042662
> [3] The good folks at Travis CI in the past have been kind enough to increase the timeout from 50 minutes to 180 minutes and they might be willing to make the same change for syncope. Larger tasks need to be broken down to separate jobs, of course.
>
> --Misagh

Re: Reorganizing Travis CI with build stages

[Andrea Patricelli <an...@apache.org>]

Hi Misagh,

+1 for me!

I'm not so expert of Travis, but this proposal sounds very good to me, 
mostly for the possibility to run all tests and organize better build 
sections. Several times I meet with "but works on my machine" problem :)

With the help of jobs, I would also enable console and enduser tests, 
maybe in two separate jobs. Moreover, like I'm seeing is done for CAS, 
we can enable other it tests, e.g. mysql-it, elasticsearch-it, etc. WDYT?

Best regards,
Andrea

Il 09/09/19 11:37, Misagh Moayyed ha scritto:
> Hello all,
>
> In discussing git workflows with Francesco, I have learned that the Travis CI process is not actually running a comprehensive set of tests and checks (such as checkstyle, etc) that would be valuable for pull requests and contributors, possibly because such jobs take a long time, more than what is allowed by Travis CI. I'd like to propose to reorganize the CI build process to take advantage of the build stages feature [1]. This [2] demonstrates an example where a given build is broken down into multiple stages, each of which has numerous jobs.
>
> A key thing to note in this configuration, as I understand, is that a given CI build can have an *unlimited* number of stages, each with an unlimited number of jobs. While each job cannot exceed a certain timeout value [3], the collective overall build stage can go on forever in theory. Also, stages do depend on each other such that failures in one would halt the build. A typical example would be: "Run the build, and if everything compiles, run tests, then publish snapshots, etc".
>
> I think this is quite valuable for contributors who, specially, work on multiple JIRAs at the same time. A sample workflow is to add a change, do some sanity tests and then push to let CI run the full build suite. While it's busy kicking off the build, one could switch to the next JIRA to repeat the same process, and as such hop back and forth between branches and JIRAs to parallelize work activity. Ultimately, the CI is the canonical reference for all changes and this would avoid the problem of "but it works on my machine" and of course, saves quite a bit of time. (A full build for me takes about 20-30 minutes, if the stars align).
>
> How do folks feel about this proposal?
>
> [1] https://docs.travis-ci.com/user/build-stages/
> [2] https://travis-ci.org/apereo/cas/builds/581042662
> [3] The good folks at Travis CI in the past have been kind enough to increase the timeout from 50 minutes to 180 minutes and they might be willing to make the same change for syncope. Larger tasks need to be broken down to separate jobs, of course.
>
> --Misagh

-- 
Dott. Andrea Patricelli
Tel. +39 3204524292

Engineer @ Tirasa S.r.l.
Viale Vittoria Colonna 97 - 65127 Pescara
Tel +39 0859116307 / FAX +39 0859111173
http://www.tirasa.net

Apache Syncope PMC Member