You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by sblackmon <sb...@apache.org> on 2016/10/17 20:42:03 UTC
Update to continuous integration flow
Hello,
I’m working on the streams jenkins setup to increase our CI coverage and support more types of testing.
What’s changed code wise to precipitate changes?
- streams-master and streams-examples are now release artifacts and need CI
- some providers now contain integration tests that collect data live, but require working api credentials to do so
- some persisters (and all examples) now contain integration tests that launch supporting services with docker
- streams-project now contains modules that build maven plugins, and has testing that relies on those plugins
What’s changing?
- integration testing (mvn verify) is being isolated from standard build (mvn install)
- integration testing steps will be configured to launch docker containers and use test credentials to run the live data tests
- snapshots and site snapshots will publish as separate steps, contingent on stability of standard builds
Attached a diagram of the new build flow, which will be including on the continuous integration page of the website.
Dashed line: checkout
Dotted line: jenkins upstream/downstream relationships
Solid line: deployment of artifacts
Hint: you can click each node in the SVG to jump to a relevant URL.
I’m working on setting this up and hope to have everything fully working before the 0.4 release. In the meantime you may see build failures as jobs and test plans are being refactored in STREAMS-429 branches.
Steve
Re: Update to continuous integration flow
Posted by Suneel Marthi <su...@gmail.com>.
Sent from my iPhone
> On Oct 20, 2016, at 11:34 PM, sblackmon <sb...@apache.org> wrote:
>
> Jenkins jobs streams-project-verify and streams-examples-verify are passing as well now, including all of the new live data collection, docker database, and flink example integration tests.
U might want to reach out to the flink folks to add streams project to their 'Powered by Flink' page.
>
> streams-project-verify is a bit particular only because the facebook credentials it’s been given have a very quick expiration. So until we source more stable credentials to use for testing, a new token will need to be provider prior to any execution where we’re looking for 100% integration test pass rate - probably just builds where something has changed in that module or built from a release candidate.
>
> streams-project-site and streams-examples-site have been configured to run on SCM updates, but at the moment they are not setup to auto-deploy to the website - that’s still a documented manual process. You can however click to view the staging site in jenkins.
>
> Steve
> On October 18, 2016 at 10:58:07 PM, sblackmon (sblackmon@apache.org) wrote:
>
> Update:
>
> Jenkins jobs streams-master streams-project and streams-examples are blue with no test failures.
>
> Jenkins jobs streams-project-verify and streams-examples-verify have been created and configured.
>
> I’ve observed them successfully launching and reading/writing to docker containers, as well as collecting data from twitter using credentials supplied by the job.
>
> After merging the open PRs and a bit more tinkering in Jenkins, I expect those jobs to be blue as well with all ITs passing project-wide.
>
> Steve
>
> On October 17, 2016 at 3:42:04 PM, sblackmon (sblackmon@apache.org) wrote:
>
> Hello,
>
> I’m working on the streams jenkins setup to increase our CI coverage and support more types of testing.
>
> What’s changed code wise to precipitate changes?
>
> - streams-master and streams-examples are now release artifacts and need CI
> - some providers now contain integration tests that collect data live, but require working api credentials to do so
> - some persisters (and all examples) now contain integration tests that launch supporting services with docker
> - streams-project now contains modules that build maven plugins, and has testing that relies on those plugins
>
> What’s changing?
>
> - integration testing (mvn verify) is being isolated from standard build (mvn install)
> - integration testing steps will be configured to launch docker containers and use test credentials to run the live data tests
> - snapshots and site snapshots will publish as separate steps, contingent on stability of standard builds
>
> Attached a diagram of the new build flow, which will be including on the continuous integration page of the website.
>
> Dashed line: checkout
> Dotted line: jenkins upstream/downstream relationships
> Solid line: deployment of artifacts
>
> Hint: you can click each node in the SVG to jump to a relevant URL.
>
> I’m working on setting this up and hope to have everything fully working before the 0.4 release. In the meantime you may see build failures as jobs and test plans are being refactored in STREAMS-429 branches.
>
> Steve
>
>
Re: Update to continuous integration flow
Posted by sblackmon <sb...@apache.org>.
Jenkins jobs streams-project-verify and streams-examples-verify are passing as well now, including all of the new live data collection, docker database, and flink example integration tests.
streams-project-verify is a bit particular only because the facebook credentials it’s been given have a very quick expiration. So until we source more stable credentials to use for testing, a new token will need to be provider prior to any execution where we’re looking for 100% integration test pass rate - probably just builds where something has changed in that module or built from a release candidate.
streams-project-site and streams-examples-site have been configured to run on SCM updates, but at the moment they are not setup to auto-deploy to the website - that’s still a documented manual process. You can however click to view the staging site in jenkins.
Steve
On October 18, 2016 at 10:58:07 PM, sblackmon (sblackmon@apache.org) wrote:
Update:
Jenkins jobs streams-master streams-project and streams-examples are blue with no test failures.
Jenkins jobs streams-project-verify and streams-examples-verify have been created and configured.
I’ve observed them successfully launching and reading/writing to docker containers, as well as collecting data from twitter using credentials supplied by the job.
After merging the open PRs and a bit more tinkering in Jenkins, I expect those jobs to be blue as well with all ITs passing project-wide.
Steve
On October 17, 2016 at 3:42:04 PM, sblackmon (sblackmon@apache.org) wrote:
Hello,
I’m working on the streams jenkins setup to increase our CI coverage and support more types of testing.
What’s changed code wise to precipitate changes?
- streams-master and streams-examples are now release artifacts and need CI
- some providers now contain integration tests that collect data live, but require working api credentials to do so
- some persisters (and all examples) now contain integration tests that launch supporting services with docker
- streams-project now contains modules that build maven plugins, and has testing that relies on those plugins
What’s changing?
- integration testing (mvn verify) is being isolated from standard build (mvn install)
- integration testing steps will be configured to launch docker containers and use test credentials to run the live data tests
- snapshots and site snapshots will publish as separate steps, contingent on stability of standard builds
Attached a diagram of the new build flow, which will be including on the continuous integration page of the website.
Dashed line: checkout
Dotted line: jenkins upstream/downstream relationships
Solid line: deployment of artifacts
Hint: you can click each node in the SVG to jump to a relevant URL.
I’m working on setting this up and hope to have everything fully working before the 0.4 release. In the meantime you may see build failures as jobs and test plans are being refactored in STREAMS-429 branches.
Steve
Re: Update to continuous integration flow
Posted by sblackmon <sb...@apache.org>.
Update:
Jenkins jobs streams-master streams-project and streams-examples are blue with no test failures.
Jenkins jobs streams-project-verify and streams-examples-verify have been created and configured.
I’ve observed them successfully launching and reading/writing to docker containers, as well as collecting data from twitter using credentials supplied by the job.
After merging the open PRs and a bit more tinkering in Jenkins, I expect those jobs to be blue as well with all ITs passing project-wide.
Steve
On October 17, 2016 at 3:42:04 PM, sblackmon (sblackmon@apache.org) wrote:
Hello,
I’m working on the streams jenkins setup to increase our CI coverage and support more types of testing.
What’s changed code wise to precipitate changes?
- streams-master and streams-examples are now release artifacts and need CI
- some providers now contain integration tests that collect data live, but require working api credentials to do so
- some persisters (and all examples) now contain integration tests that launch supporting services with docker
- streams-project now contains modules that build maven plugins, and has testing that relies on those plugins
What’s changing?
- integration testing (mvn verify) is being isolated from standard build (mvn install)
- integration testing steps will be configured to launch docker containers and use test credentials to run the live data tests
- snapshots and site snapshots will publish as separate steps, contingent on stability of standard builds
Attached a diagram of the new build flow, which will be including on the continuous integration page of the website.
Dashed line: checkout
Dotted line: jenkins upstream/downstream relationships
Solid line: deployment of artifacts
Hint: you can click each node in the SVG to jump to a relevant URL.
I’m working on setting this up and hope to have everything fully working before the 0.4 release. In the meantime you may see build failures as jobs and test plans are being refactored in STREAMS-429 branches.
Steve