You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by GitBox <gi...@apache.org> on 2023/01/08 19:51:28 UTC
[GitHub] [streampipes] bossenti created a discussion: Setup docker dev deployment on a daily basis
GitHub user bossenti created a discussion: Setup docker dev deployment on a daily basis
### Body
Currently, we have [GitHub workflow ](https://github.com/apache/streampipes/blob/dev/.github/workflows/build.yml) that builds new docker images for the current code base on every push for the `dev` branch and every `release` branch.
For release branches this should stay as it is, but for our `dev` branch we should switch to a less frequent deployment mode.
I'd suggest to go with a daily build executed e.g. at `00:00 UTC`.
### Mentoring
As this ticket is marked as good first issue: @dominikriemer, @tenthe or @bossenti are happy to provide help for getting started, just tag (one of) them if you want to start working on this issue and need some help.
This issue might be a good issue for people interested in GitHub workflows.
### StreamPipes Committer
I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.
GitHub link: https://github.com/apache/streampipes/discussions/1062
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
@bossenti in terms of ui we have three options. Currently we use the e2e test, but it requires the entire system to run. Also, Cypress has a new feature called component testing. This allows you to test components "unit." I have played with it a bit and it is actually very cool. Further, it is also possible to apply unit tests, those files are within each component directory.
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4671899
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
Hi @bossenti,
I think we can remove the Docker builds from the normal build pipeline and add a new pipeline to build the images once a day. I would suggest that we also add a manual trigger (`workflow_dispatch`) in case we need to build the dev images to test something.
Maybe we can also discuss what should be part of the regular build pipeline?
Especially because some of the steps already run during PR validation.
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4633875
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
Looks like we already have some ideas, how do we want to continue here?
Are there same easy steps we can directly write issues for or is there so more research and conceptual work to be done first?
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4748947
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
I've created a discussion of the above issue first.
From my point of view building the docker image for every commit is quite some overhead with our current development speed.
Therefore, I'd suggest to switch to a daily schedule for our dev branch instead.
Are there any negative side effects I don't have on my radar for now? @tenthe @dominikriemer
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4627290
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] dominikriemer added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Setup docker dev deployment on a daily basis
I'm ok with nightly builds and running the complete build process in nightly builds.
We could add a flag in the Maven configuration to skip the integration tests, as I think the default when running an mvn package on the project should be that all tests including integration tests are run.
@RobertIndie what do you think would be a best practice approach for running the builds and tests concerning the trade-off between build time and complete building?
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4653260
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
My suggestion would be to run the static code validations (e.g. license header checks, linting) and additionally the unit test on every commit. For PRs, we can additionally run the smoke e2e tests. During the nightly build, we can then run the integration test and all the e2e tests, and if everything was successful, we push the Docker snapshot containers to the registry.What do you think of this?
If you like the idea, we can discuss what commands need to be executed to perform the appropriate validations for the different parts (e.g. UI, backend, Python).
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4645568
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
@tenthe removing duplicated steps sounds great to me
From a first look, only the `maven build` seems to be duplicated to me
This could be removed, right?
But from my point of view this then requires some adaptions we are discussing in #882 to ensure that every commit runs once through the whole validation
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4637133
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user tenthe added a comment to the discussion: Setup docker dev deployment on a daily basis
I think we can start by creating issues and then defining the tasks directly in the issues. Maybe we can also use the issues to "collect" all the necessary commands.
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4758416
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
@tenthe I really like your suggestion and think this could be great outcome of this discussion.
However, I don't really know how to achieve this except talking about Python.
Having one eye on our test coverage: With regard to the UI, we only have e2e tests in place so far, right? Might it be of value to think about unit tests here as well? Probably @smlabt has some ideas about how to provide a good setup for this.
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4660573
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user bossenti added a comment to the discussion: Setup docker dev deployment on a daily basis
@tenthe agree, a manual trigger would be of value
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4637083
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org
[GitHub] [streampipes] dominikriemer added a comment to the discussion: Setup docker dev deployment on a daily basis
Posted by GitBox <gi...@apache.org>.
GitHub user dominikriemer added a comment to the discussion: Setup docker dev deployment on a daily basis
+1 for daily docker snapshot builds and manual trigger
GitHub link: https://github.com/apache/streampipes/discussions/1062#discussioncomment-4635592
----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org