You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Patrick Wiener <wi...@apache.org> on 2020/11/06 13:55:59 UTC
CI docker builds & minor repo structure refactoring
Hi guys,
i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
- I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
- We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
- We could rename ui/ -> streampipes-ui/
What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
Patrick
Re: CI docker builds & minor repo structure refactoring
Posted by Patrick Wiener <wi...@apache.org>.
I actually found it harder in the beginning getting an overview. The reason was because my entrypoint was the project installer (docker-compose file) where I saw the listed SP images. Then going back to the repository itself I had to navigate through the various modules to find the ones that contained the relevant Dockerfile.
By having them in a top-level folder I think it would have made it a little bit easier for me to keep an overview. Additionally, the naming schemes don’t match the module names in all cases, e.g. module: streampipes-connect-container-master — docker: connect-master. Same with connect adapters in extensions.
Additionally, since e.g. the build for streampipes-backend/ that produces the output jar that is then integrated in the Dockerfile relies on multiple other modules, wouldn’t we just run a 'mvn package‘ from the root folder anyways?
These were just my observations. I’m happy to further discuss on that topic to get the best solution for CI, dev and onboarding new users.
Patrick
> Am 10.11.2020 um 08:05 schrieb Dominik Riemer <ri...@apache.org>:
>
> Hi,
>
> I personally like having a Dockerfile right next to the build file as you can perform build + docker build from a single directory. Isn't that more intuitive for users who clone the repo without reading any documentation?
>
> Concerning renaming of the UI, I think that renaming it to streampipes-ui might make it harder to find in the list of modules. We could also think of renaming it to something like "JS", as we will probably not only have the UI, but also some JS core modules and an SDK/wrapper for JS in the near future as well.
>
> Dominik
>
> On 2020/11/09 15:46:14, Patrick Wiener <wi...@fzi.de> wrote:
>> Well, not only for the CI. For the maintainability of the project, in my eyes it is more clear to make changes and get and overview of the Dockerfiles by having them in a central location. In addition, I think of having a build script for building the docker images and one for pushing. So we can also build the images locally for testing by using the same script.
>>
>> Patrick
>>
>>> Am 09.11.2020 um 12:54 schrieb Philipp Zehnder <ze...@apache.org>:
>>>
>>> Ok, I personally prefer it when all the files regarding one package are located within that package.
>>> But when you say that it is much easier to manage the CI pipelines, I am not against it.
>>>
>>> The only thing I would like is if the base image in the docker file and the base image used in the build is the same.
>>> This has caused me some confusion in the past, because I think this was overwritten in the build and I didn’t know that.
>>>
>>>
>>> Philipp
>>>
>>>> On 6. Nov 2020, at 16:03, Patrick Wiener <wi...@apache.org> wrote:
>>>>
>>>> Ok - I’ll start another discussion in a separate thread.
>>>>
>>>> Regarding the Dockerfile: yepp - this works! you simply have to adjust the build context, e.g. the following command executed from the root dir of the project with a Dockerfile in docker/Dockerfile.backend:
>>>>
>>>> docker build -t apachestreampipes/backend:text -f docker/Dockerfile.backend streampipes-backend/
>>>>
>>>> Patrick
>>>>
>>>>
>>>>> Am 06.11.2020 um 15:30 schrieb Philipp Zehnder <ze...@apache.org>:
>>>>>
>>>>> Yes we can discuss this in a separate thread.
>>>>>
>>>>> I was wondering if this change would also effect the location of the ui folder. Currently this folder is within the backend project, but I think in the new structure the ui could be a top level folder.
>>>>>
>>>>> Regarding the docker files. I am not sure if we can move them, because each Dockerfile needs all the build artifacts.
>>>>> Would this mapping also work, when the docker files are in a different location?
>>>>>
>>>>> Philipp
>>>>>
>>>>>
>>>>>> On 6. Nov 2020, at 15:16, Patrick Wiener <wi...@apache.org> wrote:
>>>>>>
>>>>>> This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
>>>>>> From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
>>>>>>
>>>>>> Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
>>>>>>
>>>>>> Patrick
>>>>>>
>>>>>>> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
>>>>>>>
>>>>>>> Hi Patrick,
>>>>>>>
>>>>>>> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
>>>>>>> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
>>>>>>>
>>>>>>> Whats your opinion on that?
>>>>>>>
>>>>>>> Philipp
>>>>>>>
>>>>>>>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Hi guys,
>>>>>>>>
>>>>>>>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>>>>>>>>
>>>>>>>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
>>>>>>>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
>>>>>>>> - We could rename ui/ -> streampipes-ui/
>>>>>>>>
>>>>>>>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>>>>>>>>
>>>>>>>> Patrick
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
Re: CI docker builds & minor repo structure refactoring
Posted by Dominik Riemer <ri...@apache.org>.
Hi,
I personally like having a Dockerfile right next to the build file as you can perform build + docker build from a single directory. Isn't that more intuitive for users who clone the repo without reading any documentation?
Concerning renaming of the UI, I think that renaming it to streampipes-ui might make it harder to find in the list of modules. We could also think of renaming it to something like "JS", as we will probably not only have the UI, but also some JS core modules and an SDK/wrapper for JS in the near future as well.
Dominik
On 2020/11/09 15:46:14, Patrick Wiener <wi...@fzi.de> wrote:
> Well, not only for the CI. For the maintainability of the project, in my eyes it is more clear to make changes and get and overview of the Dockerfiles by having them in a central location. In addition, I think of having a build script for building the docker images and one for pushing. So we can also build the images locally for testing by using the same script.
>
> Patrick
>
> > Am 09.11.2020 um 12:54 schrieb Philipp Zehnder <ze...@apache.org>:
> >
> > Ok, I personally prefer it when all the files regarding one package are located within that package.
> > But when you say that it is much easier to manage the CI pipelines, I am not against it.
> >
> > The only thing I would like is if the base image in the docker file and the base image used in the build is the same.
> > This has caused me some confusion in the past, because I think this was overwritten in the build and I didn’t know that.
> >
> >
> > Philipp
> >
> >> On 6. Nov 2020, at 16:03, Patrick Wiener <wi...@apache.org> wrote:
> >>
> >> Ok - I’ll start another discussion in a separate thread.
> >>
> >> Regarding the Dockerfile: yepp - this works! you simply have to adjust the build context, e.g. the following command executed from the root dir of the project with a Dockerfile in docker/Dockerfile.backend:
> >>
> >> docker build -t apachestreampipes/backend:text -f docker/Dockerfile.backend streampipes-backend/
> >>
> >> Patrick
> >>
> >>
> >>> Am 06.11.2020 um 15:30 schrieb Philipp Zehnder <ze...@apache.org>:
> >>>
> >>> Yes we can discuss this in a separate thread.
> >>>
> >>> I was wondering if this change would also effect the location of the ui folder. Currently this folder is within the backend project, but I think in the new structure the ui could be a top level folder.
> >>>
> >>> Regarding the docker files. I am not sure if we can move them, because each Dockerfile needs all the build artifacts.
> >>> Would this mapping also work, when the docker files are in a different location?
> >>>
> >>> Philipp
> >>>
> >>>
> >>>> On 6. Nov 2020, at 15:16, Patrick Wiener <wi...@apache.org> wrote:
> >>>>
> >>>> This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
> >>>> From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
> >>>>
> >>>> Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
> >>>>
> >>>> Patrick
> >>>>
> >>>>> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
> >>>>>
> >>>>> Hi Patrick,
> >>>>>
> >>>>> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
> >>>>> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
> >>>>>
> >>>>> Whats your opinion on that?
> >>>>>
> >>>>> Philipp
> >>>>>
> >>>>>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
> >>>>>>
> >>>>>> Hi guys,
> >>>>>>
> >>>>>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
> >>>>>>
> >>>>>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
> >>>>>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
> >>>>>> - We could rename ui/ -> streampipes-ui/
> >>>>>>
> >>>>>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
> >>>>>>
> >>>>>> Patrick
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
Re: CI docker builds & minor repo structure refactoring
Posted by Patrick Wiener <wi...@fzi.de>.
Well, not only for the CI. For the maintainability of the project, in my eyes it is more clear to make changes and get and overview of the Dockerfiles by having them in a central location. In addition, I think of having a build script for building the docker images and one for pushing. So we can also build the images locally for testing by using the same script.
Patrick
> Am 09.11.2020 um 12:54 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Ok, I personally prefer it when all the files regarding one package are located within that package.
> But when you say that it is much easier to manage the CI pipelines, I am not against it.
>
> The only thing I would like is if the base image in the docker file and the base image used in the build is the same.
> This has caused me some confusion in the past, because I think this was overwritten in the build and I didn’t know that.
>
>
> Philipp
>
>> On 6. Nov 2020, at 16:03, Patrick Wiener <wi...@apache.org> wrote:
>>
>> Ok - I’ll start another discussion in a separate thread.
>>
>> Regarding the Dockerfile: yepp - this works! you simply have to adjust the build context, e.g. the following command executed from the root dir of the project with a Dockerfile in docker/Dockerfile.backend:
>>
>> docker build -t apachestreampipes/backend:text -f docker/Dockerfile.backend streampipes-backend/
>>
>> Patrick
>>
>>
>>> Am 06.11.2020 um 15:30 schrieb Philipp Zehnder <ze...@apache.org>:
>>>
>>> Yes we can discuss this in a separate thread.
>>>
>>> I was wondering if this change would also effect the location of the ui folder. Currently this folder is within the backend project, but I think in the new structure the ui could be a top level folder.
>>>
>>> Regarding the docker files. I am not sure if we can move them, because each Dockerfile needs all the build artifacts.
>>> Would this mapping also work, when the docker files are in a different location?
>>>
>>> Philipp
>>>
>>>
>>>> On 6. Nov 2020, at 15:16, Patrick Wiener <wi...@apache.org> wrote:
>>>>
>>>> This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
>>>> From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
>>>>
>>>> Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
>>>>
>>>> Patrick
>>>>
>>>>> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
>>>>>
>>>>> Hi Patrick,
>>>>>
>>>>> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
>>>>> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
>>>>>
>>>>> Whats your opinion on that?
>>>>>
>>>>> Philipp
>>>>>
>>>>>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>>>>>>
>>>>>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
>>>>>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
>>>>>> - We could rename ui/ -> streampipes-ui/
>>>>>>
>>>>>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>>>>>>
>>>>>> Patrick
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: CI docker builds & minor repo structure refactoring
Posted by Philipp Zehnder <ze...@apache.org>.
Ok, I personally prefer it when all the files regarding one package are located within that package.
But when you say that it is much easier to manage the CI pipelines, I am not against it.
The only thing I would like is if the base image in the docker file and the base image used in the build is the same.
This has caused me some confusion in the past, because I think this was overwritten in the build and I didn’t know that.
Philipp
> On 6. Nov 2020, at 16:03, Patrick Wiener <wi...@apache.org> wrote:
>
> Ok - I’ll start another discussion in a separate thread.
>
> Regarding the Dockerfile: yepp - this works! you simply have to adjust the build context, e.g. the following command executed from the root dir of the project with a Dockerfile in docker/Dockerfile.backend:
>
> docker build -t apachestreampipes/backend:text -f docker/Dockerfile.backend streampipes-backend/
>
> Patrick
>
>
>> Am 06.11.2020 um 15:30 schrieb Philipp Zehnder <ze...@apache.org>:
>>
>> Yes we can discuss this in a separate thread.
>>
>> I was wondering if this change would also effect the location of the ui folder. Currently this folder is within the backend project, but I think in the new structure the ui could be a top level folder.
>>
>> Regarding the docker files. I am not sure if we can move them, because each Dockerfile needs all the build artifacts.
>> Would this mapping also work, when the docker files are in a different location?
>>
>> Philipp
>>
>>
>>> On 6. Nov 2020, at 15:16, Patrick Wiener <wi...@apache.org> wrote:
>>>
>>> This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
>>> From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
>>>
>>> Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
>>>
>>> Patrick
>>>
>>>> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
>>>>
>>>> Hi Patrick,
>>>>
>>>> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
>>>> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
>>>>
>>>> Whats your opinion on that?
>>>>
>>>> Philipp
>>>>
>>>>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>>>>>
>>>>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
>>>>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
>>>>> - We could rename ui/ -> streampipes-ui/
>>>>>
>>>>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>>>>>
>>>>> Patrick
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: CI docker builds & minor repo structure refactoring
Posted by Patrick Wiener <wi...@apache.org>.
Ok - I’ll start another discussion in a separate thread.
Regarding the Dockerfile: yepp - this works! you simply have to adjust the build context, e.g. the following command executed from the root dir of the project with a Dockerfile in docker/Dockerfile.backend:
docker build -t apachestreampipes/backend:text -f docker/Dockerfile.backend streampipes-backend/
Patrick
> Am 06.11.2020 um 15:30 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Yes we can discuss this in a separate thread.
>
> I was wondering if this change would also effect the location of the ui folder. Currently this folder is within the backend project, but I think in the new structure the ui could be a top level folder.
>
> Regarding the docker files. I am not sure if we can move them, because each Dockerfile needs all the build artifacts.
> Would this mapping also work, when the docker files are in a different location?
>
> Philipp
>
>
>> On 6. Nov 2020, at 15:16, Patrick Wiener <wi...@apache.org> wrote:
>>
>> This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
>> From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
>>
>> Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
>>
>> Patrick
>>
>>> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
>>>
>>> Hi Patrick,
>>>
>>> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
>>> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
>>>
>>> Whats your opinion on that?
>>>
>>> Philipp
>>>
>>>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>>>>
>>>> Hi guys,
>>>>
>>>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>>>>
>>>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
>>>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
>>>> - We could rename ui/ -> streampipes-ui/
>>>>
>>>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>>>>
>>>> Patrick
>>>>
>>>>
>>>
>>
>
Re: CI docker builds & minor repo structure refactoring
Posted by Philipp Zehnder <ze...@apache.org>.
Yes we can discuss this in a separate thread.
I was wondering if this change would also effect the location of the ui folder. Currently this folder is within the backend project, but I think in the new structure the ui could be a top level folder.
Regarding the docker files. I am not sure if we can move them, because each Dockerfile needs all the build artifacts.
Would this mapping also work, when the docker files are in a different location?
Philipp
> On 6. Nov 2020, at 15:16, Patrick Wiener <wi...@apache.org> wrote:
>
> This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
> From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
>
> Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
>
> Patrick
>
>> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
>>
>> Hi Patrick,
>>
>> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
>> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
>>
>> Whats your opinion on that?
>>
>> Philipp
>>
>>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>>>
>>> Hi guys,
>>>
>>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>>>
>>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
>>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
>>> - We could rename ui/ -> streampipes-ui/
>>>
>>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>>>
>>> Patrick
>>>
>>>
>>
>
Re: CI docker builds & minor repo structure refactoring
Posted by Patrick Wiener <wi...@apache.org>.
This would definitely have a major effect on it. I like the idea of merging extensions + core together into one holistic streampipes project.
From the CI point of view however, this also increase build times but I argue it’s ok to reduce the setup, development complexity in conjunction with easier releasing.
Should we discuss this in a separate thread because the adaptations here would only be a subset of things to do when merging the two repos?
Patrick
> Am 06.11.2020 um 15:07 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Hi Patrick,
>
> during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
> I was wondering if we should do this first because it will also have a big effect on the CI job, right?
>
> Whats your opinion on that?
>
> Philipp
>
>> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>>
>> Hi guys,
>>
>> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>>
>> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
>> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
>> - We could rename ui/ -> streampipes-ui/
>>
>> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>>
>> Patrick
>>
>>
>
Re: CI docker builds & minor repo structure refactoring
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Patrick,
during the release we started the discussion if we should merge the extensions project with the backend project in GitHub to simplify our project structure.
I was wondering if we should do this first because it will also have a big effect on the CI job, right?
Whats your opinion on that?
Philipp
> On 6. Nov 2020, at 14:55, Patrick Wiener <wi...@apache.org> wrote:
>
> Hi guys,
>
> i noticed the current CI workflow file using Github Actions is pretty confusing when looking at it.
>
> - I’d suggest to have a more clean structure and using a separate shell script that performs the actual build, e.g., tools/ci/docker_build_push.sh
> - We could move the Dockerfiles (especially in the core repo) to a central location within the repo, e.g. docker/
> - We could rename ui/ -> streampipes-ui/
>
> What do you think? Are there any other suggestions that might help improve the CI + overall project structure?
>
> Patrick
>
>