You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Patrick Wiener <wi...@apache.org> on 2020/11/06 15:18:24 UTC
[DISCUSS] Single StreamPipes repository
Hi all,
we have thrown up the discussion on merging our repositories into a single StreamPipes repo for core and extensions on several occasions.
Now, we should discuss the pros and cons about this to see if it’s worth doing this step.
Context: Merge core [1] and extensions [2] into a single repo.
Pro:
- everything in one spot (reduced complexity, no separate repos, where you have to look for stuff, makes it easier for releasing as well)
- maintainability (easier for CI automation, mostly due to the point above)
- improved dev experience (everything in one project, especially when doing changes to the core that affect extensions, i.e. adapter and processors, sinks, no different env setups)
Cons:
- longer CI builds (changes to core not affecting extensions still lead to extensions being build and pushed and the other way around)
- would need to refactor project structure to maintain project clarity
Please share you opinion to bring this topic forward.
Patrick
[1] https://github.com/apache/incubator-streampipes
[2] https://github.com/apache/incubator-streampipes-extensions
Re: [DISCUSS] Single StreamPipes repository
Posted by Patrick Wiener <wi...@apache.org>.
I haven’t thought about the installer repo so far since we first should agree on a common goal how to proceed in this manner.
Currently, the mood seems to be pro merge. So what would be possible structures?
Option1: Separate (at first glance looks really small):
root/
- core/
- extensions/
- pom.xml
Option2: Integrating extensions as a module:
root/
- streampipes-backend
- streampipes-code-generation
- …
- streampipes-extensions
- …
- pom.xml
Any other option? I would prefer option 2 as it shows a bit more of the richness of our project.
Patrick
> Am 10.11.2020 um 07:59 schrieb Dominik Riemer <ri...@apache.org>:
>
> ..and another question, I think one of the main advantage of this approach is a simplified voting process. So are we going to release the installer independently of core+extensions in the future?
> If so, how do we make sure that the installer always points to the latest release version?
>
> Dominik
>
> On 2020/11/10 06:55:05, Dominik Riemer <ri...@apache.org> wrote:
>> Hi,
>>
>> one thing to consider: We would need to modify the pom files after merging as only core artifacts make use of the Maven deploy step (we skip that for extensions as the produced artifacts are mainly fat jars).
>> So we would need to configure deployment on module level in the module's pom files.
>>
>> How would the new structure look like? Do we plan to merge the two parent pom files into a single one and integrate the extension modules in the top-level folder of the core repo?
>>
>> Dominik
>>
>> On 2020/11/09 15:38:52, Patrick Wiener <wi...@apache.org> wrote:
>>> For the extensions CI pipeline, we first checkout the latest dev commit and build the project.
>>>
>>> In a combined repo we would have the following steps:
>>>
>>> - maven build (core + extensions)
>>> - ui build
>>> - docker build (all)
>>> - docker push (all)
>>>
>>> Patrick
>>>
>>>
>>>> Am 09.11.2020 um 12:48 schrieb Philipp Zehnder <ze...@apache.org>:
>>>>
>>>> Hi,
>>>>
>>>> thanks Patrick for starting the discussion and the good overview of the advantages and disadvantages.
>>>>
>>>> I am in favor of merging the repositories, because I think the advantages outweigh the disadvantages.
>>>>
>>>> Would the changes in Maven and the pom files be easily done or could there be any problems we must think of?
>>>>
>>>> Regarding the build times.
>>>> That's right, it would probably take much longer to build the project in the CI pipeline.
>>>> But that's already the case today when you build the extensions project, because the backend has to be built first, right?
>>>>
>>>> Philipp
>>>>
>>>>> On 6. Nov 2020, at 16:18, Patrick Wiener <wi...@apache.org> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> we have thrown up the discussion on merging our repositories into a single StreamPipes repo for core and extensions on several occasions.
>>>>> Now, we should discuss the pros and cons about this to see if it’s worth doing this step.
>>>>>
>>>>> Context: Merge core [1] and extensions [2] into a single repo.
>>>>>
>>>>> Pro:
>>>>> - everything in one spot (reduced complexity, no separate repos, where you have to look for stuff, makes it easier for releasing as well)
>>>>> - maintainability (easier for CI automation, mostly due to the point above)
>>>>> - improved dev experience (everything in one project, especially when doing changes to the core that affect extensions, i.e. adapter and processors, sinks, no different env setups)
>>>>>
>>>>> Cons:
>>>>> - longer CI builds (changes to core not affecting extensions still lead to extensions being build and pushed and the other way around)
>>>>> - would need to refactor project structure to maintain project clarity
>>>>>
>>>>> Please share you opinion to bring this topic forward.
>>>>>
>>>>> Patrick
>>>>>
>>>>> [1] https://github.com/apache/incubator-streampipes
>>>>> [2] https://github.com/apache/incubator-streampipes-extensions
>>>>
>>>
>>>
>>
Re: [DISCUSS] Single StreamPipes repository
Posted by Dominik Riemer <ri...@apache.org>.
..and another question, I think one of the main advantage of this approach is a simplified voting process. So are we going to release the installer independently of core+extensions in the future?
If so, how do we make sure that the installer always points to the latest release version?
Dominik
On 2020/11/10 06:55:05, Dominik Riemer <ri...@apache.org> wrote:
> Hi,
>
> one thing to consider: We would need to modify the pom files after merging as only core artifacts make use of the Maven deploy step (we skip that for extensions as the produced artifacts are mainly fat jars).
> So we would need to configure deployment on module level in the module's pom files.
>
> How would the new structure look like? Do we plan to merge the two parent pom files into a single one and integrate the extension modules in the top-level folder of the core repo?
>
> Dominik
>
> On 2020/11/09 15:38:52, Patrick Wiener <wi...@apache.org> wrote:
> > For the extensions CI pipeline, we first checkout the latest dev commit and build the project.
> >
> > In a combined repo we would have the following steps:
> >
> > - maven build (core + extensions)
> > - ui build
> > - docker build (all)
> > - docker push (all)
> >
> > Patrick
> >
> >
> > > Am 09.11.2020 um 12:48 schrieb Philipp Zehnder <ze...@apache.org>:
> > >
> > > Hi,
> > >
> > > thanks Patrick for starting the discussion and the good overview of the advantages and disadvantages.
> > >
> > > I am in favor of merging the repositories, because I think the advantages outweigh the disadvantages.
> > >
> > > Would the changes in Maven and the pom files be easily done or could there be any problems we must think of?
> > >
> > > Regarding the build times.
> > > That's right, it would probably take much longer to build the project in the CI pipeline.
> > > But that's already the case today when you build the extensions project, because the backend has to be built first, right?
> > >
> > > Philipp
> > >
> > >> On 6. Nov 2020, at 16:18, Patrick Wiener <wi...@apache.org> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> we have thrown up the discussion on merging our repositories into a single StreamPipes repo for core and extensions on several occasions.
> > >> Now, we should discuss the pros and cons about this to see if it’s worth doing this step.
> > >>
> > >> Context: Merge core [1] and extensions [2] into a single repo.
> > >>
> > >> Pro:
> > >> - everything in one spot (reduced complexity, no separate repos, where you have to look for stuff, makes it easier for releasing as well)
> > >> - maintainability (easier for CI automation, mostly due to the point above)
> > >> - improved dev experience (everything in one project, especially when doing changes to the core that affect extensions, i.e. adapter and processors, sinks, no different env setups)
> > >>
> > >> Cons:
> > >> - longer CI builds (changes to core not affecting extensions still lead to extensions being build and pushed and the other way around)
> > >> - would need to refactor project structure to maintain project clarity
> > >>
> > >> Please share you opinion to bring this topic forward.
> > >>
> > >> Patrick
> > >>
> > >> [1] https://github.com/apache/incubator-streampipes
> > >> [2] https://github.com/apache/incubator-streampipes-extensions
> > >
> >
> >
>
Re: [DISCUSS] Single StreamPipes repository
Posted by Dominik Riemer <ri...@apache.org>.
Hi,
one thing to consider: We would need to modify the pom files after merging as only core artifacts make use of the Maven deploy step (we skip that for extensions as the produced artifacts are mainly fat jars).
So we would need to configure deployment on module level in the module's pom files.
How would the new structure look like? Do we plan to merge the two parent pom files into a single one and integrate the extension modules in the top-level folder of the core repo?
Dominik
On 2020/11/09 15:38:52, Patrick Wiener <wi...@apache.org> wrote:
> For the extensions CI pipeline, we first checkout the latest dev commit and build the project.
>
> In a combined repo we would have the following steps:
>
> - maven build (core + extensions)
> - ui build
> - docker build (all)
> - docker push (all)
>
> Patrick
>
>
> > Am 09.11.2020 um 12:48 schrieb Philipp Zehnder <ze...@apache.org>:
> >
> > Hi,
> >
> > thanks Patrick for starting the discussion and the good overview of the advantages and disadvantages.
> >
> > I am in favor of merging the repositories, because I think the advantages outweigh the disadvantages.
> >
> > Would the changes in Maven and the pom files be easily done or could there be any problems we must think of?
> >
> > Regarding the build times.
> > That's right, it would probably take much longer to build the project in the CI pipeline.
> > But that's already the case today when you build the extensions project, because the backend has to be built first, right?
> >
> > Philipp
> >
> >> On 6. Nov 2020, at 16:18, Patrick Wiener <wi...@apache.org> wrote:
> >>
> >> Hi all,
> >>
> >> we have thrown up the discussion on merging our repositories into a single StreamPipes repo for core and extensions on several occasions.
> >> Now, we should discuss the pros and cons about this to see if it’s worth doing this step.
> >>
> >> Context: Merge core [1] and extensions [2] into a single repo.
> >>
> >> Pro:
> >> - everything in one spot (reduced complexity, no separate repos, where you have to look for stuff, makes it easier for releasing as well)
> >> - maintainability (easier for CI automation, mostly due to the point above)
> >> - improved dev experience (everything in one project, especially when doing changes to the core that affect extensions, i.e. adapter and processors, sinks, no different env setups)
> >>
> >> Cons:
> >> - longer CI builds (changes to core not affecting extensions still lead to extensions being build and pushed and the other way around)
> >> - would need to refactor project structure to maintain project clarity
> >>
> >> Please share you opinion to bring this topic forward.
> >>
> >> Patrick
> >>
> >> [1] https://github.com/apache/incubator-streampipes
> >> [2] https://github.com/apache/incubator-streampipes-extensions
> >
>
>
Re: [DISCUSS] Single StreamPipes repository
Posted by Patrick Wiener <wi...@apache.org>.
For the extensions CI pipeline, we first checkout the latest dev commit and build the project.
In a combined repo we would have the following steps:
- maven build (core + extensions)
- ui build
- docker build (all)
- docker push (all)
Patrick
> Am 09.11.2020 um 12:48 schrieb Philipp Zehnder <ze...@apache.org>:
>
> Hi,
>
> thanks Patrick for starting the discussion and the good overview of the advantages and disadvantages.
>
> I am in favor of merging the repositories, because I think the advantages outweigh the disadvantages.
>
> Would the changes in Maven and the pom files be easily done or could there be any problems we must think of?
>
> Regarding the build times.
> That's right, it would probably take much longer to build the project in the CI pipeline.
> But that's already the case today when you build the extensions project, because the backend has to be built first, right?
>
> Philipp
>
>> On 6. Nov 2020, at 16:18, Patrick Wiener <wi...@apache.org> wrote:
>>
>> Hi all,
>>
>> we have thrown up the discussion on merging our repositories into a single StreamPipes repo for core and extensions on several occasions.
>> Now, we should discuss the pros and cons about this to see if it’s worth doing this step.
>>
>> Context: Merge core [1] and extensions [2] into a single repo.
>>
>> Pro:
>> - everything in one spot (reduced complexity, no separate repos, where you have to look for stuff, makes it easier for releasing as well)
>> - maintainability (easier for CI automation, mostly due to the point above)
>> - improved dev experience (everything in one project, especially when doing changes to the core that affect extensions, i.e. adapter and processors, sinks, no different env setups)
>>
>> Cons:
>> - longer CI builds (changes to core not affecting extensions still lead to extensions being build and pushed and the other way around)
>> - would need to refactor project structure to maintain project clarity
>>
>> Please share you opinion to bring this topic forward.
>>
>> Patrick
>>
>> [1] https://github.com/apache/incubator-streampipes
>> [2] https://github.com/apache/incubator-streampipes-extensions
>
Re: [DISCUSS] Single StreamPipes repository
Posted by Philipp Zehnder <ze...@apache.org>.
Hi,
thanks Patrick for starting the discussion and the good overview of the advantages and disadvantages.
I am in favor of merging the repositories, because I think the advantages outweigh the disadvantages.
Would the changes in Maven and the pom files be easily done or could there be any problems we must think of?
Regarding the build times.
That's right, it would probably take much longer to build the project in the CI pipeline.
But that's already the case today when you build the extensions project, because the backend has to be built first, right?
Philipp
> On 6. Nov 2020, at 16:18, Patrick Wiener <wi...@apache.org> wrote:
>
> Hi all,
>
> we have thrown up the discussion on merging our repositories into a single StreamPipes repo for core and extensions on several occasions.
> Now, we should discuss the pros and cons about this to see if it’s worth doing this step.
>
> Context: Merge core [1] and extensions [2] into a single repo.
>
> Pro:
> - everything in one spot (reduced complexity, no separate repos, where you have to look for stuff, makes it easier for releasing as well)
> - maintainability (easier for CI automation, mostly due to the point above)
> - improved dev experience (everything in one project, especially when doing changes to the core that affect extensions, i.e. adapter and processors, sinks, no different env setups)
>
> Cons:
> - longer CI builds (changes to core not affecting extensions still lead to extensions being build and pushed and the other way around)
> - would need to refactor project structure to maintain project clarity
>
> Please share you opinion to bring this topic forward.
>
> Patrick
>
> [1] https://github.com/apache/incubator-streampipes
> [2] https://github.com/apache/incubator-streampipes-extensions