You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by Patrick Wiener <wi...@apache.org> on 2020/08/17 10:20:49 UTC
[Discuss] New installer structure, and CLI
Hi,
I was working on completely refactoring the installer on the weekend [1]
The problem was always having some sort of too many areas where we needed to touch and
that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
environment.
Thus, I propose my initial work to discuss with you:
- StreamPipes Compose - The User's Choice
- StreamPipes CLI - The Developer's Favorite
- StreamPipes k8s - The Operator's Dream
======
StreamPipes Compose: User tailored
Two compose files with a default minimal version including few pipeline elments and a optional full option including more
pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
we do not need to maintain .bat or ps1 files for windows.
No ports are mapped except port 80 - so minimal surface for conflicting ports.
======
StreamPipes CLI: Developer oriented
Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
new core features.
Ports of services are mapped by default as specified in the respective service directory.
======
StreamPipes k8s: Operator oriented
Allow StreamPipes to be deployed on k8s using helm.
======
Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
helm chart using the CLI.
In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
Cheers
Patrick
[1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>
Re: [Discuss] New installer structure, and CLI
Posted by Philipp Zehnder <ze...@apache.org>.
Ah ok thanks for the explanation.
As far as I know, we do need this subnet, but we can check if this was changes in newer consul versions.
Philipp
> On 24. Aug 2020, at 22:38, Patrick Wiener <wi...@apache.org> wrote:
>
> Awesome - thx for the feedback.
>
> @Philipp: SP_SUBNET is needed so define the custom StreamPipes docker network using a custom IP address range.
> This is necessary for Consul to be started with a „static“ IP within that address range in order to bind to. Consul
> is often run in —net=host mode and thus uses the host IP.
>
> Maybe we can remove this explicit container IP assignment by now, but AFAIK thats not the case.
>
> Do you know anything about this? I mean, we could test and verify it quiet easily :)
>
> Patrick
>
>
>> Am 24.08.2020 um 22:20 schrieb Dominik Riemer <ri...@apache.org>:
>>
>> Hi,
>>
>> I also tested the new installer (under Windows + Gitbash) and it works great, and: no more encoding errors :-)
>>
>> The help feature is awesome!
>>
>> Dominik
>>
>> On 2020/08/24 16:04:21, Philipp Zehnder <ze...@apache.org> wrote:
>>> Hi Patrick,
>>>
>>> I really like the new structure and documentation, especially the help functionality for the commands is cool.
>>>
>>> I tested the new CLI on a linux machine and everything worked as expected.
>>>
>>> There is a new variable in the .env file “SP_SUBNET”. Which service requires this environment variable?
>>> Is there anything I need to be aware of during an installation?
>>>
>>> Philipp
>>>
>>>> On 17. Aug 2020, at 21:40, Patrick Wiener <wi...@apache.org> wrote:
>>>>
>>>> Good point - I guess both work. So it’s the user’s/developer’s choice.
>>>>
>>>> For the CLI, I currently modeled it in a flexible way, such that one could also use the CLI
>>>> in order to start the services specified in the environment template in so called „user“ mode.
>>>>
>>>> $ streampipes start —help
>>>> Start StreamPipes environment
>>>>
>>>> Usage: streampipes start [OPTIONS]
>>>>
>>>> Examples:
>>>> streampipes start
>>>> streampipes start --user
>>>>
>>>> Options:
>>>> -d, --dev Map docker service ports to host in dev mode (default)
>>>> -u, --user No docker service ports mapped to host (user mode)
>>>>
>>>> Thereby, only the UI port gets mapped to the host. Technically, I leverage a multi-compose file deployment
>>>> where there always exists a `docker-compose.yml` and a `docker-compose.dev.yml`, see here for reference [1].
>>>>
>>>> The dev yaml „extends“ the corresponding base yaml service with additional features, e.g. port mappings.
>>>>
>>>> When now selecting —user we only load the base yaml service description ignoring the port mappings since the
>>>> corresponding `docker-compose.dev.yml` files are not part of the deployment description.
>>>>
>>>> For the UI this is a little bit different [1] - we always have the ports mapped and have an empty `dev` yaml thus
>>>> not extending anything - we only have it for ease of loading the files in the for loop of the bash script, see
>>>> concatenate_compose_files() [3].
>>>>
>>>> Patrick
>>>>
>>>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend>>
>>>> [2] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui>>
>>>> [3] https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common> <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common>>
>>>>
>>>>
>>>>> Am 17.08.2020 um 19:32 schrieb Dominik Riemer <riemer@apache.org <ma...@apache.org>>:
>>>>>
>>>>> Hi Patrick,
>>>>>
>>>>> very cool! I also like the new README a lot.
>>>>> I think it's a good idea to have a very easy-to-use docker-compose files for users. This also reduces the maintenance effort of the current installation scripts.
>>>>> Also, keeping the CLI for developers and separate helm charts makes sense, we can think of integrating the helm charts into the CLI for the future, but for the upcoming release I think we are totally fine with the new docker-compose files, CLI and Helm charts.
>>>>>
>>>>> One question I have: Which version are users encouraged to use for non-k8s production usage? Should we take the standard docker-compose files for that? The advantage is that all ports are closed by default there, which is probably a rather important security consideration.
>>>>>
>>>>> Dominik
>>>>>
>>>>> On 2020/08/17 10:20:49, Patrick Wiener <wiener@apache.org <ma...@apache.org> <mailto:wiener@apache.org <ma...@apache.org>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I was working on completely refactoring the installer on the weekend [1]
>>>>>>
>>>>>> The problem was always having some sort of too many areas where we needed to touch and
>>>>>> that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
>>>>>> environment.
>>>>>>
>>>>>> Thus, I propose my initial work to discuss with you:
>>>>>>
>>>>>> - StreamPipes Compose - The User's Choice
>>>>>> - StreamPipes CLI - The Developer's Favorite
>>>>>> - StreamPipes k8s - The Operator's Dream
>>>>>>
>>>>>> ======
>>>>>> StreamPipes Compose: User tailored
>>>>>>
>>>>>> Two compose files with a default minimal version including few pipeline elments and a optional full option including more
>>>>>> pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
>>>>>> we do not need to maintain .bat or ps1 files for windows.
>>>>>>
>>>>>> No ports are mapped except port 80 - so minimal surface for conflicting ports.
>>>>>>
>>>>>> ======
>>>>>> StreamPipes CLI: Developer oriented
>>>>>>
>>>>>> Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
>>>>>> new core features.
>>>>>>
>>>>>> Ports of services are mapped by default as specified in the respective service directory.
>>>>>>
>>>>>> ======
>>>>>> StreamPipes k8s: Operator oriented
>>>>>>
>>>>>> Allow StreamPipes to be deployed on k8s using helm.
>>>>>>
>>>>>> ======
>>>>>>
>>>>>> Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
>>>>>> helm chart using the CLI.
>>>>>>
>>>>>> In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
>>>>>>
>>>>>>
>>>>>> Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
>>>>>>
>>>>>>
>>>>>> Cheers
>>>>>> Patrick
>>>>>>
>>>>>>
>>>>>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>>>
>>>
>>>
>
Re: [Discuss] New installer structure, and CLI
Posted by Patrick Wiener <wi...@apache.org>.
Awesome - thx for the feedback.
@Philipp: SP_SUBNET is needed so define the custom StreamPipes docker network using a custom IP address range.
This is necessary for Consul to be started with a „static“ IP within that address range in order to bind to. Consul
is often run in —net=host mode and thus uses the host IP.
Maybe we can remove this explicit container IP assignment by now, but AFAIK thats not the case.
Do you know anything about this? I mean, we could test and verify it quiet easily :)
Patrick
> Am 24.08.2020 um 22:20 schrieb Dominik Riemer <ri...@apache.org>:
>
> Hi,
>
> I also tested the new installer (under Windows + Gitbash) and it works great, and: no more encoding errors :-)
>
> The help feature is awesome!
>
> Dominik
>
> On 2020/08/24 16:04:21, Philipp Zehnder <ze...@apache.org> wrote:
>> Hi Patrick,
>>
>> I really like the new structure and documentation, especially the help functionality for the commands is cool.
>>
>> I tested the new CLI on a linux machine and everything worked as expected.
>>
>> There is a new variable in the .env file “SP_SUBNET”. Which service requires this environment variable?
>> Is there anything I need to be aware of during an installation?
>>
>> Philipp
>>
>>> On 17. Aug 2020, at 21:40, Patrick Wiener <wi...@apache.org> wrote:
>>>
>>> Good point - I guess both work. So it’s the user’s/developer’s choice.
>>>
>>> For the CLI, I currently modeled it in a flexible way, such that one could also use the CLI
>>> in order to start the services specified in the environment template in so called „user“ mode.
>>>
>>> $ streampipes start —help
>>> Start StreamPipes environment
>>>
>>> Usage: streampipes start [OPTIONS]
>>>
>>> Examples:
>>> streampipes start
>>> streampipes start --user
>>>
>>> Options:
>>> -d, --dev Map docker service ports to host in dev mode (default)
>>> -u, --user No docker service ports mapped to host (user mode)
>>>
>>> Thereby, only the UI port gets mapped to the host. Technically, I leverage a multi-compose file deployment
>>> where there always exists a `docker-compose.yml` and a `docker-compose.dev.yml`, see here for reference [1].
>>>
>>> The dev yaml „extends“ the corresponding base yaml service with additional features, e.g. port mappings.
>>>
>>> When now selecting —user we only load the base yaml service description ignoring the port mappings since the
>>> corresponding `docker-compose.dev.yml` files are not part of the deployment description.
>>>
>>> For the UI this is a little bit different [1] - we always have the ports mapped and have an empty `dev` yaml thus
>>> not extending anything - we only have it for ease of loading the files in the for loop of the bash script, see
>>> concatenate_compose_files() [3].
>>>
>>> Patrick
>>>
>>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend>>
>>> [2] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui>>
>>> [3] https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common> <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common>>
>>>
>>>
>>>> Am 17.08.2020 um 19:32 schrieb Dominik Riemer <riemer@apache.org <ma...@apache.org>>:
>>>>
>>>> Hi Patrick,
>>>>
>>>> very cool! I also like the new README a lot.
>>>> I think it's a good idea to have a very easy-to-use docker-compose files for users. This also reduces the maintenance effort of the current installation scripts.
>>>> Also, keeping the CLI for developers and separate helm charts makes sense, we can think of integrating the helm charts into the CLI for the future, but for the upcoming release I think we are totally fine with the new docker-compose files, CLI and Helm charts.
>>>>
>>>> One question I have: Which version are users encouraged to use for non-k8s production usage? Should we take the standard docker-compose files for that? The advantage is that all ports are closed by default there, which is probably a rather important security consideration.
>>>>
>>>> Dominik
>>>>
>>>> On 2020/08/17 10:20:49, Patrick Wiener <wiener@apache.org <ma...@apache.org> <mailto:wiener@apache.org <ma...@apache.org>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I was working on completely refactoring the installer on the weekend [1]
>>>>>
>>>>> The problem was always having some sort of too many areas where we needed to touch and
>>>>> that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
>>>>> environment.
>>>>>
>>>>> Thus, I propose my initial work to discuss with you:
>>>>>
>>>>> - StreamPipes Compose - The User's Choice
>>>>> - StreamPipes CLI - The Developer's Favorite
>>>>> - StreamPipes k8s - The Operator's Dream
>>>>>
>>>>> ======
>>>>> StreamPipes Compose: User tailored
>>>>>
>>>>> Two compose files with a default minimal version including few pipeline elments and a optional full option including more
>>>>> pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
>>>>> we do not need to maintain .bat or ps1 files for windows.
>>>>>
>>>>> No ports are mapped except port 80 - so minimal surface for conflicting ports.
>>>>>
>>>>> ======
>>>>> StreamPipes CLI: Developer oriented
>>>>>
>>>>> Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
>>>>> new core features.
>>>>>
>>>>> Ports of services are mapped by default as specified in the respective service directory.
>>>>>
>>>>> ======
>>>>> StreamPipes k8s: Operator oriented
>>>>>
>>>>> Allow StreamPipes to be deployed on k8s using helm.
>>>>>
>>>>> ======
>>>>>
>>>>> Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
>>>>> helm chart using the CLI.
>>>>>
>>>>> In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
>>>>>
>>>>>
>>>>> Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
>>>>>
>>>>>
>>>>> Cheers
>>>>> Patrick
>>>>>
>>>>>
>>>>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>>>
>>
>>
Re: [Discuss] New installer structure, and CLI
Posted by Dominik Riemer <ri...@apache.org>.
Hi,
I also tested the new installer (under Windows + Gitbash) and it works great, and: no more encoding errors :-)
The help feature is awesome!
Dominik
On 2020/08/24 16:04:21, Philipp Zehnder <ze...@apache.org> wrote:
> Hi Patrick,
>
> I really like the new structure and documentation, especially the help functionality for the commands is cool.
>
> I tested the new CLI on a linux machine and everything worked as expected.
>
> There is a new variable in the .env file “SP_SUBNET”. Which service requires this environment variable?
> Is there anything I need to be aware of during an installation?
>
> Philipp
>
> > On 17. Aug 2020, at 21:40, Patrick Wiener <wi...@apache.org> wrote:
> >
> > Good point - I guess both work. So it’s the user’s/developer’s choice.
> >
> > For the CLI, I currently modeled it in a flexible way, such that one could also use the CLI
> > in order to start the services specified in the environment template in so called „user“ mode.
> >
> > $ streampipes start —help
> > Start StreamPipes environment
> >
> > Usage: streampipes start [OPTIONS]
> >
> > Examples:
> > streampipes start
> > streampipes start --user
> >
> > Options:
> > -d, --dev Map docker service ports to host in dev mode (default)
> > -u, --user No docker service ports mapped to host (user mode)
> >
> > Thereby, only the UI port gets mapped to the host. Technically, I leverage a multi-compose file deployment
> > where there always exists a `docker-compose.yml` and a `docker-compose.dev.yml`, see here for reference [1].
> >
> > The dev yaml „extends“ the corresponding base yaml service with additional features, e.g. port mappings.
> >
> > When now selecting —user we only load the base yaml service description ignoring the port mappings since the
> > corresponding `docker-compose.dev.yml` files are not part of the deployment description.
> >
> > For the UI this is a little bit different [1] - we always have the ports mapped and have an empty `dev` yaml thus
> > not extending anything - we only have it for ease of loading the files in the for loop of the bash script, see
> > concatenate_compose_files() [3].
> >
> > Patrick
> >
> > [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend>>
> > [2] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui>>
> > [3] https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common> <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common>>
> >
> >
> >> Am 17.08.2020 um 19:32 schrieb Dominik Riemer <riemer@apache.org <ma...@apache.org>>:
> >>
> >> Hi Patrick,
> >>
> >> very cool! I also like the new README a lot.
> >> I think it's a good idea to have a very easy-to-use docker-compose files for users. This also reduces the maintenance effort of the current installation scripts.
> >> Also, keeping the CLI for developers and separate helm charts makes sense, we can think of integrating the helm charts into the CLI for the future, but for the upcoming release I think we are totally fine with the new docker-compose files, CLI and Helm charts.
> >>
> >> One question I have: Which version are users encouraged to use for non-k8s production usage? Should we take the standard docker-compose files for that? The advantage is that all ports are closed by default there, which is probably a rather important security consideration.
> >>
> >> Dominik
> >>
> >> On 2020/08/17 10:20:49, Patrick Wiener <wiener@apache.org <ma...@apache.org> <mailto:wiener@apache.org <ma...@apache.org>>> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I was working on completely refactoring the installer on the weekend [1]
> >>>
> >>> The problem was always having some sort of too many areas where we needed to touch and
> >>> that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
> >>> environment.
> >>>
> >>> Thus, I propose my initial work to discuss with you:
> >>>
> >>> - StreamPipes Compose - The User's Choice
> >>> - StreamPipes CLI - The Developer's Favorite
> >>> - StreamPipes k8s - The Operator's Dream
> >>>
> >>> ======
> >>> StreamPipes Compose: User tailored
> >>>
> >>> Two compose files with a default minimal version including few pipeline elments and a optional full option including more
> >>> pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
> >>> we do not need to maintain .bat or ps1 files for windows.
> >>>
> >>> No ports are mapped except port 80 - so minimal surface for conflicting ports.
> >>>
> >>> ======
> >>> StreamPipes CLI: Developer oriented
> >>>
> >>> Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
> >>> new core features.
> >>>
> >>> Ports of services are mapped by default as specified in the respective service directory.
> >>>
> >>> ======
> >>> StreamPipes k8s: Operator oriented
> >>>
> >>> Allow StreamPipes to be deployed on k8s using helm.
> >>>
> >>> ======
> >>>
> >>> Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
> >>> helm chart using the CLI.
> >>>
> >>> In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
> >>>
> >>>
> >>> Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
> >>>
> >>>
> >>> Cheers
> >>> Patrick
> >>>
> >>>
> >>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>>>
>
>
Re: [Discuss] New installer structure, and CLI
Posted by Philipp Zehnder <ze...@apache.org>.
Hi Patrick,
I really like the new structure and documentation, especially the help functionality for the commands is cool.
I tested the new CLI on a linux machine and everything worked as expected.
There is a new variable in the .env file “SP_SUBNET”. Which service requires this environment variable?
Is there anything I need to be aware of during an installation?
Philipp
> On 17. Aug 2020, at 21:40, Patrick Wiener <wi...@apache.org> wrote:
>
> Good point - I guess both work. So it’s the user’s/developer’s choice.
>
> For the CLI, I currently modeled it in a flexible way, such that one could also use the CLI
> in order to start the services specified in the environment template in so called „user“ mode.
>
> $ streampipes start —help
> Start StreamPipes environment
>
> Usage: streampipes start [OPTIONS]
>
> Examples:
> streampipes start
> streampipes start --user
>
> Options:
> -d, --dev Map docker service ports to host in dev mode (default)
> -u, --user No docker service ports mapped to host (user mode)
>
> Thereby, only the UI port gets mapped to the host. Technically, I leverage a multi-compose file deployment
> where there always exists a `docker-compose.yml` and a `docker-compose.dev.yml`, see here for reference [1].
>
> The dev yaml „extends“ the corresponding base yaml service with additional features, e.g. port mappings.
>
> When now selecting —user we only load the base yaml service description ignoring the port mappings since the
> corresponding `docker-compose.dev.yml` files are not part of the deployment description.
>
> For the UI this is a little bit different [1] - we always have the ports mapped and have an empty `dev` yaml thus
> not extending anything - we only have it for ease of loading the files in the for loop of the bash script, see
> concatenate_compose_files() [3].
>
> Patrick
>
> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend>>
> [2] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui>>
> [3] https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common> <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common>>
>
>
>> Am 17.08.2020 um 19:32 schrieb Dominik Riemer <riemer@apache.org <ma...@apache.org>>:
>>
>> Hi Patrick,
>>
>> very cool! I also like the new README a lot.
>> I think it's a good idea to have a very easy-to-use docker-compose files for users. This also reduces the maintenance effort of the current installation scripts.
>> Also, keeping the CLI for developers and separate helm charts makes sense, we can think of integrating the helm charts into the CLI for the future, but for the upcoming release I think we are totally fine with the new docker-compose files, CLI and Helm charts.
>>
>> One question I have: Which version are users encouraged to use for non-k8s production usage? Should we take the standard docker-compose files for that? The advantage is that all ports are closed by default there, which is probably a rather important security consideration.
>>
>> Dominik
>>
>> On 2020/08/17 10:20:49, Patrick Wiener <wiener@apache.org <ma...@apache.org> <mailto:wiener@apache.org <ma...@apache.org>>> wrote:
>>>
>>> Hi,
>>>
>>> I was working on completely refactoring the installer on the weekend [1]
>>>
>>> The problem was always having some sort of too many areas where we needed to touch and
>>> that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
>>> environment.
>>>
>>> Thus, I propose my initial work to discuss with you:
>>>
>>> - StreamPipes Compose - The User's Choice
>>> - StreamPipes CLI - The Developer's Favorite
>>> - StreamPipes k8s - The Operator's Dream
>>>
>>> ======
>>> StreamPipes Compose: User tailored
>>>
>>> Two compose files with a default minimal version including few pipeline elments and a optional full option including more
>>> pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
>>> we do not need to maintain .bat or ps1 files for windows.
>>>
>>> No ports are mapped except port 80 - so minimal surface for conflicting ports.
>>>
>>> ======
>>> StreamPipes CLI: Developer oriented
>>>
>>> Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
>>> new core features.
>>>
>>> Ports of services are mapped by default as specified in the respective service directory.
>>>
>>> ======
>>> StreamPipes k8s: Operator oriented
>>>
>>> Allow StreamPipes to be deployed on k8s using helm.
>>>
>>> ======
>>>
>>> Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
>>> helm chart using the CLI.
>>>
>>> In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
>>>
>>>
>>> Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
>>>
>>>
>>> Cheers
>>> Patrick
>>>
>>>
>>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>>>
Re: [Discuss] New installer structure, and CLI
Posted by Patrick Wiener <wi...@apache.org>.
Good point - I guess both work. So it’s the user’s/developer’s choice.
For the CLI, I currently modeled it in a flexible way, such that one could also use the CLI
in order to start the services specified in the environment template in so called „user“ mode.
$ streampipes start —help
Start StreamPipes environment
Usage: streampipes start [OPTIONS]
Examples:
streampipes start
streampipes start --user
Options:
-d, --dev Map docker service ports to host in dev mode (default)
-u, --user No docker service ports mapped to host (user mode)
Thereby, only the UI port gets mapped to the host. Technically, I leverage a multi-compose file deployment
where there always exists a `docker-compose.yml` and a `docker-compose.dev.yml`, see here for reference [1].
The dev yaml „extends“ the corresponding base yaml service with additional features, e.g. port mappings.
When now selecting —user we only load the base yaml service description ignoring the port mappings since the
corresponding `docker-compose.dev.yml` files are not part of the deployment description.
For the UI this is a little bit different [1] - we always have the ports mapped and have an empty `dev` yaml thus
not extending anything - we only have it for ease of loading the files in the for loop of the bash script, see
concatenate_compose_files() [3].
Patrick
[1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/backend>
[2] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli/deploy/standalone/ui>
[3] https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common <https://github.com/apache/incubator-streampipes-installer/blob/new_installer/cli/bin/common>
> Am 17.08.2020 um 19:32 schrieb Dominik Riemer <ri...@apache.org>:
>
> Hi Patrick,
>
> very cool! I also like the new README a lot.
> I think it's a good idea to have a very easy-to-use docker-compose files for users. This also reduces the maintenance effort of the current installation scripts.
> Also, keeping the CLI for developers and separate helm charts makes sense, we can think of integrating the helm charts into the CLI for the future, but for the upcoming release I think we are totally fine with the new docker-compose files, CLI and Helm charts.
>
> One question I have: Which version are users encouraged to use for non-k8s production usage? Should we take the standard docker-compose files for that? The advantage is that all ports are closed by default there, which is probably a rather important security consideration.
>
> Dominik
>
> On 2020/08/17 10:20:49, Patrick Wiener <wiener@apache.org <ma...@apache.org>> wrote:
>>
>> Hi,
>>
>> I was working on completely refactoring the installer on the weekend [1]
>>
>> The problem was always having some sort of too many areas where we needed to touch and
>> that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
>> environment.
>>
>> Thus, I propose my initial work to discuss with you:
>>
>> - StreamPipes Compose - The User's Choice
>> - StreamPipes CLI - The Developer's Favorite
>> - StreamPipes k8s - The Operator's Dream
>>
>> ======
>> StreamPipes Compose: User tailored
>>
>> Two compose files with a default minimal version including few pipeline elments and a optional full option including more
>> pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
>> we do not need to maintain .bat or ps1 files for windows.
>>
>> No ports are mapped except port 80 - so minimal surface for conflicting ports.
>>
>> ======
>> StreamPipes CLI: Developer oriented
>>
>> Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
>> new core features.
>>
>> Ports of services are mapped by default as specified in the respective service directory.
>>
>> ======
>> StreamPipes k8s: Operator oriented
>>
>> Allow StreamPipes to be deployed on k8s using helm.
>>
>> ======
>>
>> Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
>> helm chart using the CLI.
>>
>> In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
>>
>>
>> Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
>>
>>
>> Cheers
>> Patrick
>>
>>
>> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli> <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>>
Re: [Discuss] New installer structure, and CLI
Posted by Dominik Riemer <ri...@apache.org>.
Hi Patrick,
very cool! I also like the new README a lot.
I think it's a good idea to have a very easy-to-use docker-compose files for users. This also reduces the maintenance effort of the current installation scripts.
Also, keeping the CLI for developers and separate helm charts makes sense, we can think of integrating the helm charts into the CLI for the future, but for the upcoming release I think we are totally fine with the new docker-compose files, CLI and Helm charts.
One question I have: Which version are users encouraged to use for non-k8s production usage? Should we take the standard docker-compose files for that? The advantage is that all ports are closed by default there, which is probably a rather important security consideration.
Dominik
On 2020/08/17 10:20:49, Patrick Wiener <wi...@apache.org> wrote:
>
> Hi,
>
> I was working on completely refactoring the installer on the weekend [1]
>
> The problem was always having some sort of too many areas where we needed to touch and
> that we did not have a clear focus on what kinda user role should use what kinda way to set up his/her
> environment.
>
> Thus, I propose my initial work to discuss with you:
>
> - StreamPipes Compose - The User's Choice
> - StreamPipes CLI - The Developer's Favorite
> - StreamPipes k8s - The Operator's Dream
>
> ======
> StreamPipes Compose: User tailored
>
> Two compose files with a default minimal version including few pipeline elments and a optional full option including more
> pipeline elements. Thus this should work for almost all OS platforms having Docker and Docker-Compose installed. So,
> we do not need to maintain .bat or ps1 files for windows.
>
> No ports are mapped except port 80 - so minimal surface for conflicting ports.
>
> ======
> StreamPipes CLI: Developer oriented
>
> Command line interface to easily setup various dev environments locally in order to either write new pipeline elements or
> new core features.
>
> Ports of services are mapped by default as specified in the respective service directory.
>
> ======
> StreamPipes k8s: Operator oriented
>
> Allow StreamPipes to be deployed on k8s using helm.
>
> ======
>
> Overall we layed the foundation. However, I guess we can further integrate the k8s option in the CLI in order to deploy our
> helm chart using the CLI.
>
> In addition, the CLI could be easily extended with new features such as creating new project skeletons using our existing maven archetype.
>
>
> Take a look at it, try it out. Discuss what kind of features to include or tell whats your general opinion.
>
>
> Cheers
> Patrick
>
>
> [1] https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli <https://github.com/apache/incubator-streampipes-installer/tree/new_installer/cli>