Entities as system services

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

Hi,

I created an enricher which when attached to an entity will register it as a system service, so the entity is started on machine restarts. See [1] for the PR. Currently old-style /etc/init.d services are implemented (used in RHEL 6).
It probably makes sense to move it to SoftwareProcess entity, but an enricher initially looked like a better choice because it can be used in existing installations, wdyt?

How it works - it takes the launch script from the start effector, saves it as a file and then calls it from an /etc/init.d script.The launch script is updated on each restart so changes in config keys will be updated in the service.

Svet.

[1] https://github.com/apache/incubator-brooklyn/pull/556

Re: Entities as system services

[Andrew Kennedy <an...@cloudsoftcorp.com>]

On 18 March 2015 at 10:45, Aled Sage <al...@gmail.com> wrote:

> _*SoftwareProcess installs*_
> As was mentioned on IRC recently, should we switch to everywhere
> installing services (initd/systemd/etc).
>
> Currently, some entities are designed to support localhost — without
> modifying files outside their install/run directories. This is possible for
> things like Tomcat where one common way to install it is to unpack the
> tar.gz.
>
> Switching everywhere to RPMs/initd/etc would arguably be better for
> production deployments, but would make it harder to test+run on
> localhost/osx.
> We'd need to use Vargrant/virtualbox a lot more.
>

Note that this will not work very well with Clocker and Docker containers.
In that case, the service is the only thing running, and restart is handled
as a new container usually.

Andrew.
--
-- andrew kennedy ; clocker.io project founder ; @grkvlt ;

Re: Entities as system services

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

Aled,

* Monit - Didn't know of the monit entity, could be very useful. The entity could be reworked into an enricher, similarly to SystemServiceEnricher, so it doesn't require SameServerEntity and needs even less manual configuration. At the very least it could benefit from SystemServiceEnricher for the startup/stop commands.

* Brooklyn restart policies, versus init.d / monit - The goal here was not to have a watchdog of the process, but simply to start it on machine restarts. I'd expect to have it combined with restart policies anyway. Systemd supports watchdog though, so could be useful in certain scenarios.
The current implementation of the restart effector has a deficiency when faced with network downtime. It will try to restart the entity and fail permanently, while the entity is happily running on the machine. When the network comes back the only way to repair the entity is to restart it which is surprising for users - why restart a perfectly fine running process? We should add network detection in the restart effector and back off if needed. Alternatively we could have monit for process restarts + policy which handles catastrophic events (replaces the machine).

* SoftwareProcess installs - +1 for installing system packages (and services) + VM. This is how I work atm - combination of jclouds vagrant + byon virtualbox machines. I don't even have localhost ssh enabled.
Certain software would still make sense to be installed from downloads but we could take additional steps to integrate it with the system.

* Mixins for entities - "one has to drop into Java to override various methods."
This was raised by @grkvlt in [1], we need to introduce some callback/event based mechanism, so it is easier to customize the behaviour of MachineLifecycleEffectorTasks and its subclasses.

Svet.


[1] https://github.com/apache/incubator-brooklyn/pull/389#discussion_r21784597 <https://github.com/apache/incubator-brooklyn/pull/389#discussion_r21784597>



> On 18.03.2015 г., at 12:45, Aled Sage <al...@gmail.com> wrote:
> 
> Svet,
> 
> Nice - interesting and clever approach.
> 
> I like the enricher for being able to use as a mixin for other entities.
> Agree that having it in SoftwareProcess (e.g. enabled via setting a config key) would also be really useful.
> 
> Longer term, it raises several questions. Each of these probably deserves its own new thread, if we're going to kick-off in depth conversations about them.
> 
> _*Monit*_
> We have support for monit, which can be associated with an entity.
> software/monitoring/src/main/java/brooklyn/entity/monitoring/monit/MonitNode.java
> 
> However, its configuration in Brooklyn is a bit fiddly: it requires use of a SameServerEntity for co-location, and the monitrc config file is not entirely auto-generated as with your init.d approach.
> 
> Would be good to think about both the system service approach and monit.
> 
> _*Brooklyn restart policies, versus init.d / monit*_
> There is an interesting relationship between writing policies to restart processes, and this approach of auto-configuring init.d.
> 
> The advantage of using policies is that it's more visible how often restarts are required (e.g. if need to restart more than 5 times in 10 minutes, then we'll escalate to replace the VM).
> 
> However, there is definitely a place for this kind of on-box restart: it's fast (no networking traffic for polling etc), well established, and can still be combined with higher level policies.
> 
> _*SoftwareProcess installs*_
> As was mentioned on IRC recently, should we switch to everywhere installing services (initd/systemd/etc).
> 
> Currently, some entities are designed to support localhost — without modifying files outside their install/run directories. This is possible for things like Tomcat where one common way to install it is to unpack the tar.gz.
> 
> Switching everywhere to RPMs/initd/etc would arguably be better for production deployments, but would make it harder to test+run on localhost/osx.
> We'd need to use Vargrant/virtualbox a lot more.
> 
> _*Mixins for entities*_
> How do we better support mixins for entities (including for SoftwareProcess).
> 
> Enrichers/policies are good for synchronous behaviour, but not for doing customisation at specific points (e.g. pre/post the install, customize or launch).
> 
> We can already wire in at various points extra commands to be executed or files/templates to be uploaded. But for some more advanced customization use-cases one has to drop into Java to override various methods.
> 
> Aled
> 
> 
> On 17/03/2015 18:41, Svetoslav Neykov wrote:
>> Hi,
>> 
>> I created an enricher which when attached to an entity will register it as a system service, so the entity is started on machine restarts. See [1] for the PR. Currently old-style /etc/init.d services are implemented (used in RHEL 6).
>> It probably makes sense to move it to SoftwareProcess entity, but an enricher initially looked like a better choice because it can be used in existing installations, wdyt?
>> 
>> How it works - it takes the launch script from the start effector, saves it as a file and then calls it from an /etc/init.d script.The launch script is updated on each restart so changes in config keys will be updated in the service.
>> 
>> Svet.
>> 
>> [1] https://github.com/apache/incubator-brooklyn/pull/556
> 

Re: Entities as system services

[Aled Sage <al...@gmail.com>]

Svet,

Nice - interesting and clever approach.

I like the enricher for being able to use as a mixin for other entities.
Agree that having it in SoftwareProcess (e.g. enabled via setting a 
config key) would also be really useful.

Longer term, it raises several questions. Each of these probably 
deserves its own new thread, if we're going to kick-off in depth 
conversations about them.

_*Monit*_
We have support for monit, which can be associated with an entity.
software/monitoring/src/main/java/brooklyn/entity/monitoring/monit/MonitNode.java

However, its configuration in Brooklyn is a bit fiddly: it requires use 
of a SameServerEntity for co-location, and the monitrc config file is 
not entirely auto-generated as with your init.d approach.

Would be good to think about both the system service approach and monit.

_*Brooklyn restart policies, versus init.d / monit*_
There is an interesting relationship between writing policies to restart 
processes, and this approach of auto-configuring init.d.

The advantage of using policies is that it's more visible how often 
restarts are required (e.g. if need to restart more than 5 times in 10 
minutes, then we'll escalate to replace the VM).

However, there is definitely a place for this kind of on-box restart: 
it's fast (no networking traffic for polling etc), well established, and 
can still be combined with higher level policies.

_*SoftwareProcess installs*_
As was mentioned on IRC recently, should we switch to everywhere 
installing services (initd/systemd/etc).

Currently, some entities are designed to support localhost — without 
modifying files outside their install/run directories. This is possible 
for things like Tomcat where one common way to install it is to unpack 
the tar.gz.

Switching everywhere to RPMs/initd/etc would arguably be better for 
production deployments, but would make it harder to test+run on 
localhost/osx.
We'd need to use Vargrant/virtualbox a lot more.

_*Mixins for entities*_
How do we better support mixins for entities (including for 
SoftwareProcess).

Enrichers/policies are good for synchronous behaviour, but not for doing 
customisation at specific points (e.g. pre/post the install, customize 
or launch).

We can already wire in at various points extra commands to be executed 
or files/templates to be uploaded. But for some more advanced 
customization use-cases one has to drop into Java to override various 
methods.

Aled


On 17/03/2015 18:41, Svetoslav Neykov wrote:
> Hi,
>
> I created an enricher which when attached to an entity will register it as a system service, so the entity is started on machine restarts. See [1] for the PR. Currently old-style /etc/init.d services are implemented (used in RHEL 6).
> It probably makes sense to move it to SoftwareProcess entity, but an enricher initially looked like a better choice because it can be used in existing installations, wdyt?
>
> How it works - it takes the launch script from the start effector, saves it as a file and then calls it from an /etc/init.d script.The launch script is updated on each restart so changes in config keys will be updated in the service.
>
> Svet.
>
> [1] https://github.com/apache/incubator-brooklyn/pull/556