You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Duncan Grant <du...@cloudsoftcorp.com> on 2015/08/25 16:45:48 UTC
Brooklyn logstash enricher
I'm considering adding logstash functionality so that a blueprint can be
extended so that logstash is installed on each vm and collect logs and
forward them to something like elastic search.
My initial plan is to create a logstash enricher that can be attached to an
entity. It would install logstash and configure it to look at the parent
entity's log location. The filter and output config would be configurable
on the enricher making it fairly straightforward to configure where the
logs are sent e.g. elastic search.
So yaml might look something like
services:
-type: BasicEntity
brooklyn.enrichers:
- type: Logstash
brooklyn.config:
input:
file
path: "/path/to/file"
filter:
grok:
match:
message: "%{COMBINEDAPACHELOG}"
output:
elasticsearch
protocol: "http"
With default behaviour for each of input, filter, output if they are not in
the yaml.
Does this make sense?
Regards
Duncan
--
Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP
This e-mail message is confidential and for use by the addressee only. If
the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer. Internet e-mails are not necessarily secure. Cloudsoft
Corporation Limited does not accept responsibility for changes made to this
message after it was sent.
Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the
onward transmission, opening or use of this message and any attachments
will not adversely affect its systems or data. No responsibility is
accepted by Cloudsoft Corporation Limited in this regard and the recipient
should carry out such virus and other checks as it considers appropriate.
Re: Brooklyn logstash enricher
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
Martin, All-
I think there needs to be an Entity so we have visibility around the
logstash forwarder -- this is installed, configured to point at the
collector, and could have errors when it runs.
This can be added as a child to a SoftwareProcess entity with a policy
or default config that it collects the log files which are often
published as a sensor on the SP node (so logstash entity subscribes to
parents log file locations).
To start with this is added manually in the YAML, so similar to how
Martin wrote it, but under `brooklyn.children` not `brooklyn.enrichers`.
THEN to Hadrian's point there could be a policy you could add at the
root of your Brooklyn app spec (or you could do a global subscription
hitting all deployed apps automatically), and this policy automatically
adds a logstash node to any SP node (or any SP node which doesn't come
with a logstash already). So with this you would NOT need to put it in
your PLAN but it would be present in the tree.
At runtime often it would be noise to see it, but I think that is a
separate problem, how do we make it flexible to view the items of
interest. (Relationships and a graphical GUI should help here!) This
doesn't take away from the fact that logstash SHOULD be an entity.
Best
Alex
On 25/08/2015 16:59, Svetoslav Neykov wrote:
> +1 for the functionality
>
> I'd say enricher is the right abstraction. My view on policies is that they are a piece of code which reacts to events with an action. You can suspend them to prevent them from acting. The logstash thing is way more passive.
>
> Svet.
>
>> On 25.08.2015 г., at 16:21, Hadrian Zbarcea <hz...@gmail.com> wrote:
>>
>> Shouldn't this be a policy?
>> Hadrian
>>
>> On 08/25/2015 10:45 AM, Duncan Grant wrote:
>>> I'm considering adding logstash functionality so that a blueprint can be
>>> extended so that logstash is installed on each vm and collect logs and
>>> forward them to something like elastic search.
>>>
>>> My initial plan is to create a logstash enricher that can be attached to an
>>> entity. It would install logstash and configure it to look at the parent
>>> entity's log location. The filter and output config would be configurable
>>> on the enricher making it fairly straightforward to configure where the
>>> logs are sent e.g. elastic search.
>>>
>>> So yaml might look something like
>>>
>>> services:
>>> -type: BasicEntity
>>> brooklyn.enrichers:
>>> - type: Logstash
>>> brooklyn.config:
>>> input:
>>> file
>>> path: "/path/to/file"
>>> filter:
>>> grok:
>>> match:
>>> message: "%{COMBINEDAPACHELOG}"
>>> output:
>>> elasticsearch
>>> protocol: "http"
>>>
>>> With default behaviour for each of input, filter, output if they are not in
>>> the yaml.
>>>
>>> Does this make sense?
>>>
>>> Regards
>>>
>>> Duncan
>>>
>
--
Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP
This e-mail message is confidential and for use by the addressee only. If
the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer. Internet e-mails are not necessarily secure. Cloudsoft
Corporation Limited does not accept responsibility for changes made to this
message after it was sent.
Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the
onward transmission, opening or use of this message and any attachments
will not adversely affect its systems or data. No responsibility is
accepted by Cloudsoft Corporation Limited in this regard and the recipient
should carry out such virus and other checks as it considers appropriate.
Re: Brooklyn logstash enricher
Posted by Svetoslav Neykov <sv...@cloudsoftcorp.com>.
+1 for the functionality
I'd say enricher is the right abstraction. My view on policies is that they are a piece of code which reacts to events with an action. You can suspend them to prevent them from acting. The logstash thing is way more passive.
Svet.
> On 25.08.2015 г., at 16:21, Hadrian Zbarcea <hz...@gmail.com> wrote:
>
> Shouldn't this be a policy?
> Hadrian
>
> On 08/25/2015 10:45 AM, Duncan Grant wrote:
>> I'm considering adding logstash functionality so that a blueprint can be
>> extended so that logstash is installed on each vm and collect logs and
>> forward them to something like elastic search.
>>
>> My initial plan is to create a logstash enricher that can be attached to an
>> entity. It would install logstash and configure it to look at the parent
>> entity's log location. The filter and output config would be configurable
>> on the enricher making it fairly straightforward to configure where the
>> logs are sent e.g. elastic search.
>>
>> So yaml might look something like
>>
>> services:
>> -type: BasicEntity
>> brooklyn.enrichers:
>> - type: Logstash
>> brooklyn.config:
>> input:
>> file
>> path: "/path/to/file"
>> filter:
>> grok:
>> match:
>> message: "%{COMBINEDAPACHELOG}"
>> output:
>> elasticsearch
>> protocol: "http"
>>
>> With default behaviour for each of input, filter, output if they are not in
>> the yaml.
>>
>> Does this make sense?
>>
>> Regards
>>
>> Duncan
>>
--
Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP
This e-mail message is confidential and for use by the addressee only. If
the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer. Internet e-mails are not necessarily secure. Cloudsoft
Corporation Limited does not accept responsibility for changes made to this
message after it was sent.
Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the
onward transmission, opening or use of this message and any attachments
will not adversely affect its systems or data. No responsibility is
accepted by Cloudsoft Corporation Limited in this regard and the recipient
should carry out such virus and other checks as it considers appropriate.
Re: Brooklyn logstash enricher
Posted by Hadrian Zbarcea <hz...@gmail.com>.
Shouldn't this be a policy?
Hadrian
On 08/25/2015 10:45 AM, Duncan Grant wrote:
> I'm considering adding logstash functionality so that a blueprint can be
> extended so that logstash is installed on each vm and collect logs and
> forward them to something like elastic search.
>
> My initial plan is to create a logstash enricher that can be attached to an
> entity. It would install logstash and configure it to look at the parent
> entity's log location. The filter and output config would be configurable
> on the enricher making it fairly straightforward to configure where the
> logs are sent e.g. elastic search.
>
> So yaml might look something like
>
> services:
> -type: BasicEntity
> brooklyn.enrichers:
> - type: Logstash
> brooklyn.config:
> input:
> file
> path: "/path/to/file"
> filter:
> grok:
> match:
> message: "%{COMBINEDAPACHELOG}"
> output:
> elasticsearch
> protocol: "http"
>
> With default behaviour for each of input, filter, output if they are not in
> the yaml.
>
> Does this make sense?
>
> Regards
>
> Duncan
>