You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@metron.apache.org by Christian Tramnitz <tr...@trasec.de> on 2017/05/09 16:00:38 UTC

telemetry integration

Correct me if I’m wrong, but we currently have a lot of hard-coded configuration files with the standard telemetry (yaf, bro, snort) included, but no consistent way to add additional telemetry configuration where needed?!
I.e. we have
- /usr/metron/<version>/config/zookeeper/[enrichments|parsers|indexing]/<telemetry-name>.json for topology configuration on the FS and a matching layout in zk
- ES templates
- Kibana configuration (both for index names and within some searches and vis elements)
- Kafka topic creation and configuration
The above exist for yaf, bro and snort only and it’s hard to get an additional telemetry source added consistenly. I.e. if I make manual adjustments to ES templates and Kibana configuration any “publish templates” action from Ambari will overwrite those changes.

Wouldn’t it make sense to “register” telemetry sources centrally and then have the required configuration derived from there? Instead of starting with an exported/imported .kibana index it could be dynamically created (at runtime, not at build time), keeping user-defined values and only overwriting specific configuration statements.

Is metron_service.py (METRON-777) supposed to solve some of those issues? If yes, would every telemetry source be treated as plugin?


BR,
   Christian

Re: telemetry integration

Posted by Otto Fowler <ot...@gmail.com>.

Also, you will be able to create the archetype project outside of the
metron code tree. Although the root parent pom should still be pointing to
*into* metron.
My goal is to eliminate any necessity to fork metron is possible, you
shouldn’t have to unless you really want to.

On May 9, 2017 at 13:25:03, Otto Fowler (ottobackwards@gmail.com) wrote:

Yes, METRON-777 is the first step to resolving that. The follow on, that
I’ll be putting up as a follow up soon is METRON-942 ( Rest interface for
installing, querying and uninstalling extensions ).
As of METRON-777, every parser is a plugin ( or extension as I am calling
them ) and has been refactored as such. The only parsers that are not
extensions, are the root type parsers ( JSON, CSV, GROK ).

When both of them are landed we will be able to:

Use an archetype to create a new parser extension.
That archetype will include:
* Sample parser
* Configurations for indexing, enrichment, parser
* Configuration for elasticsearch
* grok patterns dir if you choose.
* unit and integration tests for the sample to expand on

You can have multiple parsers in an extension library.
You can delete the java for the parser, and keep the tests ( if you look at
Yaf in METRON-777 you can see ) and
have a configuration only parser, but still have tests.

The product of building the archetype produced project is a tar.gz
assembly, that
contains the configurations and the ‘bundle’, which is the code/jar package.

Using the rest-api ( swagger let’s say ) -
You can then install the extension, and all the bits will be played out.
You can then use the rest api to create the kafka topic, the storm job etc
and run the parsers.

You can edit the configuration that are installed after that using the rest
api ( the management ui may need some work as a follow on ).

The elasticsearch templates are installed with the extension configuration
in ZK, the idea being that down the road we will allow editing and
deployment
of those configurations. At the moment though ( or in the future when this
lands initially ) the installation of elasticsearch templates is not
handled.

I am trying to set up what will be required to have a
Install -> configure -> deploy
flow.

Also, the rest api does not talk to ambari, so if you want your parser to
survive a reboot etc and start, you need to go into the ambari
configuration and add it to the parsers list. ( if you don’t want to use
rest, you could install the extension assembly, then go into ambari and add
it to the list and restart everything as well ).

There is a lot to review here, and it has not been reviewed yet. I expect
there to be a lot of great feedback and possible changes however.

There has been a few people on the list asking around this type of thing.
Hopefully we are close to having it go through review and revise to get it
going.

Do you have any questions I may try to answer?

On May 9, 2017 at 12:00:44, Christian Tramnitz (tramnitz@trasec.de) wrote:

Correct me if I’m wrong, but we currently have a lot of hard-coded
configuration files with the standard telemetry (yaf, bro, snort) included,
but no consistent way to add additional telemetry configuration where
needed?!
I.e. we have
-
/usr/metron/<version>/config/zookeeper/[enrichments|parsers|indexing]/<telemetry-name>.json
for topology configuration on the FS and a matching layout in zk
- ES templates
- Kibana configuration (both for index names and within some searches and
vis elements)
- Kafka topic creation and configuration
The above exist for yaf, bro and snort only and it’s hard to get an
additional telemetry source added consistenly. I.e. if I make manual
adjustments to ES templates and Kibana configuration any “publish
templates” action from Ambari will overwrite those changes.

Wouldn’t it make sense to “register” telemetry sources centrally and then
have the required configuration derived from there? Instead of starting
with an exported/imported .kibana index it could be dynamically created (at
runtime, not at build time), keeping user-defined values and only
overwriting specific configuration statements.

Is metron_service.py (METRON-777) supposed to solve some of those issues?
If yes, would every telemetry source be treated as plugin?

BR,
Christian