You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2017/10/09 13:39:15 UTC
metron git commit: METRON-1235: Document the properties pulled from
the global configuration closes apache/incubator-metron#791
Repository: metron
Updated Branches:
refs/heads/master 830e5ccf4 -> b47c9a572
METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/b47c9a57
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/b47c9a57
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/b47c9a57
Branch: refs/heads/master
Commit: b47c9a57260faa006173980a765cd565af7f12dd
Parents: 830e5cc
Author: cstella <ce...@gmail.com>
Authored: Mon Oct 9 09:39:04 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Mon Oct 9 09:39:04 2017 -0400
----------------------------------------------------------------------
metron-platform/metron-common/README.md | 27 ++++++++++++++++++-
metron-platform/metron-elasticsearch/README.md | 29 +++++++++++++++++++++
metron-platform/metron-enrichment/README.md | 21 ++++++++++++++-
metron-platform/metron-indexing/README.md | 13 +++++++++
metron-platform/metron-parsers/README.md | 17 +++++++++++-
5 files changed, 104 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index 54738f8..5f9fec6 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -47,7 +47,7 @@ This configuration is stored in zookeeper, but looks something like
"es.ip": "node1",
"es.port": "9300",
"es.date.format": "yyyy.MM.dd.HH",
- "parser.error.topic": "indexing"
+ "parser.error.topic": "indexing",
"fieldValidations" : [
{
"input" : [ "ip_src_addr", "ip_dst_addr" ],
@@ -60,6 +60,31 @@ This configuration is stored in zookeeper, but looks something like
}
```
+Various parts of our stack uses the global config are documented throughout the Metron documentation,
+but a convenient index is provided here:
+
+| Property Name | Subsystem | Type | Ambari Property |
+|---------------------------------------------------------------------------------------------------------------------|---------------|------------|----------------------------|
+| [`es.clustername`](../metron-elasticsearch#esclustername) | Indexing | String | `es_cluster_name` |
+| [`es.ip`](../metron-elasticsearch#esip) | Indexing | String | `es_hosts` |
+| [`es.port`](../metron-elasticsearch#esport) | Indexing | String | `es_port` |
+| [`es.date.format`](../metron-elasticsearch#esdateformat) | Indexing | String | `es_date_format` |
+| [`fieldValidations`](#validation-framework) | Parsing | Object | N/A |
+| [`parser.error.topic`](../metron-parsers#parsererrortopic) | Parsing | String | N/A |
+| [`stellar.function.paths`](../../metron-stellar/stellar-common#stellarfunctionpaths) | Stellar | CSV String | N/A |
+| [`stellar.function.resolver.includes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes) | Stellar | CSV String | N/A |
+| [`stellar.function.resolver.excludes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes) | Stellar | CSV String | N/A |
+| [`profiler.period.duration`](../../metron-analytics/metron-profiler#profilerperiodduration) | Profiler | Integer | `profiler_period_duration` |
+| [`profiler.period.duration.units`](../../metron-analytics/metron-profiler#profilerperioddurationunits) | Profiler | String | `profiler_period_units` |
+| [`update.hbase.table`](../metron-indexing#updatehbasetable) | REST/Indexing | String | `update_hbase_table` |
+| [`update.hbase.cf`](../metron-indexing#updatehbasecf) | REST/Indexing | String | `update_hbase_cf` |
+| [`geo.hdfs.file`](../metron-enrichment#geohdfsfile) | Enrichment | String | `geo_hdfs_file` |
+
+## Note Configs in Ambari
+If a field is managed via ambari, you should change the field via
+ambari. Otherwise, upon service restarts, you may find your update
+overwritten.
+
# Validation Framework
Inside of the global configuration, there is a validation framework in
http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-elasticsearch/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-elasticsearch/README.md b/metron-platform/metron-elasticsearch/README.md
index 2d2b139..1e39691 100644
--- a/metron-platform/metron-elasticsearch/README.md
+++ b/metron-platform/metron-elasticsearch/README.md
@@ -4,6 +4,35 @@
Elasticsearch can be used as the real-time portion of the datastore resulting from [metron-indexing](../metron-indexing.README.md).
+## Properties
+
+### `es.clustername`
+
+The name of the elasticsearch Cluster. See [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name)
+
+### `es.ip`
+
+Specifies the nodes in the elasticsearch cluster to use for writing.
+The format is one of the following:
+* A hostname or IP address with a port (e.g. `hostname1:1234`), in which case `es.port` is ignored.
+* A hostname or IP address without a port (e.g. `hostname1`), in which case `es.port` is used.
+* A string containing a CSV of hostnames without ports (e.g. `hostname1,hostname2,hostname3`) without spaces between. `es.port` is assumed to be the port for each host.
+* A string containing a CSV of hostnames with ports (e.g. `hostname1:1234,hostname2:1234,hostname3:1234`) without spaces between. `es.port` is ignored.
+* A list of hostnames with ports (e.g. `[ "hostname1:1234", "hostname2:1234"]`). Note, `es.port` is NOT used in this construction.
+
+### `es.port`
+
+The port for the elasticsearch hosts. This will be used in accordance with the discussion of `es.ip`.
+
+### `es.date.format`
+
+The date format to use when constructing the indices. For every message, the date format will be applied
+to the current time and that will become the last part of the index name where the message is written to.
+
+For instance, an `es.date.format` of `yyyy.MM.dd.HH` would have the consequence that the indices would
+roll hourly, whereas an `es.date.format` of `yyyy.MM.dd` would have the consequence that the indices would
+roll daily.
+
## Using Metron with Elasticsearch 2.x
With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See [Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields) for more information
http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-enrichment/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md
index 10f2cd4..70bf832 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -25,9 +25,27 @@ defined by JSON documents stored in zookeeper.
There are two types of configurations at the moment, `global` and
`sensor` specific.
+
## Global Configuration
-See the "[Global Configuration](../metron-common)" section.
+There are a few enrichments which have independent configurations, such
+as from the global config.
+
+Also, see the "[Global Configuration](../metron-common)" section for
+more discussion of the global config.
+
+### GeoIP
+Metron supports enrichment of IP information using
+[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The
+location of the file is managed in the global config.
+
+#### `geo.hdfs.file`
+
+The location on HDFS of the GeoLite2 database file to use for GeoIP
+lookups. This file will be localized on the storm supervisors running
+the topology and used from there. This is lazy, so if this property
+changes in a running topology, the file will be localized from HDFS upon first
+time the file is used via the geo enrichment.
## Sensor Enrichment Configuration
@@ -269,6 +287,7 @@ An example configuration for the YAF sensor is as follows:
ThreatIntel alert levels are emitted as a new field "threat.triage.level." So for the example above, an incoming message that trips the `ip_src_addr` rule will have a new field threat.triage.level=10.
+
# Example Enrichment via Stellar
Let's walk through doing a simple enrichment using Stellar on your cluster using the Squid topology.
http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index 6f47507..6dbcb98 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -150,6 +150,19 @@ in parallel. This enables a flexible strategy for specifying your backing store
For instance, currently the REST API supports the update functionality and may be configured with a list of
IndexDao implementations to use to support the updates.
+### The `HBaseDao`
+
+Updates will be written to HBase. The key structure is the GUID and
+for each new version, a new column is created with value as the message.
+
+The HBase table and column family are configured via fields in the global configuration.
+
+#### `update.hbase.table`
+The HBase table to use for message updates.
+
+#### `update.hbase.cf`
+The HBase column family to use for message updates.
+
### The `MetaAlertDao`
The goal of meta alerts is to be able to group together a set of alerts while being able to transparently perform actions
http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-parsers/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md
index 01eae71..75559c0 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -76,7 +76,22 @@ So putting it all together a typical Metron message with all 5-tuple fields pres
## Global Configuration
-See the "[Global Configuration](../metron-common)" section.
+There are a few properties which can be managed in the global configuration that have pertinence to
+parsers and parsing in general.
+
+### `parser.error.topic`
+
+The topic where messages which were unable to be parsed due to error are sent.
+Error messages will be indexed under a sensor type of `error` and the messages will have
+the following fields:
+* `sensor.type`: `error`
+* `failed_sensor_type` : The sensor type of the message which wasn't able to be parsed
+* `error_type` : The error type, in this case `parser`.
+* `stack` : The stack trace of the error
+* `hostname` : The hostname of the node where the error happened
+* `raw_message` : The raw message in string form
+* `raw_message_bytes` : The raw message bytes
+* `error_hash` : A hash of the error message
## Parser Configuration