You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by mm...@apache.org on 2018/07/11 01:32:46 UTC
[30/50] [abbrv] metron git commit: METRON-1629 Update Solr
documentation (merrimanr via justinleet) closes apache/metron#1072
METRON-1629 Update Solr documentation (merrimanr via justinleet) closes apache/metron#1072
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/6159c6f9
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/6159c6f9
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/6159c6f9
Branch: refs/heads/feature/METRON-1554-pcap-query-panel
Commit: 6159c6f997851fabe1b3c21cc0e06bf49c49d9f7
Parents: ed4dfb9
Author: merrimanr <me...@gmail.com>
Authored: Fri Jun 22 07:48:27 2018 -0400
Committer: leet <le...@apache.org>
Committed: Fri Jun 22 07:48:27 2018 -0400
----------------------------------------------------------------------
metron-interface/metron-alerts/README.md | 3 ++-
metron-interface/metron-rest/README.md | 1 +
metron-platform/metron-indexing/README.md | 14 ++++++++++++++
metron-platform/metron-parsers/README.md | 2 ++
metron-platform/metron-solr/README.md | 26 ++++++++++++++++++++++++++
5 files changed, 45 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-interface/metron-alerts/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-alerts/README.md b/metron-interface/metron-alerts/README.md
index 2d34505..486a885 100644
--- a/metron-interface/metron-alerts/README.md
+++ b/metron-interface/metron-alerts/README.md
@@ -40,7 +40,8 @@ Sorting has a similar caveat, in that if we are matching on multiple alerts, the
Alerts that are contained in a a meta alert are generally excluded from search results, because a user has already grouped them in a meaningful way.
## Prerequisites
-* The Metron REST application should be up and running and Elasticsearch should have some alerts populated by Metron topologies
+* The Metron REST application should be up and running
+* Elasticsearch or Solr should have some alerts populated by Metron topologies, depending on which real-time store is enabled
* The Management UI should be installed (which includes [Express](https://expressjs.com/))
* The alerts can be populated using Full Dev or any other setup
* UI is developed using angular4 and uses angular-cli
http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index 2a6a0e0..44594f7 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -22,6 +22,7 @@ This module provides a RESTful API for interacting with Metron.
## Prerequisites
* A running Metron cluster
+* A running real-time store, either Elasticsearch or Solr depending on which one is enabled
* Java 8 installed
* Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
* A relational database
http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index 5a35d62..46e511b 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -77,6 +77,20 @@ Alerts can be grouped, after appropriate searching, into a set of alerts called
### Elasticsearch
Metron comes with built-in templates for the default sensors for Elasticsearch. When adding a new sensor, it will be necessary to add a new template defining the output fields appropriately. In addition, there is a requirement for a field `alert` of type `nested` for Elasticsearch 2.x installs. This is detailed at [Using Metron with Elasticsearch 2.x](../metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x)
+### Solr
+
+Metron comes with built-in schemas for the default sensors for Solr. When adding a new sensor, it will be necessary to add a new schema defining the output fields appropriately. In addition, these fields are used internally by Metron and also required:
+
+* `<field name="guid" type="string" indexed="true" stored="true" required="true" multiValued="false" />`
+* `<field name="source.type" type="string" indexed="true" stored="true" />`
+* `<field name="timestamp" type="timestamp" indexed="true" stored="true" />`
+* `<field name="comments" type="string" indexed="true" stored="true" multiValued="true"/>`
+* `<field name="metaalerts" type="string" multiValued="true" indexed="true" stored="true"/>`
+
+The unique key should be set to `guid` by including `<uniqueKey>guid</uniqueKey>` in the schema.
+
+It is strongly suggested the `fieldTypes` match those in the built-in schemas.
+
### Indexing Configuration Examples
For a given sensor, the following scenarios would be indicated by
the following cases:
http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-platform/metron-parsers/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md
index 8254baf..d79b9ce 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -561,6 +561,8 @@ it is necessary to add an additional field to the templates and mapping for exis
Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](../../metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x)
+If Solr is selected as the real-time store, it is also necessary to add additional fields. See the [Solr](../metron-indexing#solr) section in metron-indexing for more details.
+
## Kafka Queue
The kafka queue associated with your parser is a collection point for
all of the data sent to your parser. As such, make sure that the number of partitions in
http://git-wip-us.apache.org/repos/asf/metron/blob/6159c6f9/metron-platform/metron-solr/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-solr/README.md b/metron-platform/metron-solr/README.md
index 159779c..ca90c73 100644
--- a/metron-platform/metron-solr/README.md
+++ b/metron-platform/metron-solr/README.md
@@ -92,6 +92,32 @@ Navigate to `$METRON_HOME/bin` and spin up Solr Cloud by running `install_solr.s
Elasticsearch and Kibana will have been stopped and you should now have an instance of Solr Cloud up and running at http://localhost:8983/solr/#/~cloud. This manner of starting Solr
will also spin up an embedded Zookeeper instance at port 9983. More information can be found [here](https://lucene.apache.org/solr/guide/6_6/getting-started-with-solrcloud.html)
+Solr can also be installed using [HDP Search 3](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_solr-search-installation/content/ch_hdp_search_30.html). HDP Search 3 sets the Zookeeper root to
+`/solr` so this will need to be added to each url in the comma-separated list in Ambari UI -> Services -> Metron -> Configs -> Index Settings -> Solr Zookeeper Urls. For example, in full dev
+this would be `node1:2181/solr`.
+
+## Enabling Solr
+
+Elasticsearch is the real-time store used by default in Metron. Solr can be enabled following these steps:
+
+1. Stop the Metron Indexing component in Ambari.
+1. Update Ambari UI -> Services -> Metron -> Configs -> Index Settings -> Solr Zookeeper Urls to match the Solr installation described in the previous section.
+1. Change Ambari UI -> Services -> Metron -> Configs -> Indexing -> Index Writer - Random Access -> Random Access Search Engine to `Solr`.
+1. Set the `source.type.field` property to `source.type` in the [Global Configuration](../metron-common#global-configuration).
+1. Set the `threat.triage.score.field` property to `threat.triage.score` in the [Global Configuration](../metron-common#global-configuration).
+1. Start the Metron Indexing component in Ambari.
+1. Restart Metron REST and the Alerts UI in Ambari.
+
+This will automatically create collections for the schemas shipped with Metron:
+
+* bro
+* snort
+* yaf
+* error (used internally by Metron)
+* metaalert (used internall by Metron)
+
+Any other collections must be created manually before starting the Indexing component. Alerts should be present in the Alerts UI after enabling Solr.
+
## Schemas
As of now, we have mapped out the Schemas in `src/main/config/schema`.