You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2017/02/10 19:50:15 UTC
[1/2] incubator-metron git commit: METRON-710 Rev MPack Version to
0.3.1.0 (justinleet) closes apache/incubator-metron#448
Repository: incubator-metron
Updated Branches:
refs/heads/Metron_0.3.1 1d632646e -> f6c253f44
METRON-710 Rev MPack Version to 0.3.1.0 (justinleet) closes apache/incubator-metron#448
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/9dcec28a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/9dcec28a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/9dcec28a
Branch: refs/heads/Metron_0.3.1
Commit: 9dcec28a0890a5f3996e33a7ec04492491148e2d
Parents: 1d63264
Author: justinleet <ju...@gmail.com>
Authored: Fri Feb 10 09:22:48 2017 -0500
Committer: leet <le...@apache.org>
Committed: Fri Feb 10 09:22:48 2017 -0500
----------------------------------------------------------------------
metron-deployment/packaging/ambari/metron-mpack/pom.xml | 2 +-
.../packaging/ambari/metron-mpack/src/main/resources/mpack.json | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9dcec28a/metron-deployment/packaging/ambari/metron-mpack/pom.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/pom.xml b/metron-deployment/packaging/ambari/metron-mpack/pom.xml
index 2520a81..00965d4 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/pom.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/pom.xml
@@ -20,7 +20,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.metron.packaging.mpacks</groupId>
<artifactId>metron_mpack</artifactId>
- <version>1.0.0.0-SNAPSHOT</version>
+ <version>0.3.1.0</version>
<name>Metron Ambari Management Pack</name>
<parent>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/9dcec28a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/mpack.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/mpack.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/mpack.json
index 5643d71..c675410 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/mpack.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/mpack.json
@@ -1,7 +1,7 @@
{
"type": "full-release",
"name": "metron-ambari.mpack",
- "version": "1.0.0.0",
+ "version": "0.3.1.0",
"description": "Ambari Management Pack for Apache Metron",
"prerequisites": {
"min-ambari-version": "2.4.0.0",
[2/2] incubator-metron git commit: METRON-708: Update metron
documentation closes apache/incubator-metron#447
Posted by ce...@apache.org.
METRON-708: Update metron documentation closes apache/incubator-metron#447
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/f6c253f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/f6c253f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/f6c253f4
Branch: refs/heads/Metron_0.3.1
Commit: f6c253f44fe7ca5a6b31c46075f537940751c934
Parents: 9dcec28
Author: JonZeolla <ze...@gmail.com>
Authored: Fri Feb 10 14:45:26 2017 -0500
Committer: cstella <ce...@gmail.com>
Committed: Fri Feb 10 14:45:26 2017 -0500
----------------------------------------------------------------------
.../metron-profiler-client/README.md | 14 +++---
metron-analytics/metron-profiler/README.md | 48 ++++++++++----------
metron-deployment/README.md | 10 ++--
metron-deployment/amazon-ec2/README.md | 3 +-
metron-deployment/packer-build/README.md | 6 +--
.../vagrant/codelab-platform/README.md | 2 +-
.../vagrant/full-dev-platform/README.md | 8 ++--
.../vagrant/quick-dev-platform/README.md | 2 +-
metron-platform/metron-common/README.md | 17 +++----
site-book/bin/generate-md.sh | 2 +-
10 files changed, 55 insertions(+), 57 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-analytics/metron-profiler-client/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/README.md b/metron-analytics/metron-profiler-client/README.md
index 7260336..94da919 100644
--- a/metron-analytics/metron-profiler-client/README.md
+++ b/metron-analytics/metron-profiler-client/README.md
@@ -4,7 +4,7 @@ This project provides a client API for accessing the profiles generated by the [
## Stellar Client API
-The following are usage examples that show how the Stellar API can be used to read profiles generated by the [Metron Profiler](../metron-profiler). This API would be used in conjunction with other Stellar functions like [`MAAS_MODEL_APPLY`](../../metron-platform/metron-common) to perform model scoring on streaming data.
+The following are usage examples that show how the Stellar API can be used to read profiles generated by the [Metron Profiler](../metron-profiler). This API would be used in conjunction with other Stellar functions like [`MAAS_MODEL_APPLY`](../../metron-platform/metron-common#maas_model_apply) to perform model scoring on streaming data.
These examples assume a profile has been defined called 'snort-alerts' that tracks the number of Snort alerts associated with an IP address over time. The profile definition might look similar to the following.
@@ -83,13 +83,13 @@ when historical profiles have been created with a different Profiler configurati
want to change the global Client configuration so as not to disrupt the work of other analysts working with current profiles.
| Key | Description | Required | Default |
-| ------------------------------------- | -------- | -------- | -------- |
-| profiler.client.period.duration | The duration of each profile period. This value should be defined along with `profiler.client.period.duration.units`. | Optional | 15 |
+| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | -------- | -------- |
+| profiler.client.period.duration | The duration of each profile period. This value should be defined along with `profiler.client.period.duration.units`. | Optional | 15 |
| profiler.client.period.duration.units | The units used to specify the profile period duration. This value should be defined along with `profiler.client.period.duration`. | Optional | MINUTES |
-| profiler.client.hbase.table | The name of the HBase table used to store profile data. | Optional | profiler |
-| profiler.client.hbase.column.family | The name of the HBase column family used to store profile data. | Optional | P |
-| profiler.client.salt.divisor | The salt divisor used to store profile data. | Optional | 1000 |
-| hbase.provider.impl | The name of the HBaseTableProvider implementation class. | Optional | |
+| profiler.client.hbase.table | The name of the HBase table used to store profile data. | Optional | profiler |
+| profiler.client.hbase.column.family | The name of the HBase column family used to store profile data. | Optional | P |
+| profiler.client.salt.divisor | The salt divisor used to store profile data. | Optional | 1000 |
+| hbase.provider.impl | The name of the HBaseTableProvider implementation class. | Optional | |
### Errors
The most common result of incorrect PROFILE_GET arguments or Client configuration parameters is an empty result set, rather than an error. The Client cannot effectively validate the arguments, because the Profiler configuration parameters may be changed and the profile itself does not store them. The person doing the querying must carry forward the knowledge of the Profiler configuration parameters from the time of profile creation, and use corresponding PROFILE_GET arguments and Client configuration parameters when querying the data.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-analytics/metron-profiler/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md
index dfff277..9768c07 100644
--- a/metron-analytics/metron-profiler/README.md
+++ b/metron-analytics/metron-profiler/README.md
@@ -80,16 +80,16 @@ The Profiler specification requires a JSON-formatted set of elements, many of wh
The specification for the Profiler topology is stored in Zookeeper at `/metron/topology/profiler`. These properties also exist in the local filesystem at `$METRON_HOME/config/zookeeper/profiler.json`.
The values can be changed on disk and then uploaded to Zookeeper using `$METRON_HOME/bin/zk_load_configs.sh`.
-| Name | | Description
-|--- |--- |---
-| [profile](#profile) | Required | Unique name identifying the profile.
-| [foreach](#foreach) | Required | A separate profile is maintained "for each" of these.
-| [onlyif](#onlyif) | Optional | Boolean expression that determines if a message should be applied to the profile.
-| [groupBy](#groupby) | Optional | One or more Stellar expressions used to group the profile measurements when persisted.
-| [init](#init) | Optional | One or more expressions executed at the start of a window period.
-| [update](#update) | Required | One or more expressions executed when a message is applied to the profile.
-| [result](#result) | Required | A Stellar expression that is executed when the window period expires.
-| [expires](#expires) | Optional | Profile data is purged after this period of time, specified in milliseconds.
+| Name | Required | Description |
+| --------------------- | -------- | --------------------------------------------------------------------------------------- |
+| [profile](#profile) | Required | Unique name identifying the profile. |
+| [foreach](#foreach) | Required | A separate profile is maintained "for each" of these. |
+| [onlyif](#onlyif) | Optional | Boolean expression that determines if a message should be applied to the profile. |
+| [groupBy](#groupby) | Optional | One or more Stellar expressions used to group the profile measurements when persisted. |
+| [init](#init) | Optional | One or more expressions executed at the start of a window period. |
+| [update](#update) | Required | One or more expressions executed when a message is applied to the profile. |
+| [result](#result) | Required | A Stellar expression that is executed when the window period expires. |
+| [expires](#expires) | Optional | Profile data is purged after this period of time, specified in milliseconds. |
### `profile`
@@ -166,20 +166,20 @@ A numeric value that defines how many days the profile data is retained. After
The Profiler runs as an independent Storm topology. The configuration for the Profiler topology is stored in local filesystem at `$METRON_HOME/config/profiler.properties`.
The values can be changed on disk and then the Profiler topology must be restarted.
-| Setting | Description |
-|--- |--- |
-| profiler.workers | The number of worker processes to create for the topology. |
-| profiler.executors | The number of executors to spawn per component. |
-| profiler.input.topic | The name of the Kafka topic from which to consume data. |
-| profiler.period.duration | The duration of each profile period. This value should be defined along with `profiler.period.duration.units`. |
-| profiler.period.duration.units | The units used to specify the `profiler.period.duration`. |
-| profiler.ttl | If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with `profiler.ttl.units`. |
-| profiler.ttl.units | The units used to specify the `profiler.ttl`. |
-| profiler.hbase.salt.divisor | A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. |
-| profiler.hbase.table | The name of the HBase table that profiles are written to. |
-| profiler.hbase.column.family | The column family used to store profiles. |
-| profiler.hbase.batch | The number of puts that are written in a single batch. |
-| profiler.hbase.flush.interval.seconds | The maximum number of seconds between batch writes to HBase. |
+| Setting | Description |
+| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| profiler.workers | The number of worker processes to create for the topology. |
+| profiler.executors | The number of executors to spawn per component. |
+| profiler.input.topic | The name of the Kafka topic from which to consume data. |
+| profiler.period.duration | The duration of each profile period. This value should be defined along with `profiler.period.duration.units`. |
+| profiler.period.duration.units | The units used to specify the `profiler.period.duration`. |
+| profiler.ttl | If a message has not been applied to a Profile in this period of time, the Profile will be forgotten and its resources will be cleaned up. This value should be defined along with `profiler.ttl.units`. |
+| profiler.ttl.units | The units used to specify the `profiler.ttl`. |
+| profiler.hbase.salt.divisor | A salt is prepended to the row key to help prevent hotspotting. This constant is used to generate the salt. Ideally, this constant should be roughly equal to the number of nodes in the Hbase cluster. |
+| profiler.hbase.table | The name of the HBase table that profiles are written to. |
+| profiler.hbase.column.family | The column family used to store profiles. |
+| profiler.hbase.batch | The number of puts that are written in a single batch. |
+| profiler.hbase.flush.interval.seconds | The maximum number of seconds between batch writes to HBase. |
After altering the configuration, start the Profiler.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 46a5f8d..b973353 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -8,9 +8,9 @@ In addition, an Ambari Management Pack can be built which can be deployed in con
## Prerequisites
The following tools are required to run these scripts:
-- Maven - https://maven.apache.org/
-- Git - https://git-scm.com/
-- Ansible - http://www.ansible.com/ (version 2.0 or greater)
+- [Maven](https://maven.apache.org/)
+- [Git](https://git-scm.com/)
+- [Ansible](http://www.ansible.com/) (version 2.0 or greater)
Currently Metron must be built from source. Before running these scripts perform the following steps:
@@ -86,7 +86,7 @@ Navigate to `incubator-metron/metron-deployment/playbooks` and run:
A VagrantFile is included and will install a working version of the entire Metron stack. The following is required to
run this:
-- Vagrant - https://www.vagrantup.com/
+- [Vagrant](https://www.vagrantup.com/)
- Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is
installed
@@ -133,7 +133,7 @@ Notably, the URL for the GeoIP database that is preloaded (and is prefilled by d
After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates. Similar to this, a custom Kibana action to Load Template is available.
-Another custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing README.md](../metron-platform/metron-indexing/README.md)
+Another custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing)
#### Offline installation
Currently there is only one point that would reach out to the internet during an install. This is the URL for the GeoIP database information.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-deployment/amazon-ec2/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md
index 18c1e91..5f1cb38 100644
--- a/metron-deployment/amazon-ec2/README.md
+++ b/metron-deployment/amazon-ec2/README.md
@@ -16,7 +16,7 @@ The host used to deploy Apache Metron will need the following software tools ins
- Python 2.7.11
- Maven 3.3.9
-Any platform that supports these tools is suitable, but the following instructions cover only Mac OS X. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
+Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
1. Install Homebrew by running the following command in a terminal. Refer to the [Homebrew](http://brew.sh/) home page for the latest installation instructions.
@@ -27,6 +27,7 @@ Any platform that supports these tools is suitable, but the following instructio
2. With Homebrew installed, run the following command in a terminal to install all of the required tools.
```
+ brew cask install java
brew install maven git
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ee1273bf919a5e4e50838513a9e55ea423e1d7ce/Formula/ansible.rb
brew switch ansible 2.0.0.2
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-deployment/packer-build/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packer-build/README.md b/metron-deployment/packer-build/README.md
index 4c75287..1100d5f 100644
--- a/metron-deployment/packer-build/README.md
+++ b/metron-deployment/packer-build/README.md
@@ -10,9 +10,9 @@ Images Provided
Prerequisites
---------------------
-- Packer 0.10.1 https://www.packer.io/
-- Virtualbox 5.0.16 https://www.virtualbox.org/
-- Be sure to build Metron prior to building the images- (cd *your-project-directory*/metron-platform ; mvn clean package -DskipTests)
+- [Packer](https://www.packer.io/) 0.10.1
+- [Virtualbox](https://www.virtualbox.org/) 5.0.16
+- Be sure to build Metron prior to building the images (cd *your-project-directory*/metron-platform && mvn clean package -DskipTests)
Build Both Images
----------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-deployment/vagrant/codelab-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/codelab-platform/README.md b/metron-deployment/vagrant/codelab-platform/README.md
index 577eaca..b1da488 100644
--- a/metron-deployment/vagrant/codelab-platform/README.md
+++ b/metron-deployment/vagrant/codelab-platform/README.md
@@ -12,7 +12,7 @@ As with the Singlenode Full Image, the computer used to deploy Apache Metron wil
- [Ansible](https://github.com/ansible/ansible) 2.0.0.2
- [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Virtualbox](virtualbox.org) 5.0.16
+ - [Virtualbox](https://virtualbox.org) 5.0.16
- Python 2.7.11
- Maven 3.3.9
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-deployment/vagrant/full-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/README.md b/metron-deployment/vagrant/full-dev-platform/README.md
index 5108448..03b2904 100644
--- a/metron-deployment/vagrant/full-dev-platform/README.md
+++ b/metron-deployment/vagrant/full-dev-platform/README.md
@@ -14,20 +14,20 @@ The computer used to deploy Apache Metron will need to have the following compon
- [Ansible](https://github.com/ansible/ansible) 2.0.0.2
- [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Virtualbox](virtualbox.org) 5.0.16
+ - [Virtualbox](https://virtualbox.org) 5.0.16
- Python 2.7.11
- Maven 3.3.9
-#### OS X
+#### macOS
-Any platform that supports these tools is suitable, but the following instructions cover installation on Mac OS X. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
+Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.
1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
2. Run the following command in a terminal to install all of the required tools.
```
- brew cask install vagrant virtualbox
+ brew cask install vagrant virtualbox java
brew install maven git
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ee1273bf919a5e4e50838513a9e55ea423e1d7ce/Formula/ansible.rb
brew switch ansible 2.0.0.2
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-deployment/vagrant/quick-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/quick-dev-platform/README.md b/metron-deployment/vagrant/quick-dev-platform/README.md
index 5e2442d..1015563 100644
--- a/metron-deployment/vagrant/quick-dev-platform/README.md
+++ b/metron-deployment/vagrant/quick-dev-platform/README.md
@@ -16,7 +16,7 @@ As with the Full Development Platform (`metron-deployment/vagrant/full-dev-platf
- [Ansible](https://github.com/ansible/ansible) 2.0.0.2
- [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Virtualbox](virtualbox.org) 5.0.16
+ - [Virtualbox](https://virtualbox.org) 5.0.16
- Python 2.7.11
- Maven 3.3.9
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index fbf3b50..872b320 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -32,15 +32,12 @@ The query language supports the following:
## Stellar Language Keywords
The following keywords need to be single quote escaped in order to be used in Stellar expressions:
-| | | |
-| :---: | :---: | :---: |
-| not| else | exists |
-| if | then | and |
-| or | == | != | < |
-| <= | \> | \>= |
-| ? | \+ | \- |
-| , | \* | / |
-| | \* | / |
+| | | | | |
+| :-----------: | :-----------: | :---------: | :---------: | :---------: |
+| not | else | exists | if | then |
+| and | or | == | != | \< |
+| \<= | \> | \>= | \+ | \- |
+| ? | \* | / | , | |
Using parens such as: "foo" : "\<ok\>" requires escaping; "foo": "\'\<ok\>\'"
@@ -96,7 +93,7 @@ The `!=` operator is the negation of the above.
| [ `ENRICHMENT_GET`](#enrichment_get) |
| [ `FILL_LEFT`](#fill_left) |
| [ `FILL_RIGHT`](#fill_right) |
-| [ `FORMAT`](#format) |
+| [ `FORMAT`](#format) |
| [ `HLLP_CARDINALITY`](../../metron-analytics/metron-statistics#hllp_cardinality) |
| [ `HLLP_INIT`](../../metron-analytics/metron-statistics#hllp_init) |
| [ `HLLP_MERGE`](../../metron-analytics/metron-statistics#hllp_merge) |
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/f6c253f4/site-book/bin/generate-md.sh
----------------------------------------------------------------------
diff --git a/site-book/bin/generate-md.sh b/site-book/bin/generate-md.sh
index df97c76..623e141 100755
--- a/site-book/bin/generate-md.sh
+++ b/site-book/bin/generate-md.sh
@@ -196,7 +196,7 @@ for (( i=0; i<${#HREF_REWRITE_LIST[@]} ; i+=2 )) ; do
sed -i -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}" "${HREF_REWRITE_LIST[$i]}"
;;
darwin*)
- # Mac OS X sed needs an empty-string argument after -i option to get the same result
+ # MacOS sed needs an empty-string argument after -i option to get the same result
sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}" "${HREF_REWRITE_LIST[$i]}"
;;
*)