You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ce...@apache.org on 2016/02/22 18:33:42 UTC
[3/4] incubator-metron git commit: METRON-41 Integrate Apache Rat to
Audit Source Code Licensing (nickwallen via cestella) closes
apache/incubator-metron#27
METRON-41 Integrate Apache Rat to Audit Source Code Licensing (nickwallen via cestella) closes apache/incubator-metron#27
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/244a5359
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/244a5359
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/244a5359
Branch: refs/heads/master
Commit: 244a5359b2333e4e5ab81dc64098affb2beed26f
Parents: bab4d3e
Author: nickwallen <ni...@nickallen.org>
Authored: Mon Feb 22 12:33:03 2016 -0500
Committer: cstella <ce...@gmail.com>
Committed: Mon Feb 22 12:33:03 2016 -0500
----------------------------------------------------------------------
bro-plugin-kafka/CHANGES | 17 ++
bro-plugin-kafka/CMakeLists.txt | 18 ++
bro-plugin-kafka/src/kafka.bif | 17 ++
bro-plugin-kafka/tests/btest.cfg | 17 ++
bro-plugin-kafka/tests/kafka/show-plugin.bro | 17 ++
.../inventory/metron_example/group_vars/all | 18 ++
deployment/inventory/metron_example/hosts | 20 +-
.../inventory/singlenode-vagrant/group_vars/all | 18 ++
deployment/inventory/singlenode-vagrant/hosts | 19 +-
.../elasticsearch/files/elasticsearch.repo | 20 +-
deployment/roles/pcap_replay/files/pcap-replay | 18 ++
deployment/roles/snort/files/snort.conf | 32 +--
metron-streaming/Metron-Alerts/README.md | 104 +++++++
.../Metron-Alerts/pom.xml.versionsBackup | 99 -------
metron-streaming/Metron-Alerts/readme.md | 104 -------
.../resources/config/AllAlertAdapterTest.config | 19 +-
.../Metron-Common/pom.xml.versionsBackup | 131 ---------
.../org/apache/metron/ise/parser/ISEParser.jj | 4 +-
.../config/BasicTldExtractorTest.config | 18 ++
.../Metron-DataLoads/pom.xml.versionsBackup | 97 -------
.../src/main/assembly/assembly.xml | 13 +
metron-streaming/Metron-DataServices/README.md | 17 +-
.../Metron-DataServices/conf/config.properties | 20 +-
.../main/resources/config-definition-hbase.xml | 28 +-
.../resources/hbase-config-default.properties | 19 +-
.../src/main/resources/log4j.xml | 38 ++-
.../src/main/resources/webroot/logged_in.jsp | 28 +-
.../src/main/resources/webroot/login_failed.jsp | 26 +-
.../src/main/resources/webroot/login_old.jsp | 31 ++-
.../src/main/resources/webroot/withsocket.jsp | 64 +++--
.../src/main/resources/webroot/withsocket2.jsp | 50 ++--
.../src/main/resources/webroot/withsocket3.jsp | 64 +++--
.../Metron-EnrichmentAdapters/README.md | 125 +++++++++
.../pom.xml.versionsBackup | 136 ----------
.../Metron-EnrichmentAdapters/readme.md | 125 ---------
.../Metron-FlumeConfigs/SampleFlumeConf.rtf | 43 ---
metron-streaming/Metron-Indexing/README.md | 61 +++++
.../Metron-Indexing/pom.xml.versionsBackup | 91 -------
metron-streaming/Metron-Indexing/readme.md | 61 -----
.../Metron-MessageParsers/README.md | 82 ++++++
.../pom.xml.versionsBackup | 74 -----
.../Metron-MessageParsers/readme.md | 82 ------
.../config/BasicFireEyeParserTest.config | 18 ++
.../resources/config/BasicIseParserTest.config | 18 ++
.../config/BasicLancopeParserTest.config | 18 ++
.../BasicPaloAltoFirewallParserTest.config | 18 ++
.../config/BasicSourcefireParserTest.config | 18 ++
.../test/resources/config/BroParserTest.config | 18 ++
.../resources/config/GrokAsaParserTest.config | 18 ++
.../Metron-Pcap_Service/pom.xml.versionsBackup | 268 -------------------
.../main/resources/config-definition-hbase.xml | 28 +-
metron-streaming/Metron-Topologies/README.md | 39 +++
.../Metron-Topologies/pom.xml.versionsBackup | 190 -------------
metron-streaming/Metron-Topologies/readme.md | 39 ---
.../etc/whitelists/known_hosts.conf | 19 +-
.../resources/TopologyConfigs_old/lancope.conf | 20 +-
metron-streaming/README.md | 30 +++
metron-streaming/pom.xml | 44 +--
metron-streaming/pom.xml.versionsBackup | 104 -------
metron-streaming/readme.md | 13 -
metron-ui/Gruntfile.js | 18 ++
metron-ui/Makefile | 18 +-
metron-ui/Vagrantfile | 17 ++
metron-ui/examples/pcap-panel/pcap-parse.html | 27 +-
metron-ui/examples/server/nginx.conf | 17 ++
metron-ui/index.js | 18 ++
metron-ui/lib/metron-ui.js | 18 ++
metron-ui/lib/modules/es-proxy.js | 18 ++
metron-ui/lib/modules/login.js | 20 +-
metron-ui/lib/modules/pcap.js | 18 ++
metron-ui/lib/views/alerts.jade | 21 +-
metron-ui/lib/views/index.jade | 19 +-
metron-ui/lib/views/login.jade | 24 +-
metron-ui/script/es_fetch | 18 ++
metron-ui/script/es_gen.js | 19 +-
metron-ui/script/es_seed | 18 ++
metron-ui/script/generate_alert | 18 ++
metron-ui/script/ldap_seed | 17 ++
metron-ui/script/migrate | 17 ++
metron-ui/script/parse | 18 ++
metron-ui/script/provision | 17 ++
metron-ui/test/metron-ui-test.js | 18 ++
metron-ui/test/session-test.js | 17 ++
pom.xml | 60 +++++
84 files changed, 1669 insertions(+), 1826 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/bro-plugin-kafka/CHANGES
----------------------------------------------------------------------
diff --git a/bro-plugin-kafka/CHANGES b/bro-plugin-kafka/CHANGES
index e69de29..3df60b0 100644
--- a/bro-plugin-kafka/CHANGES
+++ b/bro-plugin-kafka/CHANGES
@@ -0,0 +1,17 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/bro-plugin-kafka/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/bro-plugin-kafka/CMakeLists.txt b/bro-plugin-kafka/CMakeLists.txt
index 27bf7f3..f578eb2 100644
--- a/bro-plugin-kafka/CMakeLists.txt
+++ b/bro-plugin-kafka/CMakeLists.txt
@@ -1,3 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
cmake_minimum_required(VERSION 2.8)
project(Plugin)
include(BroPlugin)
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/bro-plugin-kafka/src/kafka.bif
----------------------------------------------------------------------
diff --git a/bro-plugin-kafka/src/kafka.bif b/bro-plugin-kafka/src/kafka.bif
index fef24b8..a95937f 100644
--- a/bro-plugin-kafka/src/kafka.bif
+++ b/bro-plugin-kafka/src/kafka.bif
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
module Kafka;
const kafka_broker_list: string;
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/bro-plugin-kafka/tests/btest.cfg
----------------------------------------------------------------------
diff --git a/bro-plugin-kafka/tests/btest.cfg b/bro-plugin-kafka/tests/btest.cfg
index 0578397..e42fefd 100644
--- a/bro-plugin-kafka/tests/btest.cfg
+++ b/bro-plugin-kafka/tests/btest.cfg
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
[btest]
TestDirs = kafka
TmpDir = %(testbase)s/.tmp
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/bro-plugin-kafka/tests/kafka/show-plugin.bro
----------------------------------------------------------------------
diff --git a/bro-plugin-kafka/tests/kafka/show-plugin.bro b/bro-plugin-kafka/tests/kafka/show-plugin.bro
index aef2c26..b305382 100644
--- a/bro-plugin-kafka/tests/kafka/show-plugin.bro
+++ b/bro-plugin-kafka/tests/kafka/show-plugin.bro
@@ -1,2 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
# @TEST-EXEC: bro -NN Metron::Kafka >output
# @TEST-EXEC: btest-diff output
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/inventory/metron_example/group_vars/all
----------------------------------------------------------------------
diff --git a/deployment/inventory/metron_example/group_vars/all b/deployment/inventory/metron_example/group_vars/all
index c6e7aab..a8b5bff 100644
--- a/deployment/inventory/metron_example/group_vars/all
+++ b/deployment/inventory/metron_example/group_vars/all
@@ -1,3 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
# metron variables
metron_version: 0.1BETA
java_home: /usr/jdk64/jdk1.8.0_40
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/inventory/metron_example/hosts
----------------------------------------------------------------------
diff --git a/deployment/inventory/metron_example/hosts b/deployment/inventory/metron_example/hosts
index 92160bc..2aa75cc 100644
--- a/deployment/inventory/metron_example/hosts
+++ b/deployment/inventory/metron_example/hosts
@@ -1,3 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
[ambari_master]
host
@@ -17,4 +35,4 @@ host ansible_ssh_private_key_file=path_to_key_file ansible_ssh_user=user
host ansible_ssh_private_key_file=path_to_key_file ansible_ssh_user=user
[sensors]
-host ansible_ssh_private_key_file=path_to_key_file ansible_ssh_user=user
\ No newline at end of file
+host ansible_ssh_private_key_file=path_to_key_file ansible_ssh_user=user
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/inventory/singlenode-vagrant/group_vars/all
----------------------------------------------------------------------
diff --git a/deployment/inventory/singlenode-vagrant/group_vars/all b/deployment/inventory/singlenode-vagrant/group_vars/all
index 0135acc..79f2d07 100644
--- a/deployment/inventory/singlenode-vagrant/group_vars/all
+++ b/deployment/inventory/singlenode-vagrant/group_vars/all
@@ -1,3 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
# metron variables
metron_version: 0.1BETA
java_home: /usr/jdk64/jdk1.8.0_40
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/inventory/singlenode-vagrant/hosts
----------------------------------------------------------------------
diff --git a/deployment/inventory/singlenode-vagrant/hosts b/deployment/inventory/singlenode-vagrant/hosts
index 379bdd7..cc31bb1 100644
--- a/deployment/inventory/singlenode-vagrant/hosts
+++ b/deployment/inventory/singlenode-vagrant/hosts
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
[ambari_master]
node1
@@ -17,4 +34,4 @@ node1
node1
[sensors]
-node1
\ No newline at end of file
+node1
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/roles/elasticsearch/files/elasticsearch.repo
----------------------------------------------------------------------
diff --git a/deployment/roles/elasticsearch/files/elasticsearch.repo b/deployment/roles/elasticsearch/files/elasticsearch.repo
index ed299b4..e0e4d27 100644
--- a/deployment/roles/elasticsearch/files/elasticsearch.repo
+++ b/deployment/roles/elasticsearch/files/elasticsearch.repo
@@ -1,6 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
[elasticsearch-1.7]
name=Elasticsearch repository for 1.7.x packages
baseurl=http://packages.elastic.co/elasticsearch/1.7/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
-enabled=1
\ No newline at end of file
+enabled=1
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/roles/pcap_replay/files/pcap-replay
----------------------------------------------------------------------
diff --git a/deployment/roles/pcap_replay/files/pcap-replay b/deployment/roles/pcap_replay/files/pcap-replay
index 074406d..912b7dc 100644
--- a/deployment/roles/pcap_replay/files/pcap-replay
+++ b/deployment/roles/pcap_replay/files/pcap-replay
@@ -1,8 +1,26 @@
#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
# pcap replay daemon
# chkconfig: 345 20 80
# description: Replays packet capture data stored in libpcap format
# processname: pcap-replay
+#
DAEMON_PATH="/opt/pcap-replay"
PCAPIN=`ls $DAEMON_PATH/*.pcap 2> /dev/null`
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/deployment/roles/snort/files/snort.conf
----------------------------------------------------------------------
diff --git a/deployment/roles/snort/files/snort.conf b/deployment/roles/snort/files/snort.conf
index 35757c3..260d0a4 100644
--- a/deployment/roles/snort/files/snort.conf
+++ b/deployment/roles/snort/files/snort.conf
@@ -1,26 +1,20 @@
-#--------------------------------------------------
-# VRT Rule Packages Snort.conf
#
-# For more information visit us at:
-# http://www.snort.org Snort Website
-# http://vrt-blog.snort.org/ Sourcefire VRT Blog
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
#
-# Mailing list Contact: snort-sigs@lists.sourceforge.net
-# False Positive reports: fp@sourcefire.com
-# Snort bugs: bugs@snort.org
+# http://www.apache.org/licenses/LICENSE-2.0
#
-# Compatible with Snort Versions:
-# VERSIONS : 2.9.6.2
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
#
-# Snort build options:
-# OPTIONS : --enable-gre --enable-mpls --enable-targetbased --enable-ppm --enable-perfprofiling --enable-zlib --enable-active-response --enable-normalizer --enable-reload --enable-react --enable-flexresp3
-#
-# Additional information:
-# This configuration file enables active response, to run snort in
-# test mode -T you are required to supply an interface -i <interface>
-# or test mode will fail to fully validate the configuration and
-# exit with a FATAL error
-#--------------------------------------------------
###################################################
# This file contains a sample snort configuration.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Alerts/README.md
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Alerts/README.md b/metron-streaming/Metron-Alerts/README.md
new file mode 100644
index 0000000..2b87d50
--- /dev/null
+++ b/metron-streaming/Metron-Alerts/README.md
@@ -0,0 +1,104 @@
+#Metron-Alerts
+
+##Module Description
+
+This module enables telemetry alerts. It splits the mssage stream into two streams. The original message is emitted on the "message" stream. The corresponding alert is emitted on the "alerts" stream. The two are tied together through the alerts UUID.
+
+##Message Format
+
+Assuming the original message (with enrichments enabled) has the following format:
+
+```json
+{
+"message":
+{"ip_src_addr": xxxx,
+"ip_dst_addr": xxxx,
+"ip_src_port": xxxx,
+"ip_dst_port": xxxx,
+"protocol": xxxx,
+"timestamp": xxxx.
+"original_string": xxxx,
+"additional-field 1": xxxx,
+},
+"enrichment" : {"geo": xxxx, "whois": xxxx, "hosts": xxxxx, "CIF": "xxxxx"}
+
+}
+```
+
+The telemetry message will be tagged with a UUID alert tag like so:
+
+```json
+{
+"message":
+{"ip_src_addr": xxxx,
+"ip_dst_addr": xxxx,
+"ip_src_port": xxxx,
+"ip_dst_port": xxxx,
+"protocol": xxxx,
+"timestamp": xxxx,
+"original_string": xxxx,
+"additional-field 1": xxxx,
+},
+"enrichment" : {"geo": xxxx, "whois": xxxx, "hosts": xxxxx, "CIF": "xxxxx"},
+"alerts": [UUID1, UUID2, UUID3, etc]
+
+}
+```
+
+The alert will be fired on the "alerts" stream and can be customized to have any format as long as it includes the required mandatory fields. The mandatory fields are:
+
+* timestamp (epoch): The time from the message that triggered the alert
+* description: A human friendly string representation of the alert
+* alert_id: The UUID generated for the alert. This uniquely identifies an alert
+
+There are other standard but not mandatory fields that can be leveraged by metron-ui and other alert consumers:
+
+* designated_host: The IP address that corresponds to an asset. Ex. The IP address of the company device associated with the alert.
+* enrichment: A copy of the enrichment data from the message that triggered the alert
+* priority: The priority of the alert. Mustb e set to one of HIGH, MED or LOW
+
+An example of an alert with all mandatory and standard fields would look like so:
+
+```json
+{
+"timestamp": xxxx,
+"alert_id": UUID,
+"description": xxxx,
+"designated_host": xxxx,
+"enrichment": { "geo": xxxx, "whois": xxxx, "cif": xxxx },
+"priority": "MED"
+}
+```
+
+##Alerts Bolt
+
+The bolt can be extended with a variety of alerts adapters. The ability to stack alerts is currently in beta, but is not currently advisable. We advice to only have one alerts bolt per topology. The adapters are rules-based adapters which fire alerts when rules are a match. Currently only Java adapters are provided, but there are future plans to provide Grok-Based adapters as well.
+
+The signature of the Alerts bolt is as follows:
+
+```
+TelemetryAlertsBolt alerts_bolt = new TelemetryAlertsBolt()
+.withIdentifier(alerts_identifier).withMaxCacheSize(1000)
+.withMaxTimeRetain(3600).withAlertsAdapter(alerts_adapter)
+.withMetricConfiguration(config);
+```
+Identifier - JSON key where the alert is attached
+TimeRetain & MaxCacheSize - Caching parameters for the bolt
+MetricConfiguration - export custom bolt metrics to graphite (if not null)
+AlertsAdapter - pick the appropriate adapter for generating the alerts
+
+### Java Adapters
+
+Java adapters are designed for high volume topologies, but are not easily extensible. The adapters provided are:
+
+* org.apache.metron.alerts.adapters.AllAlertsAdapter - will tag every single message with the static alert (appropriate for topologies like Sourcefire, etc, where every single message is an alert)
+* org.apache.metron.alerts.adapters.HbaseWhiteAndBlacklistAdapter - will read white and blacklists from HBase and fire alerts if source or dest IP are not on the whitelist or if any IP is on the blacklist
+* org.apache.metron.alerts.adapters.CIFAlertsAdapter - will alert on messages that have results in enrichment.cif.
+* org.apache.metron.alerts.adpaters.KeywordsAlertAdapter - will alert on messages that contain any of a list of keywords
+###Grok Adapters
+
+Grok alerts adapters for Metron are still under devleopment
+
+###Stacking Alert Adapters
+
+The functionality to stack alerts adapters is still under development
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Alerts/pom.xml.versionsBackup
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Alerts/pom.xml.versionsBackup b/metron-streaming/Metron-Alerts/pom.xml.versionsBackup
deleted file mode 100644
index 3530449..0000000
--- a/metron-streaming/Metron-Alerts/pom.xml.versionsBackup
+++ /dev/null
@@ -1,99 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?><!-- Licensed to the Apache Software
- Foundation (ASF) under one or more contributor license agreements. See the
- NOTICE file distributed with this work for additional information regarding
- copyright ownership. The ASF licenses this file to You under the Apache License,
- Version 2.0 (the "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software distributed
- under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
- OR CONDITIONS OF ANY KIND, either express or implied. See the License for
- the specific language governing permissions and limitations under the License. -->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>com.opensoc</groupId>
- <artifactId>OpenSOC-Streaming</artifactId>
- <version>0.0.1-SNAPSHOT</version>
- </parent>
- <artifactId>OpenSOC-Alerts</artifactId>
- <name>OpenSOC-Alerts</name>
- <description>Taggers for alerts</description>
- <properties>
- <json.simple.version>1.1.1</json.simple.version>
-
- <storm.version>0.9.2-incubating</storm.version>
- <kafka.version>0.8.0</kafka.version>
- <metrics.version>3.0.2</metrics.version>
- <commons.config.version>1.10</commons.config.version>
- <opensoc.common.version>0.0.1-SNAPSHOT</opensoc.common.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>com.opensoc</groupId>
- <artifactId>OpenSOC-Common</artifactId>
- <version>${opensoc.common.version}</version>
- </dependency>
- <dependency>
- <groupId>com.googlecode.json-simple</groupId>
- <artifactId>json-simple</artifactId>
- <version>${json.simple.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-core</artifactId>
- <version>${storm.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.kafka</groupId>
- <artifactId>kafka_2.9.2</artifactId>
- <version>0.8.0</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>com.sun.jmx</groupId>
- <artifactId>jmxri</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jdmk</groupId>
- <artifactId>jmxtools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.jms</groupId>
- <artifactId>jms</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>com.codahale.metrics</groupId>
- <artifactId>metrics-core</artifactId>
- <version>${metrics.version}</version>
- </dependency>
- <dependency>
- <groupId>com.codahale.metrics</groupId>
- <artifactId>metrics-graphite</artifactId>
- <version>${metrics.version}</version>
- </dependency>
- <dependency>
- <groupId>commons-configuration</groupId>
- <artifactId>commons-configuration</artifactId>
- <version>${commons.config.version}</version>
- </dependency>
- <dependency>
- <groupId>commons-validator</groupId>
- <artifactId>commons-validator</artifactId>
- <version>1.4.0</version>
- </dependency>
-
- </dependencies>
- <build>
- <resources>
- <resource>
- <directory>src/main/resources</directory>
- </resource>
- </resources>
- </build>
-</project>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Alerts/readme.md
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Alerts/readme.md b/metron-streaming/Metron-Alerts/readme.md
deleted file mode 100644
index 2b87d50..0000000
--- a/metron-streaming/Metron-Alerts/readme.md
+++ /dev/null
@@ -1,104 +0,0 @@
-#Metron-Alerts
-
-##Module Description
-
-This module enables telemetry alerts. It splits the mssage stream into two streams. The original message is emitted on the "message" stream. The corresponding alert is emitted on the "alerts" stream. The two are tied together through the alerts UUID.
-
-##Message Format
-
-Assuming the original message (with enrichments enabled) has the following format:
-
-```json
-{
-"message":
-{"ip_src_addr": xxxx,
-"ip_dst_addr": xxxx,
-"ip_src_port": xxxx,
-"ip_dst_port": xxxx,
-"protocol": xxxx,
-"timestamp": xxxx.
-"original_string": xxxx,
-"additional-field 1": xxxx,
-},
-"enrichment" : {"geo": xxxx, "whois": xxxx, "hosts": xxxxx, "CIF": "xxxxx"}
-
-}
-```
-
-The telemetry message will be tagged with a UUID alert tag like so:
-
-```json
-{
-"message":
-{"ip_src_addr": xxxx,
-"ip_dst_addr": xxxx,
-"ip_src_port": xxxx,
-"ip_dst_port": xxxx,
-"protocol": xxxx,
-"timestamp": xxxx,
-"original_string": xxxx,
-"additional-field 1": xxxx,
-},
-"enrichment" : {"geo": xxxx, "whois": xxxx, "hosts": xxxxx, "CIF": "xxxxx"},
-"alerts": [UUID1, UUID2, UUID3, etc]
-
-}
-```
-
-The alert will be fired on the "alerts" stream and can be customized to have any format as long as it includes the required mandatory fields. The mandatory fields are:
-
-* timestamp (epoch): The time from the message that triggered the alert
-* description: A human friendly string representation of the alert
-* alert_id: The UUID generated for the alert. This uniquely identifies an alert
-
-There are other standard but not mandatory fields that can be leveraged by metron-ui and other alert consumers:
-
-* designated_host: The IP address that corresponds to an asset. Ex. The IP address of the company device associated with the alert.
-* enrichment: A copy of the enrichment data from the message that triggered the alert
-* priority: The priority of the alert. Mustb e set to one of HIGH, MED or LOW
-
-An example of an alert with all mandatory and standard fields would look like so:
-
-```json
-{
-"timestamp": xxxx,
-"alert_id": UUID,
-"description": xxxx,
-"designated_host": xxxx,
-"enrichment": { "geo": xxxx, "whois": xxxx, "cif": xxxx },
-"priority": "MED"
-}
-```
-
-##Alerts Bolt
-
-The bolt can be extended with a variety of alerts adapters. The ability to stack alerts is currently in beta, but is not currently advisable. We advice to only have one alerts bolt per topology. The adapters are rules-based adapters which fire alerts when rules are a match. Currently only Java adapters are provided, but there are future plans to provide Grok-Based adapters as well.
-
-The signature of the Alerts bolt is as follows:
-
-```
-TelemetryAlertsBolt alerts_bolt = new TelemetryAlertsBolt()
-.withIdentifier(alerts_identifier).withMaxCacheSize(1000)
-.withMaxTimeRetain(3600).withAlertsAdapter(alerts_adapter)
-.withMetricConfiguration(config);
-```
-Identifier - JSON key where the alert is attached
-TimeRetain & MaxCacheSize - Caching parameters for the bolt
-MetricConfiguration - export custom bolt metrics to graphite (if not null)
-AlertsAdapter - pick the appropriate adapter for generating the alerts
-
-### Java Adapters
-
-Java adapters are designed for high volume topologies, but are not easily extensible. The adapters provided are:
-
-* org.apache.metron.alerts.adapters.AllAlertsAdapter - will tag every single message with the static alert (appropriate for topologies like Sourcefire, etc, where every single message is an alert)
-* org.apache.metron.alerts.adapters.HbaseWhiteAndBlacklistAdapter - will read white and blacklists from HBase and fire alerts if source or dest IP are not on the whitelist or if any IP is on the blacklist
-* org.apache.metron.alerts.adapters.CIFAlertsAdapter - will alert on messages that have results in enrichment.cif.
-* org.apache.metron.alerts.adpaters.KeywordsAlertAdapter - will alert on messages that contain any of a list of keywords
-###Grok Adapters
-
-Grok alerts adapters for Metron are still under devleopment
-
-###Stacking Alert Adapters
-
-The functionality to stack alerts adapters is still under development
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Alerts/src/test/resources/config/AllAlertAdapterTest.config
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Alerts/src/test/resources/config/AllAlertAdapterTest.config b/metron-streaming/Metron-Alerts/src/test/resources/config/AllAlertAdapterTest.config
index fe7d109..11c545c 100644
--- a/metron-streaming/Metron-Alerts/src/test/resources/config/AllAlertAdapterTest.config
+++ b/metron-streaming/Metron-Alerts/src/test/resources/config/AllAlertAdapterTest.config
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
#Alerts Bolt
bolt.alerts.adapter=org.apache.metron.alerts.adapters.AllAlertAdapter
org.apache.metron.alerts.adapters.AllAlertAdapter.whitelist_table_name = ip_whitelist
@@ -5,4 +22,4 @@ org.apache.metron.alerts.adapters.AllAlertAdapter.blacklist_table_name = ip_blac
org.apache.metron.alerts.adapters.AllAlertAdapter.quorum=zkpr1,zkpr2,zkpr3
org.apache.metron.alerts.adapters.AllAlertAdapter.port=2181
org.apache.metron.alerts.adapters.AllAlertAdapter._MAX_CACHE_SIZE_OBJECTS_NUM=3600
-org.apache.metron.alerts.adapters.AllAlertAdapter._MAX_TIME_RETAIN_MINUTES=1000
\ No newline at end of file
+org.apache.metron.alerts.adapters.AllAlertAdapter._MAX_TIME_RETAIN_MINUTES=1000
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Common/pom.xml.versionsBackup
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Common/pom.xml.versionsBackup b/metron-streaming/Metron-Common/pom.xml.versionsBackup
deleted file mode 100644
index 8ead949..0000000
--- a/metron-streaming/Metron-Common/pom.xml.versionsBackup
+++ /dev/null
@@ -1,131 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?><!-- Licensed to the Apache Software
- Foundation (ASF) under one or more contributor license agreements. See the
- NOTICE file distributed with this work for additional information regarding
- copyright ownership. The ASF licenses this file to You under the Apache License,
- Version 2.0 (the "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
- Unless required by applicable law or agreed to in writing, software distributed
- under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
- OR CONDITIONS OF ANY KIND, either express or implied. See the License for
- the specific language governing permissions and limitations under the License. -->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>com.opensoc</groupId>
- <artifactId>OpenSOC-Streaming</artifactId>
- <version>0.0.1-SNAPSHOT</version>
- </parent>
- <artifactId>OpenSOC-Common</artifactId>
- <name>OpenSOC-Common</name>
- <description>Components common to all enrichments</description>
- <properties>
- <json.simple.version>1.1.1</json.simple.version>
-
- <storm.version>0.9.2-incubating</storm.version>
- <kafka.version>0.8.0</kafka.version>
- <metrics.version>3.0.2</metrics.version>
- <commons.config.version>1.10</commons.config.version>
- <hbase.version>0.98.5-hadoop2</hbase.version>
- </properties>
- <repositories>
- <repository>
- <id>Kraken-Repo</id>
- <name>Kraken Repository</name>
- <url>http://download.krakenapps.org</url>
- </repository>
- </repositories>
- <dependencies>
- <dependency>
- <groupId>com.googlecode.json-simple</groupId>
- <artifactId>json-simple</artifactId>
- <version>${json.simple.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-core</artifactId>
- <version>${storm.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.kafka</groupId>
- <artifactId>kafka_2.9.2</artifactId>
- <version>0.8.0</version>
- <scope>provided</scope>
- <exclusions>
- <exclusion>
- <groupId>com.sun.jmx</groupId>
- <artifactId>jmxri</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.sun.jdmk</groupId>
- <artifactId>jmxtools</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.jms</groupId>
- <artifactId>jms</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
- <dependency>
- <groupId>com.codahale.metrics</groupId>
- <artifactId>metrics-core</artifactId>
- <version>${metrics.version}</version>
- </dependency>
- <dependency>
- <groupId>com.codahale.metrics</groupId>
- <artifactId>metrics-graphite</artifactId>
- <version>${metrics.version}</version>
- </dependency>
- <dependency>
- <groupId>commons-configuration</groupId>
- <artifactId>commons-configuration</artifactId>
- <version>${commons.config.version}</version>
- </dependency>
- <dependency>
- <groupId>org.krakenapps</groupId>
- <artifactId>kraken-pcap</artifactId>
- <version>1.5.0</version>
- </dependency>
- <dependency>
- <groupId>junit</groupId>
- <artifactId>junit</artifactId>
- <version>3.8.2</version>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-client</artifactId>
- <version>${hbase.version}</version>
- </dependency>
- </dependencies>
-
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>1.7</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
-
-</project>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Common/src/main/java/org/apache/metron/ise/parser/ISEParser.jj
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Common/src/main/java/org/apache/metron/ise/parser/ISEParser.jj b/metron-streaming/Metron-Common/src/main/java/org/apache/metron/ise/parser/ISEParser.jj
index 86edd1f..6071922 100644
--- a/metron-streaming/Metron-Common/src/main/java/org/apache/metron/ise/parser/ISEParser.jj
+++ b/metron-streaming/Metron-Common/src/main/java/org/apache/metron/ise/parser/ISEParser.jj
@@ -2,8 +2,8 @@ options{ CHOICE_AMBIGUITY_CHECK = 3; OTHER_AMBIGUITY_CHECK = 2; //DEBUG_P
//DEBUG_LOOKAHEAD=true
//DEBUG_TOKEN_MANAGER=true
ERROR_REPORTING = true; JAVA_UNICODE_ESCAPE = true; UNICODE_INPUT = true; IGNORE_CASE = true; SUPPORT_CLASS_VISIBILITY_PUBLIC = false; FORCE_LA_CHECK = true; CACHE_TOKENS = true; SANITY_CHECK = true; STATIC = false; //KEEP_LINE_COLUMN=true;
-}PARSER_BEGIN(ISEParser)package org.apache.metron.ise.parser;import java.io.*;import java.util.*;import org.json.simple.*;/**
-* Basic ISE data parser generated by JavaCC.
+}PARSER_BEGIN(ISEParser)/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */package org.apache.metron.ise.parser;import java.io.*;import java.util.*;import org.json.simple.*;/**
+* Basic ISE data parser generated by JavaCC.
*/public class ISEParser implements Serializable{ private boolean nativeNumbers = false; public ISEParser() { //do nothing
} public ISEParser(String input) { this (new StringReader(input)); } /**
* Parses a ISE String into a JSON object {@code Map}.
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-Common/src/test/resources/config/BasicTldExtractorTest.config
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-Common/src/test/resources/config/BasicTldExtractorTest.config b/metron-streaming/Metron-Common/src/test/resources/config/BasicTldExtractorTest.config
index 6b3dc05..317add3 100644
--- a/metron-streaming/Metron-Common/src/test/resources/config/BasicTldExtractorTest.config
+++ b/metron-streaming/Metron-Common/src/test/resources/config/BasicTldExtractorTest.config
@@ -1,2 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
#BasicTldExtractorConfig
logFile=effective_tld_names.dat
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataLoads/pom.xml.versionsBackup
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataLoads/pom.xml.versionsBackup b/metron-streaming/Metron-DataLoads/pom.xml.versionsBackup
deleted file mode 100644
index a122057..0000000
--- a/metron-streaming/Metron-DataLoads/pom.xml.versionsBackup
+++ /dev/null
@@ -1,97 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?><!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
--->
-
-<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>com.opensoc</groupId>
- <artifactId>OpenSOC-Streaming</artifactId>
- <version>0.0.1-SNAPSHOT</version>
- </parent>
- <artifactId>OpenSOC-DataLoads</artifactId>
- <properties>
- <opensoc.common.version>0.0.1-SNAPSHOT</opensoc.common.version>
- <storm.version>0.9.1-incubating</storm.version>
- <hbase.version>0.98.0-hadoop2</hbase.version>
- </properties>
- <dependencies>
- <dependency>
- <groupId>com.opensoc</groupId>
- <artifactId>OpenSOC-Common</artifactId>
- <version>${opensoc.common.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.storm</groupId>
- <artifactId>storm-core</artifactId>
- <version>${storm.version}</version>
- <scope>provided</scope>
- </dependency>
- <dependency>
- <groupId>org.apache.hbase</groupId>
- <artifactId>hbase-client</artifactId>
- <version>${hbase.version}</version>
- </dependency>
- </dependencies>
- <build>
- <sourceDirectory>src</sourceDirectory>
- <resources>
- <resource>
- <directory>src</directory>
- <excludes>
- <exclude>**/*.java</exclude>
- </excludes>
- </resource>
- </resources>
- <plugins>
- <plugin>
- <artifactId>maven-compiler-plugin</artifactId>
- <version>3.1</version>
- <configuration>
- <source>1.7</source>
- <target>1.7</target>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <version>2.3</version>
- <executions>
- <execution>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <artifactSet>
- <excludes>
- <exclude>classworlds:classworlds</exclude>
- <exclude>junit:junit</exclude>
- <exclude>jmock:*</exclude>
- <exclude>*:xml-apis</exclude>
- <exclude>org.apache.maven:lib:tests</exclude>
- <exclude>log4j:log4j:jar:</exclude>
- <exclude>*:hbase:*</exclude>
- </excludes>
- </artifactSet>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-
-</project>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataLoads/src/main/assembly/assembly.xml
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataLoads/src/main/assembly/assembly.xml b/metron-streaming/Metron-DataLoads/src/main/assembly/assembly.xml
index 44d7216..19c5464 100644
--- a/metron-streaming/Metron-DataLoads/src/main/assembly/assembly.xml
+++ b/metron-streaming/Metron-DataLoads/src/main/assembly/assembly.xml
@@ -1,3 +1,16 @@
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See the
+ NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to You under the Apache License,
+ Version 2.0 (the "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+ the specific language governing permissions and limitations under the License.
+ -->
+
<assembly>
<id>archive</id>
<formats>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/README.md
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/README.md b/metron-streaming/Metron-DataServices/README.md
index e845566..7ee5355 100644
--- a/metron-streaming/Metron-DataServices/README.md
+++ b/metron-streaming/Metron-DataServices/README.md
@@ -1 +1,16 @@
-README
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ -->
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/conf/config.properties
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/conf/config.properties b/metron-streaming/Metron-DataServices/conf/config.properties
index 7c060fb..6a40a71 100644
--- a/metron-streaming/Metron-DataServices/conf/config.properties
+++ b/metron-streaming/Metron-DataServices/conf/config.properties
@@ -1,5 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
# sample config file for the Alerts/Proxy service. Fill in
-# the settings below with values appropriate to your environment.
+# the settings below with values appropriate to your environment.
alertsServiceImpl=elasticsearch
ldapUrl=ldap://ec2-54-88-217-194.compute-1.amazonaws.com
@@ -18,4 +35,3 @@ keystorePassword=password
authTokenAlias=authTokenKey
authTokenMaxAge=10000
alerts.cache.expiration.interval=360000
-
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/config-definition-hbase.xml
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/config-definition-hbase.xml b/metron-streaming/Metron-DataServices/src/main/resources/config-definition-hbase.xml
index efe05e8..5d092bf 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/config-definition-hbase.xml
+++ b/metron-streaming/Metron-DataServices/src/main/resources/config-definition-hbase.xml
@@ -1,3 +1,19 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
<?xml version="1.0" encoding="ISO-8859-1" ?>
<configuration>
@@ -15,20 +31,20 @@
</lookups>
</header>
<override>
- <!-- 1. properties from 'hbae-config.properties' are loaded first;
+ <!-- 1. properties from 'hbae-config.properties' are loaded first;
if a property is not present in this file, then it will search in the files in the order they are defined here.
2. 'refreshDelay' indicates the minimum delay in milliseconds between checks to see if the underlying file is changed.
- 3. 'config-optional' indicates this file is not required -->
-
+ 3. 'config-optional' indicates this file is not required -->
+
<properties fileName="${expr:System.getProperty('configPath')+'/hbase-config.properties'}" config-optional="true">
<reloadingStrategy refreshDelay="${expr:System.getProperty('configRefreshDelay')}"
config-class="org.apache.commons.configuration.reloading.FileChangedReloadingStrategy"/>
</properties>
-
+
<properties fileName="hbase-config-default.properties" config-optional="true">
<!-- <reloadingStrategy refreshDelay="${expr:System.getProperty('defaultConfigRefreshDelay')}"
config-class="org.apache.commons.configuration.reloading.FileChangedReloadingStrategy"/>
--> </properties>
-
+
</override>
-</configuration>
\ No newline at end of file
+</configuration>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/hbase-config-default.properties
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/hbase-config-default.properties b/metron-streaming/Metron-DataServices/src/main/resources/hbase-config-default.properties
index 4ee56b6..5ddf676 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/hbase-config-default.properties
+++ b/metron-streaming/Metron-DataServices/src/main/resources/hbase-config-default.properties
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
#hbase zoo keeper configuration
hbase.zookeeper.quorum=zkpr1,zkpr2,zkpr3
hbase.zookeeper.clientPort=2181
@@ -25,7 +42,7 @@ hbase.hconnection.retries.number=3
#configuration for including pcaps in the reverse traffic
pcaps.include.reverse.traffic = false
-#maximum table row size in KB or MB
+#maximum table row size in KB or MB
hbase.table.row.size.unit = KB
hbase.table.max.row.size = 70
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/log4j.xml
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/log4j.xml b/metron-streaming/Metron-DataServices/src/main/resources/log4j.xml
index 834b3b3..6096374 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/log4j.xml
+++ b/metron-streaming/Metron-DataServices/src/main/resources/log4j.xml
@@ -1,17 +1,33 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ -->
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
- <appender name="console" class="org.apache.log4j.ConsoleAppender">
- <param name="Target" value="System.out"/>
- <layout class="org.apache.log4j.PatternLayout">
- <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
- </layout>
- </appender>
+ <appender name="console" class="org.apache.log4j.ConsoleAppender">
+ <param name="Target" value="System.out"/>
+ <layout class="org.apache.log4j.PatternLayout">
+ <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
+ </layout>
+ </appender>
- <root>
- <priority value ="info" />
- <appender-ref ref="console" />
+ <root>
+ <priority value ="info" />
+ <appender-ref ref="console" />
</root>
-
-</log4j:configuration>
\ No newline at end of file
+
+</log4j:configuration>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/webroot/logged_in.jsp
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/webroot/logged_in.jsp b/metron-streaming/Metron-DataServices/src/main/resources/webroot/logged_in.jsp
index 98b1159..52fa9ae 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/webroot/logged_in.jsp
+++ b/metron-streaming/Metron-DataServices/src/main/resources/webroot/logged_in.jsp
@@ -1,14 +1,30 @@
+<%--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ --%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<html>
-
+
<head>
<title>Login</title>
</head>
-
+
<body>
<h3>Logged In!</h3>
-
+
<ul>
<c:forEach var="currentCookie" items="${cookie}">
<li>
@@ -16,7 +32,7 @@
</li>
</c:forEach>
</ul>
-
+
</body>
-
-</html>
\ No newline at end of file
+
+</html>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_failed.jsp
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_failed.jsp b/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_failed.jsp
index 457726f..510bf90 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_failed.jsp
+++ b/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_failed.jsp
@@ -1,14 +1,30 @@
+<%--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ --%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<html>
-
+
<head>
<title>Login</title>
</head>
-
+
<body>
<h3>Fail!!!!</h3>
-
+
</body>
-
-</html>
\ No newline at end of file
+
+</html>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_old.jsp
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_old.jsp b/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_old.jsp
index dbe6e48..4b3017f 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_old.jsp
+++ b/metron-streaming/Metron-DataServices/src/main/resources/webroot/login_old.jsp
@@ -1,21 +1,38 @@
+<%--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ --%>
+
<html>
-
+
<head>
<title>Login</title>
</head>
-
+
<body>
<h3>Login:</h3>
-
+
<form action="/login" method="POST">
-
+
<label for="userName">Username:</label><input type="text" name="userName" id="userName"></input>
<br />
<label for="password">Password:</label><input type="password" name="password" id="password" ></input>
<br />
<button name="login">Login</button>
</form>
-
+
</body>
-
-</html>
\ No newline at end of file
+
+</html>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket.jsp
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket.jsp b/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket.jsp
index 0530eaa..bd71bfa 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket.jsp
+++ b/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket.jsp
@@ -1,33 +1,49 @@
+<%--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ --%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<html>
-
+
<head>
<title>Do something with WebSockets</title>
-
- <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
+
+ <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
<script type="text/javascript">
$(document).ready(function() {
-
+
var exampleSocket = null;
var groupId = null;
-
+
$("#startBtn").click(
function() {
exampleSocket = new WebSocket("wss://localhost:8443/ws/messages");
if( exampleSocket != null )
{
-
+
exampleSocket.onmessage = function (event)
{
-
+
var msg = event.data;
- var index = msg.search( "groupId:" );
+ var index = msg.search( "groupId:" );
if( index >= 0 )
{
console.log( "groupId message" );
// this is our groupId from the server, save it for future use
-
+
groupId = msg.substring( index + 8 ).trim();
console.log( "groupId: " + groupId );
}
@@ -35,49 +51,49 @@
{
// console.log( "alert message" );
// this is an alert message, just render it...
-
+
// console.log( event.data );
$('#msgList').append( '<li>' + msg + '</li>' );
}
}
-
+
exampleSocket.onopen = function() {
console.log( "exampleSocket is open.")
console.log( "sending startMessages signal" );
- exampleSocket.send( "startMessages" );
+ exampleSocket.send( "startMessages" );
}
}
else
{
alert( "exampleSocket is null!" );
}
-
- }
+
+ }
)
-
+
$("#stopBtn").click(
function() {
exampleSocket.send( "stopMessages");
}
)
-
+
})
-
+
</script>
</head>
-
+
<body>
<h3>Get some data with a WebSocket</h3>
-
+
<button id="startBtn">Start</button> <span style="min-width:140px;"></span> <button id="stopBtn">Stop</button>
<div id="msgArea" style="background-color:#F8F8F8;color:red;min-height:100px; min-width:400px;">
<ul id="msgList" >
-
+
</ul>
-
+
</div>
-
+
<div id="debug">
<ul>
<c:forEach var="req" items="${header}">
@@ -96,5 +112,5 @@
</ul>
</div>
</body>
-
-</html>
\ No newline at end of file
+
+</html>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket2.jsp
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket2.jsp b/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket2.jsp
index 7ea83ba..f533c6a 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket2.jsp
+++ b/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket2.jsp
@@ -1,11 +1,27 @@
+<%--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ --%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<html>
-
+
<head>
<title>Do something with WebSockets</title>
-
- <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
+
+ <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
<script type="text/javascript">
$(document).ready(function(){
$("#startBtn").click(
@@ -13,44 +29,44 @@
var exampleSocket = new WebSocket("wss://localhost:8443/ws/messages");
if( exampleSocket != null )
{
-
+
exampleSocket.onmessage = function (event)
{
-
+
// console.log( event.data );
$('#msgList').append( '<li>' + event.data + '</li>' );
}
-
+
exampleSocket.onopen = function() {
console.log( "exampleSocket is open.")
console.log( "sending startMessages signal" );
- exampleSocket.send( "startMessages" );
+ exampleSocket.send( "startMessages" );
}
}
else
{
alert( "exampleSocket is null!" );
}
-
- }
+
+ }
)
})
-
+
</script>
</head>
-
+
<body>
<h3>Get some data with a WebSocket</h3>
-
+
<button id="startBtn">Start</button>
<div id="msgArea" style="background-color:#F8F8F8;color:red;min-height:100px; min-width:400px;">
<ul id="msgList" >
-
+
</ul>
-
+
</div>
-
+
<div id="debug">
<ul>
<c:forEach var="req" items="${header}">
@@ -69,5 +85,5 @@
</ul>
</div>
</body>
-
-</html>
\ No newline at end of file
+
+</html>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket3.jsp
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket3.jsp b/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket3.jsp
index 05c929c..7ca1a9b 100644
--- a/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket3.jsp
+++ b/metron-streaming/Metron-DataServices/src/main/resources/webroot/withsocket3.jsp
@@ -1,33 +1,49 @@
+<%--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+ --%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<html>
-
+
<head>
<title>Do something with WebSockets</title>
-
- <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
+
+ <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
<script type="text/javascript">
$(document).ready(function() {
-
+
var exampleSocket = null;
var groupId = null;
-
+
$("#startBtn").click(
function() {
exampleSocket = new WebSocket("wss://localhost:8443/ws/messages");
if( exampleSocket != null )
{
-
+
exampleSocket.onmessage = function (event)
{
-
+
var msg = event.data;
- var index = msg.search( "groupId:" );
+ var index = msg.search( "groupId:" );
if( index >= 0 )
{
console.log( "groupId message" );
// this is our groupId from the server, save it for future use
-
+
groupId = msg.substring( index + 8 ).trim();
console.log( "groupId: " + groupId );
}
@@ -35,49 +51,49 @@
{
// console.log( "alert message" );
// this is an alert message, just render it...
-
+
// console.log( event.data );
$('#msgList').append( '<li>' + msg + '</li>' );
}
}
-
+
exampleSocket.onopen = function() {
console.log( "exampleSocket is open.")
console.log( "sending startMessages signal" );
- exampleSocket.send( "startMessages:abc123" );
+ exampleSocket.send( "startMessages:abc123" );
}
}
else
{
alert( "exampleSocket is null!" );
}
-
- }
+
+ }
)
-
+
$("#stopBtn").click(
function() {
exampleSocket.send( "stopMessages");
}
)
-
+
})
-
+
</script>
</head>
-
+
<body>
<h3>Get some data with a WebSocket</h3>
-
+
<button id="startBtn">Start</button> <span style="min-width:140px;"></span> <button id="stopBtn">Stop</button>
<div id="msgArea" style="background-color:#F8F8F8;color:red;min-height:100px; min-width:400px;">
<ul id="msgList" >
-
+
</ul>
-
+
</div>
-
+
<div id="debug">
<ul>
<c:forEach var="req" items="${header}">
@@ -96,5 +112,5 @@
</ul>
</div>
</body>
-
-</html>
\ No newline at end of file
+
+</html>
http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/244a5359/metron-streaming/Metron-EnrichmentAdapters/README.md
----------------------------------------------------------------------
diff --git a/metron-streaming/Metron-EnrichmentAdapters/README.md b/metron-streaming/Metron-EnrichmentAdapters/README.md
new file mode 100644
index 0000000..01c4caa
--- /dev/null
+++ b/metron-streaming/Metron-EnrichmentAdapters/README.md
@@ -0,0 +1,125 @@
+#Metron-Enrichments
+
+##Module Description
+
+This module enables enrichment of message metafields with additional information from various enrichment sources. Currently there is only a limited number of enrichments available, but this is an extensible framework that can be extended with additional enrichments. Enrichments currently available are geo, whois, hosts, and CIF.
+
+##Message Format
+
+Enrichment bolts are designed to go after the parser bolts. Parser bolts will parse the telemetry, taking it from its native format and producing a standard JSON that would look like so:
+
+```json
+{
+"message":
+{"ip_src_addr": xxxx,
+"ip_dst_addr": xxxx,
+"ip_src_port": xxxx,
+"ip_dst_port": xxxx,
+"protocol": xxxx,
+"additional-field 1": xxx,
+}
+
+}
+```
+
+A single enrichment bolt would enrich the message and produce a JSON enrichment and attach it to the message. Enrichments are stackable so multiple enrichments can be attached sequentially after a single parser bolt. Stacked enrichments would produce messages under the "enrichment" tag and attach it to the message like so:
+
+```json
+{
+"message":
+{"ip_src_addr": xxxx,
+"ip_dst_addr": xxxx,
+"ip_src_port": xxxx,
+"ip_dst_port": xxxx,
+"protocol": xxxx,
+"additional-field 1": xxxx,
+},
+"enrichment" : {"geo": xxxx, "whois": xxxx, "hosts": xxxxx, "CIF": "xxxxx"}
+
+}
+```
+
+##Enrichment Sources
+
+Each enrichment has to have an anrichment source which can serve as a lookup table for enriching relevant message fields. In order to minimize the use of additional platforms and tools we primarily try to rely on HBase as much as possible to store the enrichment information for lookup by key. In order to use Hbase we have to pre-process the enrichment feeds for bulk-loading into HBase with specific key format optimized for retrieval as well as utilize caches within the enrichment bolts to be able to provide enrichments real-time. Our wiki contains information on how to setup the environment, pre-process feeds, and plug in the enrichment sources.
+
+##Enrichment Bolt
+
+The enrichment bolt is designed to be extensible to be re-used for all kinds of enrichment processes. The bolt signature for declaration in a storm topology is as follows:
+
+
+
+```
+GenericEnrichmentBolt geo_enrichment = new GenericEnrichmentBolt()
+.withEnrichmentTag(
+config.getString("bolt.enrichment.geo.enrichment_tag"))
+.withAdapter(geo_adapter)
+.withMaxTimeRetain(
+config.getInt("bolt.enrichment.geo.MAX_TIME_RETAIN_MINUTES"))
+.withMaxCacheSize(
+config.getInt("bolt.enrichment.geo.MAX_CACHE_SIZE_OBJECTS_NUM"))
+.withKeys(geo_keys).withMetricConfiguration(config);
+
+```
+
+EnrichmentTag - Name of the enrichment (geo, whois, hosts, etc)
+Keys - Keys which this enrichment is able to enrich (hosts field for hosts enrichment, source_ip, dest_ip, for geo enrichment, etc)
+MaxTimeToRetain & MaxCacheSize - define the caching policy of the enrichment bolt
+Adapter - which adapter to use with the enrichment bolt instance
+
+###Geo Adapter
+Geo adapter is able to do geo enrichment on hosts and destination IPs. The open source verison of the geo adapter uses the free Geo feeds from MaxMind. The format of these feeds does not easily lend itself to a no-sql DB so this adapter is designed to work with mySql. But it is extensible enough to be made work with a variety of other back ends.
+
+The signature of a geo adapter is as follows;
+
+```
+GeoMysqlAdapter geo_adapter = new GeoMysqlAdapter(
+config.getString("mysql.ip"), config.getInt("mysql.port"),
+config.getString("mysql.username"),
+config.getString("mysql.password"),
+config.getString("bolt.enrichment.geo.adapter.table"));
+
+```
+
+###Hosts Adapter
+The hosts adapter is designed to enrich message format with the static host information that can be read from a standard text file. This adapter is intended for use with a network crawling script that can identify all customer assets and place them in a text file. For example, this script would identify all workstations, printers, appliantces, etc. Then if any of these assets are seen in the telemetry messages flowing through the adapter this enrichment would fire and the relevant known information about a host would be attached. We are currently working on porting this adapter to work with HBase, but this work is not ready yet. The known hosts file is located under the /etc/whitelists config directory of Metron.
+
+The signature of the hosts adapter is as follows:
+
+```
+Map<String, JSONObject> known_hosts = SettingsLoader
+.loadKnownHosts(hosts_path);
+
+HostFromPropertiesFileAdapter host_adapter = new HostFromPropertiesFileAdapter(
+known_hosts);
+
+```
+* The source and dest ips refer to the name of the message JSON key where the host information is located
+
+###Whois Adapter
+Whois adapter enriches the host name with additional whois information obtained from our proprietary Cisco feed. The enricher itself is provided in this open source distribution, but the feed is not. You have to have your own feed in order to use it. Alternatively, you can contact us for providing you with this feed, but we would have to charge you a fee (we can't distribute it for free). The implemetation of the whois enrichment we provide works with HBase
+
+The signature of the whois adapter is as follows:
+
+```
+
+EnrichmentAdapter whois_adapter = new WhoisHBaseAdapter(
+config.getString("bolt.enrichment.whois.hbase.table.name"),
+config.getString("kafka.zk.list"),
+config.getString("kafka.zk.port"));
+```
+
+###CIF Adapter
+CIF adapter is designed to take in CIF feeds and cross-reference them against every message processed by Storm. If there is a hit then the relevant information is attached to the message.
+
+The signature of the CIF adapter is as follows:
+
+```
+CIFHbaseAdapter = new CIFHbaseAdapter(config
+.getString("kafka.zk.list"), config
+.getString("kafka.zk.port"), config
+.getString("bolt.enrichment.cif.tablename")))
+```
+
+##Stacking Enrichments
+Enrichments can be stacked. By default each enrichment bolt listens on the "message" stream. In order to create and stack enrichment bolts create a new bolt and instantiate the appropariate adapter. You can look at our sample topologies to see how enrichments can be stacked