You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/06/01 21:41:20 UTC
[01/44] metron git commit: METRON-899: Fix bad Kerberos doc merge in
METRON-835 (mmiklavc) closes apache/incubator-metron#554
Repository: metron
Updated Branches:
refs/heads/Metron_0.4.0 68bd6c520 -> 85872bd68
METRON-899: Fix bad Kerberos doc merge in METRON-835 (mmiklavc) closes apache/incubator-metron#554
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/f36db22e
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/f36db22e
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/f36db22e
Branch: refs/heads/Metron_0.4.0
Commit: f36db22eba256209d494f2ab3eda4f0c2b2448d7
Parents: 68bd6c5
Author: mmiklavc <mi...@gmail.com>
Authored: Fri Apr 28 07:52:18 2017 -0600
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Fri Apr 28 07:52:18 2017 -0600
----------------------------------------------------------------------
metron-deployment/Kerberos-ambari-setup.md | 33 ++
metron-deployment/Kerberos-manual-setup.md | 395 ++++++++++++++++------
metron-deployment/README.md | 4 +-
metron-deployment/vagrant/Kerberos-setup.md | 411 -----------------------
metron-deployment/vagrant/README.md | 1 -
site-book/bin/generate-md.sh | 27 +-
6 files changed, 349 insertions(+), 522 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/f36db22e/metron-deployment/Kerberos-ambari-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-ambari-setup.md b/metron-deployment/Kerberos-ambari-setup.md
new file mode 100644
index 0000000..149e8b2
--- /dev/null
+++ b/metron-deployment/Kerberos-ambari-setup.md
@@ -0,0 +1,33 @@
+# Setting Up Kerberos in Vagrant Full Dev
+**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
+General Kerberization notes can be found in the metron-deployment [README.md](../README.md)
+
+## Setup a KDC
+See [Setup a KDC](Kerberos-manual-setup.md#setup-a-kdc)
+
+## Ambari Setup
+1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+
+ a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
+
+ 
+
+ 
+
+ b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
+
+ 
+
+ c. Click through to “Start and Test Services.” Let the cluster spin up.
+
+## Push Data
+1. Kinit with the metron user
+ ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+See [Push Data](Kerberos-manual-setup.md#push-data)
+
+### More Information
+
+See [More Information](Kerberos-manual-setup.md#more-information)
http://git-wip-us.apache.org/repos/asf/metron/blob/f36db22e/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index 4eaa725..b444b0e 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -1,34 +1,103 @@
-# Setting Up Kerberos outside of an Ambari Management Pack
-The Ambari Management pack will manage Kerberization when used.
-**Note:** These are instructions for Kerberizing Metron Storm topologies from Kafka to Kafka. This does not cover the sensor connections or MAAS.
-General Kerberization notes can be found in the metron-deployment [README.md](README.md)
+Kerberos Setup
+==============
-## Setup the KDC
-See [Setup the KDC](vagrant/Kerberos-setup.md)
+This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
-4. Setup the admin and metron user principals. You'll kinit as the metron user when running topologies. Make sure to remember the passwords.
- ```
- kadmin.local -q "addprinc admin/admin"
- kadmin.local -q "addprinc metron"
- ```
+* [Setup](#setup)
+* [Setup a KDC](#setup-a-kdc)
+* [Enable Kerberos](#enable-kerberos)
+* [Kafka Authorization](#kafka-authorization)
+* [HBase Authorization](#hbase-authorization)
+* [Storm Authorization](#storm-authorization)
+* [Start Metron](#start-metron)
+* [Push Data](#push-data)
+* [More Information](#more-information)
-## Kerberize Metron
+Setup
+-----
-1. Stop all topologies - we will restart them again once Kerberos has been enabled.
- ```
- for topology in bro snort enrichment indexing; do storm kill $topology; done
- ```
+1. Deploy a Vagrant development environment; either [Full Dev](full-dev-platform) or [Quick Dev](quick-dev-platform).
+
+1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.
-2. Create the metron user HDFS home directory
```
- sudo -u hdfs hdfs dfs -mkdir /user/metron && \
- sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
- sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ # execute as root
+ sudo su -
+ export KAFKA_HOME="/usr/hdp/current/kafka-broker"
+ export ZOOKEEPER=node1:2181
+ export ELASTICSEARCH=node1:9200
+ export BROKERLIST=node1:6667
+
+ export HDP_HOME="/usr/hdp/current"
+ export KAFKA_HOME="${HDP_HOME}/kafka-broker"
+ export METRON_VERSION="0.4.0"
+ export METRON_HOME="/usr/metron/${METRON_VERSION}"
```
-3. In [Ambari](http://node1:8080), setup Storm to run with Kerberos and run worker jobs as the submitting user:
+1. Execute the following commands as root.
+
+ ```
+ sudo su -
+ ```
+
+1. Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.
+
+ ```
+ for topology in bro snort enrichment indexing; do
+ storm kill $topology;
+ done
+ ```
+
+1. Create the `metron` user's home directory in HDFS.
+
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
+
+Setup a KDC
+-----------
+
+1. Install dependencies.
+
+ ```
+ yum -y install krb5-server krb5-libs krb5-workstation
+ ```
+
+1. Define the host, `node1`, as the KDC.
+
+ ```
+ sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+ cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+ ```
+
+1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
+
+ ```
+ kdb5_util create -s
+
+ /etc/rc.d/init.d/krb5kdc start
+ chkconfig krb5kdc on
+
+ /etc/rc.d/init.d/kadmin start
+ chkconfig kadmin on
+ ```
+
+1. Setup the `admin` and `metron` principals. You'll `kinit` as the `metron` principal when running topologies. Make sure to remember the passwords.
+
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
+
+Enable Kerberos
+---------------
+
+1. In [Ambari](http://node1:8080), setup Storm to use Kerberos and run worker jobs as the submitting user.
+
+ a. Add the following properties to the custom storm-site:
- a. Add the following properties to custom storm-site:
```
topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
@@ -43,7 +112,7 @@ See [Setup the KDC](vagrant/Kerberos-setup.md)
-4. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
@@ -59,61 +128,118 @@ See [Setup the KDC](vagrant/Kerberos-setup.md)
-5. Setup Metron keytab
- ```
- kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM" && \
- cp metron.headless.keytab /etc/security/keytabs && \
- chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
- chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+1. Create a Metron keytab
-6. Kinit with the metron user
```
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+ cp metron.headless.keytab /etc/security/keytabs
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
+
+Kafka Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `metron` principal.
+
```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for 'yaf' telemetry.
-7. First create any additional Kafka topics you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. e.g.
```
- ${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER}:2181 --create --topic yaf --partitions 1 --replication-factor 1
+ ${KAFKA_HOME}/bin/kafka-topics.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --create \
+ --topic yaf \
+ --partitions 1 \
+ --replication-factor 1
+ ```
+
+1. Setup Kafka ACLs for the `bro`, `snort`, `enrichments`, and `indexing` topics. Run the same command against any additional topics that you might be using; for example `yaf`.
+
```
+ export KERB_USER=metron
+
+ for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --topic ${topic}
+ done
+ ```
+
+1. Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.
-8. Setup Kafka ACLs for the topics
```
export KERB_USER=metron
- for topic in bro enrichments indexing snort; do
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --topic ${topic}
- done
- ```
-9. Setup Kafka ACLs for the consumer groups
- ```
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group bro_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group snort_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group yaf_parser
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group enrichments
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --group indexing
- ```
+ for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --group ${group}
+ done
+ ```
+
+1. Add the `metron` principal to the `kafka-cluster` ACL.
-10. Add metron user to the Kafka cluster ACL
```
- ${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:${KERB_USER} --cluster kafka-cluster
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
+ --add \
+ --allow-principal User:${KERB_USER} \
+ --cluster kafka-cluster
+ ```
+
+HBase Authorization
+-------------------
+
+1. Acquire a Kerberos ticket using the `hbase` principal
+
```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ ```
+
+1. Grant permissions for the HBase tables used in Metron.
-11. We also need to grant permissions to the HBase tables. Kinit as the hbase user and add ACLs for metron.
```
- kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
- echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
- echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
+
+1. If you are using the Profiler, do the same for its HBase table.
+
```
+ echo "create 'profiler', 'P'" | hbase shell
+ echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+ ```
+
+Storm Authorization
+-------------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
-12. Create a “.storm” directory in the metron user’s home directory and switch to that directory.
```
- su metron
- mkdir ~/.storm
- cd ~/.storm
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Create the directory `/home/metron/.storm` and switch to that directory.
+
```
+ mkdir /home/metron/.storm
+ cd /home/metron/.storm
+ ```
+
+1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
-13. Create a custom client jaas file. This should look identical to the Storm client jaas file located in /etc/storm/conf/client_jaas.conf except for the addition of a Client stanza. The Client stanza is used for Zookeeper. All quotes and semicolons are necessary.
```
cat << EOF > client_jaas.conf
StormClient {
@@ -143,75 +269,152 @@ See [Setup the KDC](vagrant/Kerberos-setup.md)
EOF
```
-14. Create a storm.yaml with jaas file info. Set the array of nimbus hosts accordingly.
+1. Create a YAML file at `/home/metron/.storm/storm.yaml`. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.
+
```
- cat << EOF > storm.yaml
+ cat << EOF > /home/metron/.storm/storm.yaml
nimbus.seeds : ['node1']
java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
EOF
```
-15. Create an auxiliary storm configuration json file in the metron user’s home directory. Note the login config option in the file points to our custom client_jaas.conf.
+1. Create an auxiliary storm configuration file at `/home/metron/storm-config.json`. Note the login config option in the file points to the client JAAS file.
+
```
- cat << EOF > ~/storm-config.json
+ cat << EOF > /home/metron/storm-config.json
{
"topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
}
EOF
```
-16. Setup enrichment and indexing.
+1. Configure the Enrichment, Indexing and the Profiler topologies to use the client JAAS file. Add the following properties to each of the topology properties files.
- a. Modify enrichment.properties as root located at `${METRON_HOME}/config/enrichment.properties`
- ```
- if [[ $EUID -ne 0 ]]; then
- echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
- else
- sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/enrichment.properties
- sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/enrichment.properties
- fi
- ```
+ ```
+ kafka.security.protocol=PLAINTEXTSASL
+ topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
+ ```
+
+ * `${METRON_HOME}/config/enrichment.properties`
+ * `${METRON_HOME}/config/elasticsearch.properties`
+ * `${METRON_HOME}/config/profiler.properties`
+
+ Use the following command to automate this step.
- b. Modify elasticsearch.properties as root located at `${METRON_HOME}/config/elasticsearch.properties`
```
- if [[ $EUID -ne 0 ]]; then
- echo -e "\nERROR:\tYou must be root to run these commands. You may need to type exit."
- else
- sed -i 's/kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/' ${METRON_HOME}/config/elasticsearch.properties
- sed -i 's/topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/' ${METRON_HOME}/config/elasticsearch.properties
- fi
+ for file in enrichment.properties elasticsearch.properties profiler.properties; do
+ echo ${file}
+ sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
+ sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
+ done
```
-17. Distribute the custom jaas file and the keytab to each supervisor node, in the same locations as above. This ensures that the worker nodes can authenticate. For a one node cluster, nothing needs to be done.
+Start Metron
+------------
+
+1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
-18. Kinit with the metron user again
- ```
- su metron
- cd
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
+
+1. Restart the parser topologies. Be sure to pass in the new parameter, `-ksp` or `--kafka_security_protocol`. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example `yaf`.
-19. Restart the parser topologies. Be sure to pass in the new parameter, “-ksp” or “--kafka_security_protocol.” Run this from the metron home directory.
```
for parser in bro snort; do
- ${METRON_HOME}/bin/start_parser_topology.sh -z ${ZOOKEEPER}:2181 -s ${parser} -ksp SASL_PLAINTEXT -e storm-config.json
+ ${METRON_HOME}/bin/start_parser_topology.sh \
+ -z ${ZOOKEEPER} \
+ -s ${parser} \
+ -ksp SASL_PLAINTEXT \
+ -e /home/metron/storm-config.json;
done
```
-20. Now restart the enrichment and indexing topologies.
+1. Restart the Enrichment and Indexing topologies.
+
+ ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
+
+Metron should be ready to receive data.
+
+Push Data
+---------
+1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+
```
- ${METRON_HOME}/bin/start_enrichment_topology.sh
- ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST}
+ --security-protocol SASL_PLAINTEXT \
+ --topic bro
+ ```
+
+1. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you’ve chosen.
+
```
+ curl -XGET "${ELASTICSEARCH}/bro*/_search"
+ curl -XGET "${ELASTICSEARCH}/bro*/_count"
+ ```
+
+1. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
+
+More Information
+----------------
+
+### Kerberos
+
+Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn’t download this list, you can also check the principal manually by running the following against the keytab.
+
+```
+klist -kt /etc/security/keytabs/<keytab-file-name>
+```
+
+E.g.
+
+```
+klist -kt /etc/security/keytabs/hbase.headless.keytab
+Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab
+KVNO Timestamp Principal
+---- ----------------- --------------------------------------------------------
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+ 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
+```
+
+### Kafka with Kerberos enabled
+
+#### Write data to a topic with SASL
+
+```
+cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
+```
+
+#### View topic data from latest offset with SASL
-Metron should be ready to receieve data.
+```
+${KAFKA_HOME}/bin/kafka-console-consumer.sh \
+ --zookeeper ${ZOOKEEPER} \
+ --security-protocol PLAINTEXTSASL \
+ --topic yaf
+```
-## Push Data
-See [Push Data](vagrant/Kerberos-setup.md)
+#### Modify the sensor-stubs to send logs via SASL
+```
+sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
+for sensorstub in bro snort; do
+ service sensor-stubs stop ${sensorstub};
+ service sensor-stubs start ${sensorstub};
+done
+```
-### Other useful commands
-See [Other useful commands](vagrant/Kerberos-setup.md)
+### References
-#### References
* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
http://git-wip-us.apache.org/repos/asf/metron/blob/f36db22e/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index adb48fd..012b7b6 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -169,10 +169,10 @@ The MPack can allow Metron to be installed and then Kerberized, or installed on
* Storm (and Metron) must be restarted after Metron is installed on an already Kerberized cluster. Several Storm configs get updated, and Metron will be unable to write to Kafka without a restart.
* Kerberizing a cluster with an existing Metron already has restarts of all services during Kerberization, so it's unneeded.
-Instructions for setup on Full Dev can be found at [Kerberos-setup.md](vagrant/Kerberos-setup.md). These instructions can also be used for setting up KDC and testing.
+Instructions for setup on Full Dev can be found at [Kerberos-ambari-setup.md](Kerberos-ambari-setup.md). These instructions reference the manual install instructions.
### Kerberos Without an MPack
-Using the MPack is preferred, but instructions for Kerberizing manually can be found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md)
+Using the MPack is preferred, but instructions for Kerberizing manually can be found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md). These instructions are reference by the Ambari Kerberos install instructions and include commands for setting up a KDC.
## TODO
- Support Ubuntu deployments
http://git-wip-us.apache.org/repos/asf/metron/blob/f36db22e/metron-deployment/vagrant/Kerberos-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/Kerberos-setup.md b/metron-deployment/vagrant/Kerberos-setup.md
deleted file mode 100644
index 759d63d..0000000
--- a/metron-deployment/vagrant/Kerberos-setup.md
+++ /dev/null
@@ -1,411 +0,0 @@
-Kerberos Setup
-==============
-
-This document provides instructions for kerberizing Metron's Vagrant-based development environments; "Quick Dev" and "Full Dev". These instructions do not cover the Ambari MPack or sensors. General Kerberization notes can be found in the metron-deployment [README.md](../README.md).
-
-* [Setup](#setup)
-* [Create a KDC](#create-a-kdc)
-* [Enable Kerberos](#enable-kerberos)
-* [Kafka Authorization](#kafka-authorization)
-* [HBase Authorization](#hbase-authorization)
-* [Storm Authorization](#storm-authorization)
-* [Start Metron](#start-metron)
-
-Setup
------
-
-1. Deploy a Vagrant development environment; either [Full Dev](full-dev-platform) or [Quick Dev](quick-dev-platform).
-
-1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.
-
- ```
- export ZOOKEEPER=node1:2181
- export ELASTICSEARCH=node1:9200
- export KAFKA=node1:6667
-
- export HDP_HOME="/usr/hdp/current"
- export KAFKA_HOME="${HDP_HOME}/kafka-broker"
- export METRON_VERSION="0.4.0"
- export METRON_HOME="/usr/metron/${METRON_VERSION}"
- ```
-
-1. Execute the following commands as root.
-
- ```
- sudo su -
- ```
-
-1. Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.
-
- ```
- for topology in bro snort enrichment indexing; do
- storm kill $topology;
- done
- ```
-
-1. Create the `metron` user's home directory in HDFS.
-
- ```
- sudo -u hdfs hdfs dfs -mkdir /user/metron
- sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
- sudo -u hdfs hdfs dfs -chmod 770 /user/metron
- ```
-
-Create a KDC
-------------
-
-1. Install dependencies.
-
- ```
- yum -y install krb5-server krb5-libs krb5-workstation
- ```
-
-1. Define the host, `node1`, as the KDC.
-
- ```
- sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
- cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
- ```
-
-1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
-
- ```
- kdb5_util create -s
-
- /etc/rc.d/init.d/krb5kdc start
- chkconfig krb5kdc on
-
- /etc/rc.d/init.d/kadmin start
- chkconfig kadmin on
- ```
-
-1. Setup the `admin` and `metron` principals. You'll `kinit` as the `metron` principal when running topologies. Make sure to remember the passwords.
-
- ```
- kadmin.local -q "addprinc admin/admin"
- kadmin.local -q "addprinc metron"
- ```
-
-Enable Kerberos
----------------
-
-1. In [Ambari](http://node1:8080), setup Storm to use Kerberos and run worker jobs as the submitting user.
-
- a. Add the following properties to the custom storm-site:
-
- ```
- topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']
- nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
- supervisor.run.worker.as.user=true
- ```
-
- b. In the Storm config section in Ambari, choose “Add Property” under custom storm-site:
-
- 
-
- c. In the dialog window, choose the “bulk property add mode” toggle button and add the below values:
-
- 
-
-1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
-
- a. For this exercise, choose existing MIT KDC (this is what we setup and installed in the previous steps.)
-
- 
-
- 
-
- b. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin principal will end up as admin/admin@EXAMPLE.COM when testing the KDC. Use the password you entered during the step for adding the admin principal.
-
- 
-
- c. Click through to “Start and Test Services.” Let the cluster spin up, but don't worry about starting up Metron via Ambari - we're going to run the parsers manually against the rest of the Hadoop cluster Kerberized. The wizard will fail at starting Metron, but this is OK. Click “continue.” When you’re finished, the custom storm-site should look similar to the following:
-
- 
-
-1. Create a Metron keytab
-
- ```
- kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
- cp metron.headless.keytab /etc/security/keytabs
- chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
- chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
-
-Kafka Authorization
--------------------
-
-1. Acquire a Kerberos ticket using the `metron` principal.
-
- ```
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
-
-1. Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for 'yaf' telemetry.
-
- ```
- ${KAFKA_HOME}/bin/kafka-topics.sh \
- --zookeeper ${ZOOKEEPER} \
- --create \
- --topic yaf \
- --partitions 1 \
- --replication-factor 1
- ```
-
-1. Setup Kafka ACLs for the `bro`, `snort`, `enrichments`, and `indexing` topics. Run the same command against any additional topics that you might be using; for example `yaf`.
-
- ```
- export KERB_USER=metron
-
- for topic in bro snort enrichments indexing; do
- ${KAFKA_HOME}/bin/kafka-acls.sh \
- --authorizer kafka.security.auth.SimpleAclAuthorizer \
- --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
- --add \
- --allow-principal User:${KERB_USER} \
- --topic ${topic}
- done
- ```
-
-1. Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.
-
- ```
- export KERB_USER=metron
-
- for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
- ${KAFKA_HOME}/bin/kafka-acls.sh \
- --authorizer kafka.security.auth.SimpleAclAuthorizer \
- --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
- --add \
- --allow-principal User:${KERB_USER} \
- --group ${group}
- done
- ```
-
-1. Add the `metron` principal to the `kafka-cluster` ACL.
-
- ```
- ${KAFKA_HOME}/bin/kafka-acls.sh \
- --authorizer kafka.security.auth.SimpleAclAuthorizer \
- --authorizer-properties zookeeper.connect=${ZOOKEEPER} \
- --add \
- --allow-principal User:${KERB_USER} \
- --cluster kafka-cluster
- ```
-
-HBase Authorization
--------------------
-
-1. Acquire a Kerberos ticket using the `hbase` principal
-
- ```
- kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
- ```
-
-1. Grant permissions for the HBase tables used in Metron.
-
- ```
- echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
- echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
-
-1. If you are using the Profiler, do the same for its HBase table.
-
- ```
- echo "create 'profiler', 'P'" | hbase shell
- echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
- ```
-
-Storm Authorization
--------------------
-
-1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
-
- ```
- su metron
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
-
-1. Create the directory `/home/metron/.storm` and switch to that directory.
-
- ```
- mkdir /home/metron/.storm
- cd /home/metron/.storm
- ```
-
-1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
-
- ```
- cat << EOF > client_jaas.conf
- StormClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useTicketCache=true
- renewTicket=true
- serviceName="nimbus";
- };
- Client {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="zookeeper"
- principal="metron@EXAMPLE.COM";
- };
- KafkaClient {
- com.sun.security.auth.module.Krb5LoginModule required
- useKeyTab=true
- keyTab="/etc/security/keytabs/metron.headless.keytab"
- storeKey=true
- useTicketCache=false
- serviceName="kafka"
- principal="metron@EXAMPLE.COM";
- };
- EOF
- ```
-
-1. Create a YAML file at `/home/metron/.storm/storm.yaml`. This should point to the client JAAS file. Set the array of nimbus hosts accordingly.
-
- ```
- cat << EOF > /home/metron/.storm/storm.yaml
- nimbus.seeds : ['node1']
- java.security.auth.login.config : '/home/metron/.storm/client_jaas.conf'
- storm.thrift.transport : 'org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin'
- EOF
- ```
-
-1. Create an auxiliary storm configuration file at `/home/metron/storm-config.json`. Note the login config option in the file points to the client JAAS file.
-
- ```
- cat << EOF > /home/metron/storm-config.json
- {
- "topology.worker.childopts" : "-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf"
- }
- EOF
- ```
-
-1. Configure the Enrichment, Indexing and the Profiler topologies to use the client JAAS file. Add the following properties to each of the topology properties files.
-
- ```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
- ```
-
- * `${METRON_HOME}/config/enrichment.properties`
- * `${METRON_HOME}/config/elasticsearch.properties`
- * `${METRON_HOME}/config/profiler.properties`
-
- Use the following command to automate this step.
-
- ```
- for file in enrichment.properties elasticsearch.properties profiler.properties; do
- echo ${file}
- sed -i "s/^kafka.security.protocol=.*/kafka.security.protocol=PLAINTEXTSASL/" "${METRON_HOME}/config/${file}"
- sed -i "s/^topology.worker.childopts=.*/topology.worker.childopts=-Djava.security.auth.login.config=\/home\/metron\/.storm\/client_jaas.conf/" "${METRON_HOME}/config/${file}"
- done
- ```
-
-Start Metron
-------------
-
-1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
-
- ```
- su metron
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
-
-1. Restart the parser topologies. Be sure to pass in the new parameter, `-ksp` or `--kafka_security_protocol`. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example `yaf`.
-
- ```
- for parser in bro snort; do
- ${METRON_HOME}/bin/start_parser_topology.sh \
- -z ${ZOOKEEPER} \
- -s ${parser} \
- -ksp SASL_PLAINTEXT \
- -e /home/metron/storm-config.json;
- done
- ```
-
-1. Restart the Enrichment and Indexing topologies.
-
- ```
- ${METRON_HOME}/bin/start_enrichment_topology.sh
- ${METRON_HOME}/bin/start_elasticsearch_topology.sh
- ```
-
-1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
-
- ```
- cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
- --broker-list ${KAFKA} \
- --security-protocol SASL_PLAINTEXT \
- --topic bro
- ```
-
-1. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you’ve chosen.
-
- ```
- curl -XGET "${ELASTICSEARCH}/bro*/_search"
- curl -XGET "${ELASTICSEARCH}/bro*/_count"
- ```
-
-1. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
-
-More Information
-----------------
-
-### Kerberos
-
-Unsure of your Kerberos principal associated with a keytab? There are a couple ways to get this. One is via the list of principals that Ambari provides via downloadable csv. If you didn’t download this list, you can also check the principal manually by running the following against the keytab.
-
-```
-klist -kt /etc/security/keytabs/<keytab-file-name>
-```
-
-E.g.
-
-```
-klist -kt /etc/security/keytabs/hbase.headless.keytab
-Keytab name: FILE:/etc/security/keytabs/hbase.headless.keytab
-KVNO Timestamp Principal
----- ----------------- --------------------------------------------------------
- 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
- 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
- 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
- 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
- 1 03/28/17 19:29:36 hbase-metron_cluster@EXAMPLE.COM
-```
-
-### Kafka with Kerberos enabled
-
-#### Write data to a topic with SASL
-
-```
-cat sample-yaf.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
- --broker-list ${KAFKA} \
- --security-protocol PLAINTEXTSASL \
- --topic yaf
-```
-
-#### View topic data from latest offset with SASL
-
-```
-${KAFKA_HOME}/bin/kafka-console-consumer.sh \
- --zookeeper ${ZOOKEEPER} \
- --security-protocol PLAINTEXTSASL \
- --topic yaf
-```
-
-#### Modify the sensor-stubs to send logs via SASL
-```
-sed -i 's/node1:6667 --topic/node1:6667 --security-protocol PLAINTEXTSASL --topic/' /opt/sensor-stubs/bin/start-*-stub
-for sensorstub in bro snort; do
- service sensor-stubs stop ${sensorstub};
- service sensor-stubs start ${sensorstub};
-done
-```
-
-### References
-
-* [https://github.com/apache/storm/blob/master/SECURITY.md](https://github.com/apache/storm/blob/master/SECURITY.md)
http://git-wip-us.apache.org/repos/asf/metron/blob/f36db22e/metron-deployment/vagrant/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/README.md b/metron-deployment/vagrant/README.md
index ae49285..b629a1f 100644
--- a/metron-deployment/vagrant/README.md
+++ b/metron-deployment/vagrant/README.md
@@ -1,6 +1,5 @@
# Vagrant Deployment
-- Kerberos Setup
- Codelab Platform
- Fast CAPA Test Platform
- Full Dev Platform
http://git-wip-us.apache.org/repos/asf/metron/blob/f36db22e/site-book/bin/generate-md.sh
----------------------------------------------------------------------
diff --git a/site-book/bin/generate-md.sh b/site-book/bin/generate-md.sh
index d428fbc..686635c 100755
--- a/site-book/bin/generate-md.sh
+++ b/site-book/bin/generate-md.sh
@@ -57,12 +57,12 @@ EXCLUSION_LIST=(
## Each entry is a file path, relative to $METRON_SOURCE.
## Note: any images in site-book/src/site/src-resources/images/ will also be included.
RESOURCE_LIST=(
- metron-deployment/vagrant/readme-images/ambari-storm-site-properties.png
- metron-deployment/vagrant/readme-images/ambari-storm-site.png
- metron-deployment/vagrant/readme-images/custom-storm-site-final.png
- metron-deployment/vagrant/readme-images/enable-kerberos-configure-kerberos.png
- metron-deployment/vagrant/readme-images/enable-kerberos-started.png
- metron-deployment/vagrant/readme-images/enable-kerberos.png
+ metron-deployment/readme-images/ambari-storm-site-properties.png
+ metron-deployment/readme-images/ambari-storm-site.png
+ metron-deployment/readme-images/custom-storm-site-final.png
+ metron-deployment/readme-images/enable-kerberos-configure-kerberos.png
+ metron-deployment/readme-images/enable-kerberos-started.png
+ metron-deployment/readme-images/enable-kerberos.png
metron-platform/metron-parsers/parser_arch.png
metron-platform/metron-indexing/indexing_arch.png
metron-platform/metron-enrichment/enrichment_arch.png
@@ -73,12 +73,15 @@ RESOURCE_LIST=(
## that needs an href re-written to match a resource in the images/ directory. Odd fields are the corresponding
## one-line sed script, in single quotes, that does the rewrite. See below for examples.
HREF_REWRITE_LIST=(
- metron-deployment/vagrant/Kerberos-setup.md 's#(readme-images/ambari-storm-site-properties.png)#(../../images/ambari-storm-site-properties.png)#g'
- metron-deployment/vagrant/Kerberos-setup.md 's#(readme-images/ambari-storm-site.png)#(../../images/ambari-storm-site.png)#g'
- metron-deployment/vagrant/Kerberos-setup.md 's#(readme-images/custom-storm-site-final.png)#(../../images/custom-storm-site-final.png)#g'
- metron-deployment/vagrant/Kerberos-setup.md 's#(readme-images/enable-kerberos-configure-kerberos.png)#(../../images/enable-kerberos-configure-kerberos.png)#g'
- metron-deployment/vagrant/Kerberos-setup.md 's#(readme-images/enable-kerberos-started.png)#(../../images/enable-kerberos-started.png)#g'
- metron-deployment/vagrant/Kerberos-setup.md 's#(readme-images/enable-kerberos.png)#(../../images/enable-kerberos.png)#g'
+ metron-deployment/Kerberos-manual-setup.md 's#(readme-images/ambari-storm-site-properties.png)#(../images/ambari-storm-site-properties.png)#g'
+ metron-deployment/Kerberos-manual-setup.md 's#(readme-images/ambari-storm-site.png)#(../images/ambari-storm-site.png)#g'
+ metron-deployment/Kerberos-manual-setup.md 's#(readme-images/custom-storm-site-final.png)#(../images/custom-storm-site-final.png)#g'
+ metron-deployment/Kerberos-manual-setup.md 's#(readme-images/enable-kerberos-configure-kerberos.png)#(../images/enable-kerberos-configure-kerberos.png)#g'
+ metron-deployment/Kerberos-manual-setup.md 's#(readme-images/enable-kerberos-started.png)#(../images/enable-kerberos-started.png)#g'
+ metron-deployment/Kerberos-manual-setup.md 's#(readme-images/enable-kerberos.png)#(../images/enable-kerberos.png)#g'
+ metron-deployment/Kerberos-ambari-setup.md 's#(readme-images/enable-kerberos-configure-kerberos.png)#(../images/enable-kerberos-configure-kerberos.png)#g'
+ metron-deployment/Kerberos-ambari-setup.md 's#(readme-images/enable-kerberos-started.png)#(../images/enable-kerberos-started.png)#g'
+ metron-deployment/Kerberos-ambari-setup.md 's#(readme-images/enable-kerberos.png)#(../images/enable-kerberos.png)#g'
metron-platform/metron-enrichment/README.md 's#(enrichment_arch.png)#(../../images/enrichment_arch.png)#g'
metron-platform/metron-indexing/README.md 's#(indexing_arch.png)#(../../images/indexing_arch.png)#g'
metron-platform/metron-parsers/README.md 's#(parser_arch.png)#(../../images/parser_arch.png)#g'
[22/44] metron git commit: METRON-952 Travis CI Link in README
Pointing to Old Repo closes apache/incubator-metron#587
Posted by ma...@apache.org.
METRON-952 Travis CI Link in README Pointing to Old Repo closes apache/incubator-metron#587
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/4b3ccac3
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/4b3ccac3
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/4b3ccac3
Branch: refs/heads/Metron_0.4.0
Commit: 4b3ccac3649d9ed384774fe0573aa1812b87e5a8
Parents: 345eba8
Author: cstella <ce...@gmail.com>
Authored: Mon May 15 09:24:38 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Mon May 15 09:24:38 2017 -0400
----------------------------------------------------------------------
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/4b3ccac3/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 53a84c4..06e5471 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,4 @@
-[](https://travis-ci.org/apache/incubator-metron)
+[](https://travis-ci.org/apache/metron)
# Apache Metron
[13/44] metron git commit: METRON-944 markdown table causes infinite
hanging in site-book'
s doxia-module-markdown (justinleet) closes apache/incubator-metron#575
Posted by ma...@apache.org.
METRON-944 markdown table causes infinite hanging in site-book's doxia-module-markdown (justinleet) closes apache/incubator-metron#575
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/4662c775
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/4662c775
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/4662c775
Branch: refs/heads/Metron_0.4.0
Commit: 4662c775c033fec3e23b8df138ca489b5a96fea3
Parents: 21b79dc
Author: justinleet <ju...@gmail.com>
Authored: Tue May 9 16:27:28 2017 -0400
Committer: leet <le...@apache.org>
Committed: Tue May 9 16:27:28 2017 -0400
----------------------------------------------------------------------
metron-interface/metron-rest/README.md | 85 +++++++++++++++++------------
1 file changed, 50 insertions(+), 35 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/4662c775/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index 93beb0c..674f524 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -14,19 +14,19 @@ This module provides a RESTful API for interacting with Metron.
### From Source
1. Package the application with Maven:
-```
+ ```
mvn clean package
-```
+ ```
1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
-```
+ ```
config
rest_application.yml
bin
metron-rest
lib
metron-rest-$METRON_VERSION.jar
-```
+ ```
1. Copy the `$METRON_HOME/bin/metron-rest` script to `/etc/init.d/metron-rest`
@@ -35,36 +35,47 @@ lib
1. Deploy the RPM at `/incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
1. Install the RPM with:
- ```
- rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
- ```
+ ```
+rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
+ ```
## Configuration
The REST application depends on several configuration parameters:
-| Environment Variable | Description | Required | Default |
-| ------------------------------------- | ----------------------------------------------------------------- | -------- | ---------------------- |
-| METRON_USER | Run the application as this user | Optional | metron |
-| METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
-| METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
-| METRON_REST_PORT | REST application port | Optional | 8082 |
-| METRON_JVMFLAGS | JVM flags added to the start command | Optional | |
-| METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see [below](#spring-profiles)) | Optional | |
-| METRON_JDBC_DRIVER | JDBC driver class | Required | |
-| METRON_JDBC_URL | JDBC url | Required | |
-| METRON_JDBC_USERNAME | JDBC username | Required | |
-| METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle | Required | |
-| METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
-| METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
-| METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
-| METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional | |
-| ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) | Required | |
-| BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) | Required | |
-| HDFS_URL | HDFS url or `fs.defaultFS` Hadoop setting (ex. hdfs://node1:8020) | Required | |
-| SECURITY_ENABLED | Enables Kerberos support | Optional | false |
-| METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional | |
-| METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional | |
+### REQUIRED
+No optional parameter has a default.
+
+| Environment Variable | Description
+| ------------------------------------- | -----------
+| METRON_JDBC_DRIVER | JDBC driver class
+| METRON_JDBC_URL | JDBC url
+| METRON_JDBC_USERNAME | JDBC username
+| METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle
+| ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181)
+| BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667)
+| HDFS_URL | HDFS url or `fs.defaultFS` Hadoop setting (ex. hdfs://node1:8020)
+
+### Optional - With Defaults
+| Environment Variable | Description | Required | Default
+| ------------------------------------- | ----------------------------------------------------------------- | -------- | -------
+| METRON_USER | Run the application as this user | Optional | metron
+| METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/
+| METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/
+| METRON_REST_PORT | REST application port | Optional | 8082
+| METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled
+| METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp
+| METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns
+| SECURITY_ENABLED | Enables Kerberos support | Optional | false
+
+### Optional - Blank Defaults
+| Environment Variable | Description | Required
+| ------------------------------------- | ----------------------------------------------------------------- | --------
+| METRON_JVMFLAGS | JVM flags added to the start command | Optional
+| METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see [below](#spring-profiles)) | Optional
+| METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional
+| METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional
+| METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional
These are set in the `/etc/sysconfig/metron` file.
@@ -94,27 +105,27 @@ For example, the following configures the application for MySQL:
1. Install MySQL if not already available (this example uses version 5.7, installation instructions can be found [here](https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html))
1. Create a metron user and REST database and permission the user for that database:
-```
+ ```
CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd';
CREATE DATABASE IF NOT EXISTS metronrest;
GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
-```
+ ```
1. Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:
-```
+ ```
cd $METRON_HOME/lib
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
tar xf mysql-connector-java-5.1.41.tar.gz
-```
+ ```
1. Edit these variables in `/etc/sysconfig/metron` to configure the REST application for MySQL:
-```
+ ```
METRON_JDBC_DRIVER="com.mysql.jdbc.Driver"
METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest"
METRON_JDBC_USERNAME="metron"
METRON_JDBC_PLATFORM="mysql"
METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
-```
+ ```
## Usage
@@ -599,6 +610,7 @@ Start the [metron-docker](../../metron-docker) environment. Build the metron-re
mvn clean package
mvn spring-boot:run -Drun.profiles=docker,dev
```
+
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
### Quick Dev
@@ -608,12 +620,14 @@ Start the [Quick Dev](../../metron-deployment/vagrant/quick-dev-platform) enviro
mvn clean package
mvn spring-boot:run -Drun.profiles=vagrant,dev
```
+
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
To run the application locally on the Quick Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/sysconfig/metron`:
```
METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
+
and start the application:
```
service metron-rest start
@@ -625,6 +639,7 @@ METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
METRON_SPRING_OPTIONS="--kerberos.enabled=true"
```
+
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
## License
[34/44] metron git commit: METRON-958: PCAP Query job throws
exception when no files returned by time range query closes apache/metron#593
Posted by ma...@apache.org.
METRON-958: PCAP Query job throws exception when no files returned by time range query closes apache/metron#593
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e2197316
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e2197316
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e2197316
Branch: refs/heads/Metron_0.4.0
Commit: e2197316df5754aba601134ecce19b12092b6f37
Parents: 356881a
Author: mmiklavc <mi...@gmail.com>
Authored: Sun May 21 21:08:45 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Sun May 21 21:08:45 2017 -0400
----------------------------------------------------------------------
.../org/apache/metron/pcap/PcapJobTest.java | 110 +-------------
.../PcapTopologyIntegrationTest.java | 123 +++++++--------
.../apache/metron/pcap/PcapFilenameHelper.java | 77 ++++++++++
.../java/org/apache/metron/pcap/PcapHelper.java | 35 ++---
.../java/org/apache/metron/pcap/mr/PcapJob.java | 105 +++++--------
.../metron/pcap/utils/FileFilterUtil.java | 138 +++++++++++++++++
.../metron/pcap/PcapFilenameHelperTest.java | 75 ++++++++++
.../metron/pcap/mr/FileFilterUtilTest.java | 150 +++++++++++++++++++
8 files changed, 568 insertions(+), 245 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/PcapJobTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/PcapJobTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/PcapJobTest.java
index 81725d8..3536a7e 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/PcapJobTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/PcapJobTest.java
@@ -7,7 +7,7 @@
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
@@ -18,10 +18,10 @@
package org.apache.metron.pcap;
-import com.google.common.collect.Iterables;
+import static java.lang.Long.toUnsignedString;
+import static org.hamcrest.CoreMatchers.equalTo;
+
import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.metron.common.utils.timestamp.TimestampConverters;
@@ -29,106 +29,9 @@ import org.apache.metron.pcap.mr.PcapJob;
import org.junit.Assert;
import org.junit.Test;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-
-import static java.lang.Long.toUnsignedString;
-import static org.hamcrest.CoreMatchers.equalTo;
-
public class PcapJobTest {
@Test
- public void test_getPaths_NoFiles() throws Exception {
- PcapJob job;
- {
- final List<Path> inputFiles = new ArrayList<Path>() {{
- }};
- job = new PcapJob() {
- @Override
- protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
- return inputFiles;
- }
- };
- Iterable<String> paths = job.getPaths(null, null, 0, 1000);
- Assert.assertTrue(Iterables.isEmpty(paths));
- }
- }
-
- @Test
- public void test_getPaths_leftEdge() throws Exception {
- PcapJob job;
- {
- final List<Path> inputFiles = new ArrayList<Path>() {{
- add(new Path("/apps/metron/pcap/pcap_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new Path("/apps/metron/pcap/pcap_pcap_1561589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- }};
- job = new PcapJob() {
- @Override
- protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
- return inputFiles;
- }
- };
- Iterable<String> paths = job.getPaths(null, null, 0, TimestampConverters.MILLISECONDS.toNanoseconds(System.currentTimeMillis()));
- Assert.assertEquals(1, Iterables.size(paths));
- }
- }
-
- @Test
- public void test_getPaths_rightEdge() throws Exception {
- PcapJob job;
- {
- final List<Path> inputFiles = new ArrayList<Path>() {{
- add(new Path("/apps/metron/pcap/pcap0_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new Path("/apps/metron/pcap/pcap1_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- }};
- job = new PcapJob() {
- @Override
- protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
- return inputFiles;
- }
- };
- Iterable<String> paths = job.getPaths(null, null, 1461589333993573000L - 1L, 1461589333993573000L + 1L);
- Assert.assertEquals(2, Iterables.size(paths));
- }
- {
- final List<Path> inputFiles = new ArrayList<Path>() {{
- add(new Path("/apps/metron/pcap/pcap0_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new Path("/apps/metron/pcap/pcap1_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new Path("/apps/metron/pcap/pcap1_pcap_1461589334993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- }};
- job = new PcapJob() {
- @Override
- protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
- return inputFiles;
- }
- };
- Iterable<String> paths = job.getPaths(null, null, 1461589334993573000L - 1L, 1461589334993573000L + 1L);
- Assert.assertEquals(2, Iterables.size(paths));
- }
- }
-
- @Test
- public void test_getPaths_bothEdges() throws Exception {
- PcapJob job;
- {
- final List<Path> inputFiles = new ArrayList<Path>() {{
- add(new Path("/apps/metron/pcap/pcap_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new Path("/apps/metron/pcap/pcap_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new Path("/apps/metron/pcap/pcap1_pcap_1461589334993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- }};
- job = new PcapJob() {
- @Override
- protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
- return inputFiles;
- }
- };
- Iterable<String> paths = job.getPaths(null, null, 0, TimestampConverters.MILLISECONDS.toNanoseconds(System.currentTimeMillis()));
- Assert.assertEquals(3, Iterables.size(paths));
- }
- }
-
- @Test
public void partition_gives_value_in_range() throws Exception {
long start = 1473897600000000000L;
long end = TimestampConverters.MILLISECONDS.toNanoseconds(1473995927455L);
@@ -138,6 +41,9 @@ public class PcapJobTest {
conf.set(PcapJob.WIDTH_CONF, "" + PcapJob.findWidth(start, end, 10));
PcapJob.PcapPartitioner partitioner = new PcapJob.PcapPartitioner();
partitioner.setConf(conf);
- Assert.assertThat("Partition not in range", partitioner.getPartition(new LongWritable(1473978789181189000L), new BytesWritable(), 10), equalTo(8));
+ Assert.assertThat("Partition not in range",
+ partitioner.getPartition(new LongWritable(1473978789181189000L), new BytesWritable(), 10),
+ equalTo(8));
}
+
}
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
index d6d54dc..7d1dba8 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
@@ -23,6 +23,17 @@ import com.google.common.base.Predicate;
import com.google.common.collect.Collections2;
import com.google.common.collect.Iterables;
import com.google.common.collect.Lists;
+import java.io.ByteArrayOutputStream;
+import java.io.File;
+import java.io.FilenameFilter;
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import javax.annotation.Nullable;
import kafka.consumer.ConsumerIterator;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
@@ -55,13 +66,6 @@ import org.json.simple.JSONObject;
import org.junit.Assert;
import org.junit.Test;
-import javax.annotation.Nullable;
-import java.io.ByteArrayOutputStream;
-import java.io.File;
-import java.io.FilenameFilter;
-import java.io.IOException;
-import java.util.*;
-
public class PcapTopologyIntegrationTest {
final static String KAFKA_TOPIC = "pcap";
private static String BASE_DIR = "pcap";
@@ -69,22 +73,7 @@ public class PcapTopologyIntegrationTest {
private static String QUERY_DIR = BASE_DIR + "/query";
private String topologiesDir = "src/main/flux";
private String targetDir = "target";
- private File getOutDir(String targetDir) {
- File outDir = new File(new File(targetDir), DATA_DIR);
- if (!outDir.exists()) {
- outDir.mkdirs();
- }
-
- return outDir;
- }
- private File getQueryDir(String targetDir) {
- File outDir = new File(new File(targetDir), QUERY_DIR);
- if (!outDir.exists()) {
- outDir.mkdirs();
- }
- return outDir;
- }
private static void clearOutDir(File outDir) {
for(File f : outDir.listFiles()) {
f.delete();
@@ -100,43 +89,6 @@ public class PcapTopologyIntegrationTest {
}).length;
}
- private static Iterable<Map.Entry<byte[], byte[]>> readPcaps(Path pcapFile, boolean withHeaders) throws IOException {
- SequenceFile.Reader reader = new SequenceFile.Reader(new Configuration(),
- SequenceFile.Reader.file(pcapFile)
- );
- List<Map.Entry<byte[], byte[]> > ret = new ArrayList<>();
- IntWritable key = new IntWritable();
- BytesWritable value = new BytesWritable();
- while (reader.next(key, value)) {
- byte[] pcapWithHeader = value.copyBytes();
- //if you are debugging and want the hex dump of the packets, uncomment the following:
-
- //for(byte b : pcapWithHeader) {
- // System.out.print(String.format("%02x", b));
- //}
- //System.out.println("");
-
- long calculatedTs = PcapHelper.getTimestamp(pcapWithHeader);
- {
- List<PacketInfo> info = PcapHelper.toPacketInfo(pcapWithHeader);
- for(PacketInfo pi : info) {
- Assert.assertEquals(calculatedTs, pi.getPacketTimeInNanos());
- //IF you are debugging and want to see the packets, uncomment the following.
- //System.out.println( Long.toUnsignedString(calculatedTs) + " => " + pi.getJsonDoc());
- }
- }
- if(withHeaders) {
- ret.add(new AbstractMap.SimpleImmutableEntry<>(Bytes.toBytes(calculatedTs), pcapWithHeader));
- }
- else {
- byte[] pcapRaw = new byte[pcapWithHeader.length - PcapHelper.GLOBAL_HEADER_SIZE - PcapHelper.PACKET_HEADER_SIZE];
- System.arraycopy(pcapWithHeader, PcapHelper.GLOBAL_HEADER_SIZE + PcapHelper.PACKET_HEADER_SIZE, pcapRaw, 0, pcapRaw.length);
- ret.add(new AbstractMap.SimpleImmutableEntry<>(Bytes.toBytes(calculatedTs), pcapRaw));
- }
- }
- return Iterables.limit(ret, 2*(ret.size()/2));
- }
-
@Test
public void testTimestampInPacket() throws Exception {
testTopology(new Function<Properties, Void>() {
@@ -561,6 +513,59 @@ public class PcapTopologyIntegrationTest {
}
}
+ private File getOutDir(String targetDir) {
+ File outDir = new File(new File(targetDir), DATA_DIR);
+ if (!outDir.exists()) {
+ outDir.mkdirs();
+ }
+ return outDir;
+ }
+
+ private File getQueryDir(String targetDir) {
+ File outDir = new File(new File(targetDir), QUERY_DIR);
+ if (!outDir.exists()) {
+ outDir.mkdirs();
+ }
+ return outDir;
+ }
+
+ private static Iterable<Map.Entry<byte[], byte[]>> readPcaps(Path pcapFile, boolean withHeaders) throws IOException {
+ SequenceFile.Reader reader = new SequenceFile.Reader(new Configuration(),
+ SequenceFile.Reader.file(pcapFile)
+ );
+ List<Map.Entry<byte[], byte[]> > ret = new ArrayList<>();
+ IntWritable key = new IntWritable();
+ BytesWritable value = new BytesWritable();
+ while (reader.next(key, value)) {
+ byte[] pcapWithHeader = value.copyBytes();
+ //if you are debugging and want the hex dump of the packets, uncomment the following:
+
+ //for(byte b : pcapWithHeader) {
+ // System.out.print(String.format("%02x", b));
+ //}
+ //System.out.println("");
+
+ long calculatedTs = PcapHelper.getTimestamp(pcapWithHeader);
+ {
+ List<PacketInfo> info = PcapHelper.toPacketInfo(pcapWithHeader);
+ for(PacketInfo pi : info) {
+ Assert.assertEquals(calculatedTs, pi.getPacketTimeInNanos());
+ //IF you are debugging and want to see the packets, uncomment the following.
+ //System.out.println( Long.toUnsignedString(calculatedTs) + " => " + pi.getJsonDoc());
+ }
+ }
+ if(withHeaders) {
+ ret.add(new AbstractMap.SimpleImmutableEntry<>(Bytes.toBytes(calculatedTs), pcapWithHeader));
+ }
+ else {
+ byte[] pcapRaw = new byte[pcapWithHeader.length - PcapHelper.GLOBAL_HEADER_SIZE - PcapHelper.PACKET_HEADER_SIZE];
+ System.arraycopy(pcapWithHeader, PcapHelper.GLOBAL_HEADER_SIZE + PcapHelper.PACKET_HEADER_SIZE, pcapRaw, 0, pcapRaw.length);
+ ret.add(new AbstractMap.SimpleImmutableEntry<>(Bytes.toBytes(calculatedTs), pcapRaw));
+ }
+ }
+ return Iterables.limit(ret, 2*(ret.size()/2));
+ }
+
public static void assertInOrder(Iterable<byte[]> packets) {
long previous = 0;
for(byte[] packet : packets) {
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapFilenameHelper.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapFilenameHelper.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapFilenameHelper.java
new file mode 100644
index 0000000..a6f8546
--- /dev/null
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapFilenameHelper.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap;
+
+import java.util.Arrays;
+
+/**
+ * Expects files in the following format
+ * pcap_$TOPIC_$TS_$PARTITION_$UUID
+ */
+public class PcapFilenameHelper {
+
+ public static final String PREFIX = "pcap_";
+
+ /**
+ * Extract kafka topic from pcap filename. Resilient to underscores and hyphens in the kafka
+ * topic name when splitting the value.
+ */
+ public static String getKafkaTopic(String pcapFilename) {
+ String[] tokens = stripPrefix(pcapFilename).split("_");
+ return String.join("_", Arrays.copyOfRange(tokens, 0, tokens.length - 3));
+ }
+
+ private static String stripPrefix(String s) {
+ return s.substring(PREFIX.length());
+ }
+
+ /**
+ * Gets unsigned long timestamp from the PCAP filename
+ *
+ * @return timestamp, or null if unable to parse
+ */
+ public static Long getTimestamp(String pcapFilename) {
+ String[] tokens = stripPrefix(pcapFilename).split("_");
+ try {
+ return Long.parseUnsignedLong(tokens[tokens.length - 3]);
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ /**
+ * Gets Kafka partition number from the PCAP filename
+ *
+ * @return partition, or null if unable to parse
+ */
+ public static Integer getKafkaPartition(String pcapFilename) {
+ String[] tokens = stripPrefix(pcapFilename).split("_");
+ try {
+ return Integer.parseInt(tokens[tokens.length - 2]);
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ public static String getUUID(String pcapFilename) {
+ String[] tokens = stripPrefix(pcapFilename).split("_");
+ return tokens[tokens.length - 1];
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
index e1ad3ca..bb7d9f0 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
@@ -19,8 +19,12 @@
package org.apache.metron.pcap;
import com.google.common.base.Joiner;
-import com.google.common.base.Splitter;
-import com.google.common.collect.Iterables;
+import java.io.EOFException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.log4j.Logger;
import org.apache.metron.spout.pcap.Endianness;
@@ -37,13 +41,6 @@ import org.krakenapps.pcap.packet.PcapPacket;
import org.krakenapps.pcap.util.Buffer;
import org.krakenapps.pcap.util.ByteOrderConverter;
-import java.io.EOFException;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
public class PcapHelper {
public static final int PACKET_HEADER_SIZE = 4*Integer.BYTES;
@@ -72,16 +69,6 @@ public class PcapHelper {
}
};
- public static Long getTimestamp(String filename) {
- try {
- return Long.parseUnsignedLong(Iterables.get(Splitter.on('_').split(filename), 2));
- }
- catch(Exception e) {
- //something went wrong here.
- return null;
- }
- }
-
/**
*
* @param topic
@@ -363,4 +350,14 @@ public class PcapHelper {
}
return messages;
}
+
+ public static boolean greaterThanOrEqualTo(long a, long b) {
+ return Long.compareUnsigned(a, b) >= 0;
+ }
+
+ public static boolean lessThanOrEqualTo(long a, long b) {
+ return Long.compareUnsigned(a, b) <= 0;
+ }
+
}
+
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
index 8d40e5f..62f9844 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
@@ -18,7 +18,20 @@
package org.apache.metron.pcap.mr;
+import static org.apache.metron.pcap.PcapHelper.greaterThanOrEqualTo;
+import static org.apache.metron.pcap.PcapHelper.lessThanOrEqualTo;
+
import com.google.common.base.Joiner;
+import java.io.IOException;
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.UUID;
+import java.util.stream.Stream;
+import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configurable;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
@@ -33,22 +46,19 @@ import org.apache.hadoop.mapreduce.Partitioner;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
-import org.apache.log4j.Logger;
import org.apache.metron.common.hadoop.SequenceFileIterable;
import org.apache.metron.pcap.PacketInfo;
import org.apache.metron.pcap.PcapHelper;
import org.apache.metron.pcap.filter.PcapFilter;
import org.apache.metron.pcap.filter.PcapFilterConfigurator;
import org.apache.metron.pcap.filter.PcapFilters;
-
-import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.*;
-import java.util.stream.Stream;
+import org.apache.metron.pcap.utils.FileFilterUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
public class PcapJob {
- private static final Logger LOG = Logger.getLogger(PcapJob.class);
+
+ private static final Logger LOG = LoggerFactory.getLogger(PcapJob.class);
public static final String START_TS_CONF = "start_ts";
public static final String END_TS_CONF = "end_ts";
public static final String WIDTH_CONF = "width";
@@ -110,7 +120,7 @@ public class PcapJob {
@Override
protected void map(LongWritable key, BytesWritable value, Context context) throws IOException, InterruptedException {
- if (Long.compareUnsigned(key.get(), start) >= 0 && Long.compareUnsigned(key.get(), end) <= 0) {
+ if (greaterThanOrEqualTo(key.get(), start) && lessThanOrEqualTo(key.get(), end)) {
// It is assumed that the passed BytesWritable value is always a *single* PacketInfo object. Passing more than 1
// object will result in the whole set being passed through if any pass the filter. We cannot serialize PacketInfo
// objects back to byte arrays, otherwise we could support more than one packet.
@@ -144,56 +154,6 @@ public class PcapJob {
}
}
- protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
- List<Path> ret = new ArrayList<>();
- RemoteIterator<LocatedFileStatus> filesIt = fs.listFiles(basePath, true);
- while(filesIt.hasNext()){
- ret.add(filesIt.next().getPath());
- }
- return ret;
- }
-
- public Iterable<String> getPaths(FileSystem fs, Path basePath, long begin, long end) throws IOException {
- List<String> ret = new ArrayList<>();
- Iterator<Path> files = listFiles(fs, basePath).iterator();
- /*
- The trick here is that we need a trailing left endpoint, because we only capture the start of the
- timeseries kept in the file.
- */
- boolean isFirst = true;
- Path leftEndpoint = files.hasNext()?files.next():null;
- if(leftEndpoint == null) {
- return ret;
- }
- {
- Long ts = PcapHelper.getTimestamp(leftEndpoint.getName());
- if(ts != null && Long.compareUnsigned(ts, begin) >= 0 && Long.compareUnsigned(ts, end) <= 0) {
- ret.add(leftEndpoint.toString());
- isFirst = false;
- }
- }
- while(files.hasNext()) {
- Path p = files.next();
- Long ts = PcapHelper.getTimestamp(p.getName());
- if(ts != null && Long.compareUnsigned(ts, begin) >= 0 && Long.compareUnsigned(ts, end) <= 0) {
- if(isFirst && leftEndpoint != null) {
- ret.add(leftEndpoint.toString());
- }
- if(isFirst) {
- isFirst = false;
- }
- ret.add(p.toString());
- }
- else {
- leftEndpoint = p;
- }
- }
- if(LOG.isDebugEnabled()) {
- LOG.debug("Including files " + Joiner.on(",").join(ret));
- }
- return ret;
- }
-
/**
* Returns a lazily-read Iterable over a set of sequence files
*/
@@ -207,9 +167,7 @@ public class PcapJob {
}
files.add(p);
}
- if (LOG.isDebugEnabled()) {
- LOG.debug(outputPath);
- }
+ LOG.debug("Output path={}", outputPath);
Collections.sort(files, (o1,o2) -> o1.getName().compareTo(o2.getName()));
return new SequenceFileIterable(files, config);
}
@@ -244,11 +202,14 @@ public class PcapJob {
, fs
, filterImpl
);
+ if (job == null) {
+ LOG.info("No files to process with specified date range.");
+ return new SequenceFileIterable(new ArrayList<>(), conf);
+ }
boolean completed = job.waitForCompletion(true);
if(completed) {
return readResults(outputPath, conf, fs);
- }
- else {
+ } else {
throw new RuntimeException("Unable to complete query due to errors. Please check logs for full errors.");
}
}
@@ -282,11 +243,25 @@ public class PcapJob {
job.setPartitionerClass(PcapPartitioner.class);
job.setOutputKeyClass(LongWritable.class);
job.setOutputValueClass(BytesWritable.class);
- SequenceFileInputFormat.addInputPaths(job, Joiner.on(',').join(getPaths(fs, basePath, beginNS, endNS )));
+ Iterable<String> filteredPaths = FileFilterUtil.getPathsInTimeRange(beginNS, endNS, listFiles(fs, basePath));
+ String inputPaths = Joiner.on(',').join(filteredPaths);
+ if (StringUtils.isEmpty(inputPaths)) {
+ return null;
+ }
+ SequenceFileInputFormat.addInputPaths(job, inputPaths);
job.setInputFormatClass(SequenceFileInputFormat.class);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
SequenceFileOutputFormat.setOutputPath(job, outputPath);
return job;
+ }
+ protected Iterable<Path> listFiles(FileSystem fs, Path basePath) throws IOException {
+ List<Path> ret = new ArrayList<>();
+ RemoteIterator<LocatedFileStatus> filesIt = fs.listFiles(basePath, true);
+ while (filesIt.hasNext()) {
+ ret.add(filesIt.next().getPath());
+ }
+ return ret;
}
+
}
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/utils/FileFilterUtil.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/utils/FileFilterUtil.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/utils/FileFilterUtil.java
new file mode 100644
index 0000000..bbbc4bb
--- /dev/null
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/utils/FileFilterUtil.java
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.utils;
+
+import static org.apache.metron.pcap.PcapHelper.greaterThanOrEqualTo;
+import static org.apache.metron.pcap.PcapHelper.lessThanOrEqualTo;
+
+import com.google.common.base.Joiner;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.pcap.PcapFilenameHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class FileFilterUtil {
+
+ private static final Logger LOG = LoggerFactory.getLogger(FileFilterUtil.class);
+
+ private FileFilterUtil() {
+ }
+
+ /*
+ * The trick here is that we need a trailing left endpoint, because we only capture the start of the
+ * timeseries kept in the file.
+ */
+ public static Iterable<String> getPathsInTimeRange(long beginTs, long endTs,
+ Iterable<Path> files) {
+ Map<Integer, List<Path>> filesByPartition = getFilesByPartition(files);
+ List<String> filteredFiles = filterByTimestampLT(beginTs, endTs, filesByPartition);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Including files " + Joiner.on(",").join(filteredFiles));
+ }
+ return filteredFiles;
+ }
+
+ public static Map<Integer, List<Path>> getFilesByPartition(Iterable<Path> files) {
+ Iterator<Path> filesIt = files.iterator();
+ Map<Integer, List<Path>> filesByPartition = new HashMap<>();
+ while (filesIt.hasNext()) {
+ Path p = filesIt.next();
+ Integer partition = PcapFilenameHelper.getKafkaPartition(p.getName());
+ if (!filesByPartition.containsKey(partition)) {
+ filesByPartition.put(partition, new ArrayList<>());
+ }
+ filesByPartition.get(partition).add(p);
+ }
+ return filesByPartition;
+ }
+
+ /**
+ * Given a map of partition numbers to files, return a list of files filtered by the supplied
+ * beginning and ending timestamps. Includes a left-trailing file.
+ *
+ * @param filesByPartition list of files mapped to partitions. Incoming files do not need to be
+ * sorted as this method will perform a lexicographical sort in normal ascending order.
+ * @return filtered list of files, unsorted
+ */
+ public static List<String> filterByTimestampLT(long beginTs, long endTs,
+ Map<Integer, List<Path>> filesByPartition) {
+ List<String> filteredFiles = new ArrayList<>();
+ for (Integer key : filesByPartition.keySet()) {
+ List<Path> paths = filesByPartition.get(key);
+ filteredFiles.addAll(filterByTimestampLT(beginTs, endTs, paths));
+ }
+ return filteredFiles;
+ }
+
+ /**
+ * Return a list of files filtered by the supplied beginning and ending timestamps. Includes a
+ * left-trailing file.
+ *
+ * @param paths list of files. Incoming files do not need to be sorted as this method will perform
+ * a lexicographical sort in normal ascending order.
+ * @return filtered list of files
+ */
+ public static List<String> filterByTimestampLT(long beginTs, long endTs, List<Path> paths) {
+ List<String> filteredFiles = new ArrayList<>();
+
+ //noinspection unchecked - hadoop fs uses non-generic Comparable interface
+ Collections.sort(paths);
+ Iterator<Path> filesIt = paths.iterator();
+ Path leftTrailing = filesIt.hasNext() ? filesIt.next() : null;
+ if (leftTrailing == null) {
+ return filteredFiles;
+ }
+ boolean first = true;
+ Long fileTS = PcapFilenameHelper.getTimestamp(leftTrailing.getName());
+ if (fileTS != null
+ && greaterThanOrEqualTo(fileTS, beginTs) && lessThanOrEqualTo(fileTS, endTs)) {
+ filteredFiles.add(leftTrailing.toString());
+ first = false;
+ }
+
+ if (first && !filesIt.hasNext()) {
+ filteredFiles.add(leftTrailing.toString());
+ return filteredFiles;
+ }
+
+ while (filesIt.hasNext()) {
+ Path p = filesIt.next();
+ fileTS = PcapFilenameHelper.getTimestamp(p.getName());
+ if (fileTS != null
+ && greaterThanOrEqualTo(fileTS, beginTs) && lessThanOrEqualTo(fileTS, endTs)) {
+ if (first) {
+ filteredFiles.add(leftTrailing.toString());
+ first = false;
+ }
+ filteredFiles.add(p.toString());
+ } else {
+ leftTrailing = p;
+ }
+ }
+
+ return filteredFiles;
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/PcapFilenameHelperTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/PcapFilenameHelperTest.java b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/PcapFilenameHelperTest.java
new file mode 100644
index 0000000..03778d0
--- /dev/null
+++ b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/PcapFilenameHelperTest.java
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import org.junit.Test;
+
+public class PcapFilenameHelperTest {
+
+ @Test
+ public void extracts_info_from_filename() {
+ {
+ String pcapFilename = "pcap_pcap128_1494962815457986000_18_pcap-63-1495027314";
+ assertThat(PcapFilenameHelper.getKafkaTopic(pcapFilename), equalTo("pcap128"));
+ assertThat(
+ Long.compareUnsigned(PcapFilenameHelper.getTimestamp(pcapFilename), 1494962815457986000L),
+ equalTo(0));
+ assertThat(PcapFilenameHelper.getKafkaPartition(pcapFilename), equalTo(18));
+ assertThat(PcapFilenameHelper.getUUID(pcapFilename), equalTo("pcap-63-1495027314"));
+ }
+ {
+ String pcapFilename = "pcap_pcap-128_1494962815457986000_18_pcap-63-1495027314";
+ assertThat(PcapFilenameHelper.getKafkaTopic(pcapFilename), equalTo("pcap-128"));
+ assertThat(
+ Long.compareUnsigned(PcapFilenameHelper.getTimestamp(pcapFilename), 1494962815457986000L),
+ equalTo(0));
+ }
+ {
+ String pcapFilename = "pcap_pcap_128_1494962815457986000_18_pcap-63-1495027314";
+ assertThat(PcapFilenameHelper.getKafkaTopic(pcapFilename), equalTo("pcap_128"));
+ assertThat(
+ Long.compareUnsigned(PcapFilenameHelper.getTimestamp(pcapFilename), 1494962815457986000L),
+ equalTo(0));
+ }
+ {
+ String pcapFilename = "pcap_pcap___128___1494962815457986000_18_pcap-63-1495027314";
+ assertThat(PcapFilenameHelper.getKafkaTopic(pcapFilename), equalTo("pcap___128__"));
+ assertThat(
+ Long.compareUnsigned(PcapFilenameHelper.getTimestamp(pcapFilename), 1494962815457986000L),
+ equalTo(0));
+ }
+ {
+ String pcapFilename = "pcap___pcap___128___1494962815457986000_18_pcap-63-1495027314";
+ assertThat(PcapFilenameHelper.getKafkaTopic(pcapFilename), equalTo("__pcap___128__"));
+ assertThat(
+ Long.compareUnsigned(PcapFilenameHelper.getTimestamp(pcapFilename), 1494962815457986000L),
+ equalTo(0));
+ }
+ }
+
+ @Test
+ public void extracts_null_info_from_bad_filename_parts() {
+ String pcapFilename = "pcap_pcap128_AAA4962815457986000_BB_pcap-63-1495027314";
+ assertThat(PcapFilenameHelper.getTimestamp(pcapFilename), equalTo(null));
+ assertThat(PcapFilenameHelper.getKafkaPartition(pcapFilename), equalTo(null));
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/e2197316/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
new file mode 100644
index 0000000..cc05a9a
--- /dev/null
+++ b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.pcap.mr;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.Iterables;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.hadoop.fs.Path;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.pcap.utils.FileFilterUtil;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class FileFilterUtilTest {
+
+ private List<Path> filesIn;
+
+ @Before
+ public void setup() {
+ filesIn = new ArrayList<>();
+ filesIn.add(new Path("/apath/pcap_pcap5_1495135372055519000_2_pcap-9-1495134910"));
+ filesIn.add(new Path("/apath/pcap_pcap5_1495135372168719000_1_pcap-9-1495134910"));
+ filesIn.add(new Path("/apath/pcap_pcap5_1495135377055375000_0_pcap-9-1495134910"));
+ filesIn.add(new Path("/apath/pcap_pcap5_1495135512102506000_4_pcap-9-1495134910"));
+ filesIn.add(new Path("/apath/pcap_pcap5_1495135512123943000_3_pcap-9-1495134910"));
+ }
+
+ @Test
+ public void returns_files_by_partition() {
+ Map<Integer, List<Path>> filesByPartition = FileFilterUtil.getFilesByPartition(filesIn);
+ Map<Integer, List<Path>> expectedFilesPartitioned = new HashMap() {{
+ put(0, toList("/apath/pcap_pcap5_1495135377055375000_0_pcap-9-1495134910"));
+ put(1, toList("/apath/pcap_pcap5_1495135372168719000_1_pcap-9-1495134910"));
+ put(2, toList("/apath/pcap_pcap5_1495135372055519000_2_pcap-9-1495134910"));
+ put(3, toList("/apath/pcap_pcap5_1495135512123943000_3_pcap-9-1495134910"));
+ put(4, toList("/apath/pcap_pcap5_1495135512102506000_4_pcap-9-1495134910"));
+ }};
+ assertThat(filesByPartition, equalTo(expectedFilesPartitioned));
+ }
+
+ private List<Path> toList(String... items) {
+ return Arrays.asList(items).stream().map(i -> new Path(i)).collect(Collectors.toList());
+ }
+
+ @Test
+ public void returns_left_trailing_filtered_list() {
+ Map<Integer, List<Path>> filesByPartition = new HashMap() {{
+ put(0, toList("/apath/pcap_pcap5_1495135377055375000_0_pcap-9-1495134910"));
+ put(1, toList("/apath/pcap_pcap5_1495135372168719000_1_pcap-9-1495134910"));
+ put(2, toList("/apath/pcap_pcap5_1495135372055519000_2_pcap-9-1495134910"));
+ put(3, toList("/apath/pcap_pcap5_1495135512123943000_3_pcap-9-1495134910"));
+ put(4, toList("/apath/pcap_pcap5_1495135512102506000_4_pcap-9-1495134910"));
+ }};
+ List<String> lt = FileFilterUtil
+ .filterByTimestampLT(1495135377055375000L, 1495135512124943000L, filesByPartition);
+ List<String> expectedFiles = Arrays.asList(
+ "/apath/pcap_pcap5_1495135377055375000_0_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135372168719000_1_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135372055519000_2_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135512123943000_3_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135512102506000_4_pcap-9-1495134910");
+ assertThat(lt, equalTo(expectedFiles));
+ }
+
+ @Test
+ public void returns_left_trailing_filtered_list_from_paths() {
+ Iterable<String> paths = FileFilterUtil
+ .getPathsInTimeRange(1495135377055375000L, 1495135512124943000L, filesIn);
+ List<String> expectedFiles = Arrays.asList(
+ "/apath/pcap_pcap5_1495135377055375000_0_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135372168719000_1_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135372055519000_2_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135512123943000_3_pcap-9-1495134910",
+ "/apath/pcap_pcap5_1495135512102506000_4_pcap-9-1495134910");
+ assertThat(paths, equalTo(expectedFiles));
+ }
+
+ @Test
+ public void test_getPaths_NoFiles() throws Exception {
+ final List<Path> inputFiles = new ArrayList<Path>();
+ Iterable<String> paths = FileFilterUtil.getPathsInTimeRange(0, 1000, inputFiles);
+ Assert.assertTrue(Iterables.isEmpty(paths));
+ }
+
+ @Test
+ public void test_getPaths_leftEdge() throws Exception {
+ final List<Path> inputFiles = new ArrayList<Path>() {{
+ add(new Path("/apps/metron/pcap/pcap_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap_pcap_1561589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ }};
+ Iterable<String> paths = FileFilterUtil.getPathsInTimeRange(0, TimestampConverters.MILLISECONDS.toNanoseconds(System.currentTimeMillis()), inputFiles);
+ Assert.assertEquals(1, Iterables.size(paths));
+ }
+
+ @Test
+ public void test_getPaths_rightEdge() throws Exception {
+ {
+ final List<Path> inputFiles = new ArrayList<Path>() {{
+ add(new Path("/apps/metron/pcap/pcap0_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap1_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ }};
+ Iterable<String> paths = FileFilterUtil.getPathsInTimeRange(1461589333993573000L - 1L, 1461589333993573000L + 1L, inputFiles);
+ Assert.assertEquals(2, Iterables.size(paths));
+ }
+ {
+ final List<Path> inputFiles = new ArrayList<Path>() {{
+ add(new Path("/apps/metron/pcap/pcap0_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap1_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap1_pcap_1461589334993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ }};
+ Iterable<String> paths = FileFilterUtil.getPathsInTimeRange(1461589334993573000L - 1L, 1461589334993573000L + 1L, inputFiles);
+ Assert.assertEquals(2, Iterables.size(paths));
+ }
+ }
+
+ @Test
+ public void test_getPaths_bothEdges() throws Exception {
+ final List<Path> inputFiles = new ArrayList<Path>() {{
+ add(new Path("/apps/metron/pcap/pcap_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap1_pcap_1461589334993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ }};
+ Iterable<String> paths = FileFilterUtil.getPathsInTimeRange(0, TimestampConverters.MILLISECONDS.toNanoseconds(System.currentTimeMillis()), inputFiles);
+ Assert.assertEquals(3, Iterables.size(paths));
+ }
+}
[36/44] metron git commit: METRON-959 Trivial documentation
formatting bugs (JonZeolla) closes apache/metron#591
Posted by ma...@apache.org.
METRON-959 Trivial documentation formatting bugs (JonZeolla) closes apache/metron#591
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/08f95dc8
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/08f95dc8
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/08f95dc8
Branch: refs/heads/Metron_0.4.0
Commit: 08f95dc8f5588dc9d50e204410ca3d09f84f3c8b
Parents: 7ef929a
Author: JonZeolla <ze...@gmail.com>
Authored: Thu May 25 18:15:41 2017 -0400
Committer: jonzeolla <jo...@apache.org>
Committed: Thu May 25 18:15:41 2017 -0400
----------------------------------------------------------------------
metron-analytics/metron-maas-service/README.md | 6 +++---
metron-platform/metron-indexing/README.md | 4 ++--
metron-platform/metron-management/README.md | 2 +-
metron-platform/metron-parsers/README.md | 20 ++++++++++----------
site-book/README.md | 8 ++++----
5 files changed, 20 insertions(+), 20 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-analytics/metron-maas-service/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-maas-service/README.md b/metron-analytics/metron-maas-service/README.md
index 6b1663b..65f1a01 100644
--- a/metron-analytics/metron-maas-service/README.md
+++ b/metron-analytics/metron-maas-service/README.md
@@ -1,6 +1,6 @@
# Model Management Infrastructure
-##Introduction
+## Introduction
One of the main features envisioned and requested is the ability to augment the threat intelligence and enrichment processes with insights derived from machine learning or statistical models. The challenges with this sort of infrastructure are
* Applying the model may be sufficiently computationally/resource intensive that we need to support scaling via load balancing, which will require service discovery and management.
@@ -12,7 +12,7 @@ To support a high throughput environment that is manageable, it is evident that
* Deployment should happen using Yarn to manage resources
* Clients should have new model endpoints pushed to them
-##Architecture
+## Architecture
@@ -95,7 +95,7 @@ usage: ModelSubmission
-zr,--zk_root <arg> Zookeeper Root
```
-##Stellar Integration
+## Stellar Integration
Two Stellar functions have been added to provide the ability to call out to models deployed via Model as a Service.
One aimed at recovering a load balanced endpoint of a deployed model given the name and, optionally, the version.
http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index a536d87..0cf3a66 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -1,4 +1,4 @@
-#Indexing
+# Indexing
## Introduction
@@ -26,7 +26,7 @@ and sent to
By default, errors during indexing are sent back into the `indexing` kafka queue so that they can be indexed and archived.
-##Sensor Indexing Configuration
+## Sensor Indexing Configuration
The sensor specific configuration is intended to configure the
indexing used for a given sensor type (e.g. `snort`).
http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-platform/metron-management/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-management/README.md b/metron-platform/metron-management/README.md
index 0cfcac5..b920460 100644
--- a/metron-platform/metron-management/README.md
+++ b/metron-platform/metron-management/README.md
@@ -1,4 +1,4 @@
-#Stellar REPL Management Utilities
+# Stellar REPL Management Utilities
In order to augment the functionality of the Stellar REPL, a few
management functions surrounding the management of the configurations
http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/metron-platform/metron-parsers/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md
index cc45834..97e8e5f 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -1,4 +1,4 @@
-#Parsers
+# Parsers
## Introduction
@@ -32,7 +32,7 @@ topology in kafka. Errors are collected with the context of the error
`error` queue. Invalid messages as determined by global validation
functions are also treated as errors and sent to an `error` queue.
-##Message Format
+## Message Format
All Metron messages follow a specific format in order to ingest a message. If a message does not conform to this format it will be dropped and put onto an error queue for further examination. The message must be of a JSON format and must have a JSON tag message like so:
@@ -70,11 +70,11 @@ So putting it all together a typical Metron message with all 5-tuple fields pres
}
```
-##Global Configuration
+## Global Configuration
See the "[Global Configuration](../metron-common)" section.
-##Parser Configuration
+## Parser Configuration
The configuration for the various parser topologies is defined by JSON
documents stored in zookeeper.
@@ -103,7 +103,7 @@ transformation which can be done to a message. This transformation can
* Add new fields given the values of existing fields of a message
* Remove existing fields of a message
-###`fieldTransformation` configuration
+### `fieldTransformation` configuration
The format of a `fieldTransformation` is as follows:
* `input` : An array of fields or a single field representing the input. This is optional; if unspecified, then the whole message is passed as input.
@@ -201,7 +201,7 @@ HH:mm:ss', MAP_GET(dc, dc2tz, 'UTC') )"
Note that the `dc2tz` map is in the parser config, so it is accessible
in the functions.
-###An Example Configuration for a Sensor
+### An Example Configuration for a Sensor
Consider the following example configuration for the `yaf` sensor:
```
@@ -225,12 +225,12 @@ Consider the following example configuration for the `yaf` sensor:
}
```
-##Parser Adapters
+## Parser Adapters
Parser adapters are loaded dynamically in each Metron topology. They
are defined in the Parser Config (defined above) JSON file in Zookeeper.
-###Java Parser Adapters
+### Java Parser Adapters
Java parser adapters are indended for higher-velocity topologies and are not easily changed or extended. As the adoption of Metron continues we plan on extending our library of Java adapters to process more log formats. As of this moment the Java adapters included with Metron are:
* org.apache.metron.parsers.ise.BasicIseParser : Parse ISE messages
@@ -238,7 +238,7 @@ Java parser adapters are indended for higher-velocity topologies and are not eas
* org.apache.metron.parsers.sourcefire.BasicSourcefireParser : Parse Sourcefire messages
* org.apache.metron.parsers.lancope.BasicLancopeParser : Parse Lancope messages
-###Grok Parser Adapters
+### Grok Parser Adapters
Grok parser adapters are designed primarly for someone who is not a Java coder for quickly standing up a parser adapter for lower velocity topologies. Grok relies on Regex for message parsing, which is much slower than purpose-built Java parsers, but is more extensible. Grok parsers are defined via a config file and the topplogy does not need to be recombiled in order to make changes to them. An example of a Grok perser is:
* org.apache.metron.parsers.GrokParser
@@ -247,7 +247,7 @@ For more information on the Grok project please refer to the following link:
https://github.com/thekrakken/java-grok
-#Starting the Parser Topology
+# Starting the Parser Topology
Starting a particular parser topology on a running Metron deployment is
as easy as running the `start_parser_topology.sh` script located in
http://git-wip-us.apache.org/repos/asf/metron/blob/08f95dc8/site-book/README.md
----------------------------------------------------------------------
diff --git a/site-book/README.md b/site-book/README.md
index 6c6b7bd..bf579de 100644
--- a/site-book/README.md
+++ b/site-book/README.md
@@ -32,22 +32,22 @@ open target/site/index.html
```
-##Key Components:
+## Key Components:
-###bin/generate-md.sh
+### bin/generate-md.sh
- Copies all .md files from the code directory tree into the site tree
- Performs some transformations on them
- Generates the nav tree structure and labels
- Happens during the site:pre-site phase of Maven.
-###bin/fix-md-dialect.py
+### bin/fix-md-dialect.py
- Called by 'generate-md.sh'
- Does transforms within the text of each file
- Converts the Github-MD dialect of markdown into the doxia-markdown dialect
-###pom.xml and src/site/site.xml
+### pom.xml and src/site/site.xml
- [Doxia](https://maven.apache.org/doxia/) boilerplate, tweaked for our specific needs
[42/44] metron git commit: METRON-974 Document How to Run Stellar
REPL From Source Code (nickwallen via ottobackwards) closes apache/metron#598
Posted by ma...@apache.org.
METRON-974 Document How to Run Stellar REPL From Source Code (nickwallen via ottobackwards) closes apache/metron#598
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/74bc236d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/74bc236d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/74bc236d
Branch: refs/heads/Metron_0.4.0
Commit: 74bc236d56a9d6c96105bb2bd2b5f719c5bb3bff
Parents: 9dc8c3a
Author: nickwallen <ni...@nickallen.org>
Authored: Wed May 31 22:05:06 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Wed May 31 22:05:06 2017 -0400
----------------------------------------------------------------------
metron-platform/metron-common/README.md | 36 +++++++++++++++++++++++++++-
1 file changed, 35 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/74bc236d/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index 10800d5..9e1de7a 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -784,7 +784,9 @@ mvn exec:java -Dexec.mainClass="org.apache.metron.common.stellar.benchmark.Stell
```
## Stellar Shell
-A REPL (Read Eval Print Loop) for the Stellar language that helps in debugging, troubleshooting and learning Stellar. The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.
+The Stellar Shell is a REPL (Read Eval Print Loop) for the Stellar language that helps troubleshooting, learning Stellar or even interacting with a live Metron cluster.
+
+The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems.
The shell supports customization via `~/.inputrc` as it is
backed by a proper readline implementation.
@@ -800,6 +802,7 @@ Note: Stellar classpath configuration from the global config is honored here if
### Getting Started
+To run the Stellar Shell from within a deployed Metron cluster, run the following command on the host where Metron is installed.
```
$ $METRON_HOME/bin/stellar
@@ -927,6 +930,37 @@ IS_EMAIL
[Stellar]>>>
```
+### Advanced Usage
+
+To run the Stellar Shell directly from the Metron source code, run a command like the following. Ensure that Metron has already been built and installed with `mvn clean install -DskipTests`.
+```
+$ mvn exec:java \
+ -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
+ -pl metron-platform/metron-enrichment
+...
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+[Stellar]>>> %functions
+ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GEO_GET, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_
SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYSTEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
+```
+
+Changing the project passed to the `-pl` argument will define which dependencies are included and ultimately which Stellar functions are available within the shell environment.
+
+This can be useful for troubleshooting function resolution problems. The previous example defines which functions are available during Enrichment. For example, to determine which functions are available within the Profiler run the following.
+
+```
+ $ mvn exec:java \
+ -Dexec.mainClass="org.apache.metron.common.stellar.shell.StellarShell" \
+ -pl metron-analytics/metron-profiler
+...
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+[Stellar]>>> Functions loaded, you may refer to functions now...
+%functions
+ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, CHOMP, CHOP, COUNT_MATCHES, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, FILL_LEFT, FILL_RIGHT, FILTER, FORMAT, GET, GET_FIRST, GET_LAST, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, JOIN, LENGTH, LIST_ADD, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MONTH, OUTLIER_MAD_ADD, OUTLIER_MAD_SCORE, OUTLIER_MAD_STATE_MERGE, PREPEND_IF_MISSING, PROFILE_FIXED, PROFILE_GET, PROFILE_WINDOW, PROTOCOL_TO_NAME, REDUCE, REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_BIN, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, STRING_ENTROPY, SYS
TEM_ENV_GET, SYSTEM_PROPERTY_GET, TO_DOUBLE, TO_EPOCH_TIMESTAMP, TO_FLOAT, TO_INTEGER, TO_LONG, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR
+```
+
# Global Configuration
The format of the global enrichment is a JSON String to Object map. This is intended for
[35/44] metron git commit: METRON-811 Enforce Maven Version in Top
Level POM (zezutom via ottobackwards) closes apache/metron#558
Posted by ma...@apache.org.
METRON-811 Enforce Maven Version in Top Level POM (zezutom via ottobackwards) closes apache/metron#558
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/7ef929ae
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/7ef929ae
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/7ef929ae
Branch: refs/heads/Metron_0.4.0
Commit: 7ef929ae4b998468beb3ef08fe57e37ff04fad4e
Parents: e219731
Author: zezutom <ze...@gmail.com>
Authored: Wed May 24 06:11:16 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Wed May 24 06:11:16 2017 -0400
----------------------------------------------------------------------
pom.xml | 28 ++++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/7ef929ae/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 4bbd8ae..16b2499 100644
--- a/pom.xml
+++ b/pom.xml
@@ -107,6 +107,7 @@
<global_errorprone_core_version>2.0.14</global_errorprone_core_version>
<global_jar_version>3.0.2</global_jar_version>
<global_surefire_version>2.18</global_surefire_version>
+ <global_maven_version>[3.3.1,)</global_maven_version>
</properties>
<profiles>
@@ -275,7 +276,7 @@
</execution>
</executions>
<configuration>
- <excludes>
+ <excludes>
<exclude>dependencies_with_url.csv</exclude>
<!-- In travis we need to pull down maven 3.3.9, so we should exclude it here as it is not our code. -->
<exclude>apache-maven-3.3.9/**</exclude>
@@ -309,7 +310,7 @@
<exclude>**/hbase/data/**</exclude>
<exclude>**/kafkazk/data/**</exclude>
<exclude>**/wait-for-it.sh</exclude>
- <exclude>**/*.out</exclude>
+ <exclude>**/*.out</exclude>
<!-- Directory containing dependencies downloaded by NPM -->
<exclude>node_modules/**</exclude>
<!-- Nodejs installed locally by the frontend-maven-plugin -->
@@ -326,6 +327,29 @@
</excludes>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-enforcer-plugin</artifactId>
+ <version>1.4.1</version>
+ <executions>
+ <execution>
+ <id>enforce-versions</id>
+ <goals>
+ <goal>enforce</goal>
+ </goals>
+ <configuration>
+ <rules>
+ <requireMavenVersion>
+ <version>${global_maven_version}</version>
+ </requireMavenVersion>
+ <requireJavaVersion>
+ <version>${global_java_version}</version>
+ </requireJavaVersion>
+ </rules>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
</plugins>
</build>
[15/44] metron git commit: METRON-945 Resolve merge conflict in
metron.spec (dlyle via justinleet) closes apache/incubator-metron#576
Posted by ma...@apache.org.
METRON-945 Resolve merge conflict in metron.spec (dlyle via justinleet) closes apache/incubator-metron#576
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/8532e284
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/8532e284
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/8532e284
Branch: refs/heads/Metron_0.4.0
Commit: 8532e2848c4767405f4bfe6501c251b1aab54f01
Parents: 0d5fdf4
Author: dlyle <dl...@gmail.com>
Authored: Wed May 10 08:38:37 2017 -0400
Committer: leet <le...@apache.org>
Committed: Wed May 10 08:38:37 2017 -0400
----------------------------------------------------------------------
.../packaging/docker/rpm-docker/SPECS/metron.spec | 10 ++--------
1 file changed, 2 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/8532e284/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index da8f192..e851c7f 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -269,11 +269,8 @@ This package installs the Metron Indexing files
%{metron_home}/config/zookeeper/indexing/asa.json
%{metron_home}/config/zookeeper/indexing/error.json
%{metron_home}/config/zeppelin/metron/metron-yaf-telemetry.json
-<<<<<<< HEAD
%{metron_home}/config/zeppelin/metron/metron-connection-report.json
-=======
%{metron_home}/config/zeppelin/metron/metron-ip-report.json
->>>>>>> 1cace9ff29f31301d74fa6a7b2630d471452e985
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -403,15 +400,12 @@ This package installs the Metron Management UI %{metron_home}
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%changelog
-<<<<<<< HEAD
* Thu May 4 2017 Ryan Merriman <me...@gmail.com> - 0.4.0
- Added REST
-* Fri Apr 28 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
-- Add Zeppelin Connection Report Dashboard
-=======
* Tue May 2 2017 David Lyle <dl...@gmail.com> - 0.4.0
- Add Metron IP Report
->>>>>>> 1cace9ff29f31301d74fa6a7b2630d471452e985
+* Fri Apr 28 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
+- Add Zeppelin Connection Report Dashboard
* Thu Jan 19 2017 Justin Leet <ju...@gmail.com> - 0.3.1
- Replace GeoIP files with new implementation
* Thu Nov 03 2016 David Lyle <dl...@gmail.com> - 0.2.1
[12/44] metron git commit: METRON-905 Fix square-bracket behavior and
default network interface bindings for ES (mattf-horton) closes
apache/incubator-metron#564
Posted by ma...@apache.org.
METRON-905 Fix square-bracket behavior and default network interface bindings for ES (mattf-horton) closes apache/incubator-metron#564
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/21b79dc8
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/21b79dc8
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/21b79dc8
Branch: refs/heads/Metron_0.4.0
Commit: 21b79dc8eb0ffd7f0a3ea87744cef3d792456af7
Parents: 876a5fc
Author: mattf-horton <mf...@hortonworks.com>
Authored: Tue May 9 13:12:16 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Tue May 9 13:12:16 2017 -0700
----------------------------------------------------------------------
.../2.3.3/configuration/elastic-site.xml | 14 +++++++++++---
.../ELASTICSEARCH/2.3.3/package/scripts/params.py | 1 +
.../package/templates/elasticsearch.master.yaml.j2 | 5 +++--
.../package/templates/elasticsearch.slave.yaml.j2 | 5 +++--
.../roles/ambari_config/vars/single_node_vm.yml | 4 ++--
.../roles/ambari_config/vars/small_cluster.yml | 4 ++--
6 files changed, 22 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/21b79dc8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
index fd29e63..33e01bf 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
@@ -39,7 +39,7 @@
<!--Ideally this gets populated by the list of master eligible nodes (as an acceptable default). Unsure how to do this.-->
<!--Also need to document whether should list masters only, or all ES nodes. I think this one is all nodes, but previous inline comment said Masters.-->
<value></value>
- <description>Unicast discovery list of hosts to act as gossip routers, in comma separated format: eshost1,eshost2</description>
+ <description>Unicast discovery list of hosts to act as gossip routers, comma-separated list with square brackets: [ eshost1, eshost2 ]</description>
</property>
<property>
<name>index_number_of_shards</name>
@@ -189,7 +189,15 @@
</property>
<property>
<name>network_host</name>
- <value>"_lo:ipv4_","_eth0:ipv4_"</value>
- <description>Network interface(s) ES will bind to within each node. Confirm names via ifconfig. Should have entries for primary external and loopback interfaces, with :ipv4 annotation and quote marks around each entry: "_lo:ipv4_","_eth0:ipv4_"</description>
+ <value>[ _local_ ]</value>
+ <description>Network interface(s) ES will bind to within each node. Use [ _local_ ] for single-node install; change to [ _local_, _site_ ] (including square brackets) for typical cluster install. See https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html for ES documentation.</description>
+ </property>
+ <property>
+ <name>network_publish_host</name>
+ <value></value>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
+ <description>Network address ES will publish for client and peer use. Empty value causes it to pick from the values in network_host, which works in most simple environments. MUST set explicitly for MULTI-HOMED SYSTEMS. See https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html for ES documentation.</description>
</property>
</configuration>
http://git-wip-us.apache.org/repos/asf/metron/blob/21b79dc8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
index 00929a7..e7b8d85 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/params.py
@@ -77,3 +77,4 @@ discovery_zen_fd_ping_timeout = config['configurations']['elastic-site']['discov
discovery_zen_fd_ping_retries = config['configurations']['elastic-site']['discovery_zen_fd_ping_retries']
network_host = config['configurations']['elastic-site']['network_host']
+network_publish_host = config['configurations']['elastic-site']['network_publish_host']
http://git-wip-us.apache.org/repos/asf/metron/blob/21b79dc8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.master.yaml.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.master.yaml.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.master.yaml.j2
index ca3630a..f0d2a8f 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.master.yaml.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.master.yaml.j2
@@ -30,7 +30,7 @@ discovery:
multicast:
enabled: {{ discovery_zen_ping_multicast_enabled }}
unicast:
- hosts: "{{zen_discovery_ping_unicast_hosts}}"
+ hosts: {{zen_discovery_ping_unicast_hosts}}
node:
data: {{ masters_also_are_datanodes }}
@@ -81,4 +81,5 @@ discovery.zen.fd.ping_interval: {{discovery_zen_fd_ping_interval}}
discovery.zen.fd.ping_timeout: {{discovery_zen_fd_ping_timeout}}
discovery.zen.fd.ping_retries: {{discovery_zen_fd_ping_retries}}
-network.host: {{network_host}}
\ No newline at end of file
+network.host: {{network_host}}
+network.publish_host: {{network_publish_host}}
http://git-wip-us.apache.org/repos/asf/metron/blob/21b79dc8/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.slave.yaml.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.slave.yaml.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.slave.yaml.j2
index 58ce3d3..7d2d0cf 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.slave.yaml.j2
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/templates/elasticsearch.slave.yaml.j2
@@ -30,7 +30,7 @@ discovery:
multicast:
enabled: {{discovery_zen_ping_multicast_enabled}}
unicast:
- hosts: "{{zen_discovery_ping_unicast_hosts}}"
+ hosts: {{zen_discovery_ping_unicast_hosts}}
node:
data: true
@@ -81,4 +81,5 @@ discovery.zen.fd.ping_interval: {{discovery_zen_fd_ping_interval}}
discovery.zen.fd.ping_timeout: {{discovery_zen_fd_ping_timeout}}
discovery.zen.fd.ping_retries: {{discovery_zen_fd_ping_retries}}
-network.host: {{network_host}}
\ No newline at end of file
+network.host: {{network_host}}
+network.publish_host: {{network_publish_host}}
http://git-wip-us.apache.org/repos/asf/metron/blob/21b79dc8/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
index fe25eed..b24c153 100644
--- a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
+++ b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
@@ -95,9 +95,9 @@ configurations:
- elastic-site:
index_number_of_shards: 1
index_number_of_replicas: 0
- zen_discovery_ping_unicast_hosts: "{{ groups.search | join(',') }}"
+ zen_discovery_ping_unicast_hosts: "[ {{ groups.search | join(', ') }} ]"
gateway_recover_after_data_nodes: 1
- network_host: "_lo_,_eth0_,_eth1_"
+ network_host: "[ _local_ ]"
masters_also_are_datanodes: "1"
required_configurations:
http://git-wip-us.apache.org/repos/asf/metron/blob/21b79dc8/metron-deployment/roles/ambari_config/vars/small_cluster.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/small_cluster.yml b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
index 5f261a0..70c31ea 100644
--- a/metron-deployment/roles/ambari_config/vars/small_cluster.yml
+++ b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
@@ -89,9 +89,9 @@ configurations:
- elastic-site:
index_number_of_shards: 2
index_number_of_replicas: 1
- zen_discovery_ping_unicast_hosts: "{{ groups.web[0] }},{{ groups.search | join(',') }}"
+ zen_discovery_ping_unicast_hosts: "[ {{ groups.web[0] }}, {{ groups.search | join(', ') }} ]"
gateway_recover_after_data_nodes: 1
- network_host: _lo_,_{{ elasticsearch_network_interface }}_
+ network_host: "[ _local_, _{{ elasticsearch_network_interface }}_ ]"
required_configurations:
- metron-env:
[33/44] metron git commit: METRON-966: Pcap topology does not commit
offsets closes apache/incubator-metron#597
Posted by ma...@apache.org.
METRON-966: Pcap topology does not commit offsets closes apache/incubator-metron#597
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/356881ad
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/356881ad
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/356881ad
Branch: refs/heads/Metron_0.4.0
Commit: 356881ad22edc7c08d8a8f3812192333210f3c8e
Parents: 64473d4
Author: cstella <ce...@gmail.com>
Authored: Sat May 20 09:59:02 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Sat May 20 09:59:02 2017 -0400
----------------------------------------------------------------------
.../metron/spout/pcap/KafkaToHDFSSpout.java | 50 ++++++++++++++++++++
pom.xml | 18 +------
2 files changed, 51 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/356881ad/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/KafkaToHDFSSpout.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/KafkaToHDFSSpout.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/KafkaToHDFSSpout.java
index ddfd14a..5b3d6f6 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/KafkaToHDFSSpout.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/KafkaToHDFSSpout.java
@@ -22,9 +22,20 @@ import org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder;
import org.apache.storm.kafka.Callback;
import org.apache.storm.kafka.CallbackKafkaSpout;
+import java.util.ArrayList;
+import java.util.List;
+
public class KafkaToHDFSSpout extends CallbackKafkaSpout<byte[], byte[]> {
static final long serialVersionUID = 0xDEADBEEFL;
HDFSWriterConfig config = null;
+
+ private static ThreadLocal<List<Object>> messagesToBeAcked = new ThreadLocal<List<Object>>() {
+ @Override
+ protected List<Object> initialValue() {
+ return new ArrayList<>();
+ }
+ };
+
public KafkaToHDFSSpout( SimpleStormKafkaBuilder<byte[], byte[]> spoutConfig
, HDFSWriterConfig config
)
@@ -40,5 +51,44 @@ public class KafkaToHDFSSpout extends CallbackKafkaSpout<byte[], byte[]> {
return new HDFSWriterCallback().withConfig(config);
}
+ /**
+ * Clear all the messages that are queued to be acked.
+ */
+ private void clearMessagesToBeAcked() {
+ for (Object messageId : messagesToBeAcked.get()) {
+ super.ack(messageId);
+ }
+ messagesToBeAcked.get().clear();
+ }
+ @Override
+ public void nextTuple() {
+ /*
+ This bears some explanation; nextTuple for a spout-only topology sans ackers, will ack as part of the emit method.
+ The unfortunate part about this is that this will prevent the internal bookeeping of the KafkaSpout to keep add the
+ message ID to the offsets to commit. This is because it thinks it is not emitted by the time it gets to ack (because
+ ack is called *within* emit). The result is that no offsets are acked.
+
+ What we have here is a correction. The ack method will add the message ID to a queue to be acked and then at the end
+ of nextTuple, we will clear the cache and ack. The net result is that the contract is adhered to for spout-only topologies,
+ ack happens in nextTuple().
+ */
+ super.nextTuple();
+ clearMessagesToBeAcked();
+ }
+
+ @Override
+ public void ack(Object messageId) {
+ messagesToBeAcked.get().add(messageId);
+ }
+
+ @Override
+ public void close() {
+ try {
+ clearMessagesToBeAcked();
+ }
+ finally {
+ super.close();
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/356881ad/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 9f62249..4bbd8ae 100644
--- a/pom.xml
+++ b/pom.xml
@@ -39,22 +39,6 @@
<repositories>
<repository>
- <releases>
- <enabled>true</enabled>
- <updatePolicy>always</updatePolicy>
- <checksumPolicy>warn</checksumPolicy>
- </releases>
- <snapshots>
- <enabled>true</enabled>
- <updatePolicy>never</updatePolicy>
- <checksumPolicy>warn</checksumPolicy>
- </snapshots>
- <id>HDPPrivateReleases</id>
- <name>HDP Private Releases</name>
- <url>http://nexus-private.hortonworks.com/nexus/content/groups/public</url>
- <layout>default</layout>
- </repository>
- <repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
@@ -131,7 +115,7 @@
<properties>
<hdp_version>2.5.0.0</hdp_version>
<build_number>1245</build_number>
- <global_storm_kafka_version>1.1.0.2.6.1.0-SNAPSHOT</global_storm_kafka_version>
+ <global_storm_kafka_version>1.1.0</global_storm_kafka_version>
<global_storm_version>${base_storm_version}.${hdp_version}-${build_number}</global_storm_version>
<global_kafka_version>${base_kafka_version}.${hdp_version}-${build_number}</global_kafka_version>
</properties>
[29/44] metron git commit: METRON-963 Update document for Kerberos
manual setup (MohanDV via merrimanr) closes apache/metron#594
Posted by ma...@apache.org.
METRON-963 Update document for Kerberos manual setup (MohanDV via merrimanr) closes apache/metron#594
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/5e0cce22
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/5e0cce22
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/5e0cce22
Branch: refs/heads/Metron_0.4.0
Commit: 5e0cce220dca80c996d173b16b9721fccb373821
Parents: 47e2b73
Author: MohanDV <mo...@gmail.com>
Authored: Fri May 19 10:21:09 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Fri May 19 10:21:09 2017 -0500
----------------------------------------------------------------------
metron-deployment/Kerberos-ambari-setup.md | 2 +-
metron-deployment/Kerberos-manual-setup.md | 44 +++++++++++++++++--------
2 files changed, 31 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/5e0cce22/metron-deployment/Kerberos-ambari-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-ambari-setup.md b/metron-deployment/Kerberos-ambari-setup.md
index 149e8b2..39e8dee 100644
--- a/metron-deployment/Kerberos-ambari-setup.md
+++ b/metron-deployment/Kerberos-ambari-setup.md
@@ -3,7 +3,7 @@
General Kerberization notes can be found in the metron-deployment [README.md](../README.md)
## Setup a KDC
-See [Setup a KDC](Kerberos-manual-setup.md#setup-a-kdc)
+See [Setup a KDC](Kerberos-manual-setup.md#setup-a-kdc) and [Verify KDC](Kerberos-manual-setup.md#verify-kdc)
## Ambari Setup
1. Kerberize the cluster via Ambari. More detailed documentation can be found [here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
http://git-wip-us.apache.org/repos/asf/metron/blob/5e0cce22/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index 8a7f07a..fc46572 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -5,6 +5,7 @@ This document provides instructions for kerberizing Metron's Vagrant-based devel
* [Setup](#setup)
* [Setup a KDC](#setup-a-kdc)
+* [Verify KDC](#verify-kdc)
* [Enable Kerberos](#enable-kerberos)
* [Kafka Authorization](#kafka-authorization)
* [HBase Authorization](#hbase-authorization)
@@ -75,11 +76,7 @@ Setup a KDC
```
max_renewable_life = 7d
```
-
- If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:
- ```
- Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
- ```
+
1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
@@ -100,6 +97,33 @@ Setup a KDC
kadmin.local -q "addprinc metron"
```
+Verify KDC
+----------
+
+
+Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:
+ ```
+ Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
+ ```
+
+
+Ensure the Metron keytab is renewable. Look for the 'R' flag from the following command
+ ```
+ klist -f
+ ```
+
+If the 'R' flags are present, you may skip to next section.
+
+If the 'R' flags are absent, you will need to follow the below steps:
+If the KDC is already setup, then editing max_life and max_renewable_life in `/var/kerberos/krb5kdc/kdc.conf`, and restarting kadmin and krb5kdc services will not change the policies for existing users.
+
+You need to set the renew lifetime for existing users and krbtgt realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:
+ ```
+ kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
+ kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
+ ```
+
+
Enable Kerberos
---------------
@@ -247,16 +271,8 @@ Storm Authorization
cd /home/metron/.storm
```
-1. Ensure the Metron keytab is renewable. Look for the 'R' flag from the following command
- ```
- klist -f
- ```
+1. Ensure the Metron keytab is renewable. See [Verify KDC](#verify-kdc) above
- If not present, modify the appropriate principals to allow renewable tickets. Adjust the parameters to match desired KDC parameters
- ```
- kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
- kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
- ```
1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
[03/44] metron git commit: METRON-726 Clean up mvn site generation
(justinleet) closes apache/incubator-metron#459
Posted by ma...@apache.org.
METRON-726 Clean up mvn site generation (justinleet) closes apache/incubator-metron#459
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/29e7c70d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/29e7c70d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/29e7c70d
Branch: refs/heads/Metron_0.4.0
Commit: 29e7c70dfe946667e907ae0767f47da895e722e5
Parents: 47e5aa7
Author: justinleet <ju...@gmail.com>
Authored: Tue May 2 18:16:35 2017 -0400
Committer: leet <le...@apache.org>
Committed: Tue May 2 18:16:35 2017 -0400
----------------------------------------------------------------------
.github/PULL_REQUEST_TEMPLATE.md | 3 +-
.travis.yml | 2 +-
README.md | 15 +++
metron-analytics/metron-maas-common/pom.xml | 41 +-----
metron-analytics/metron-maas-service/pom.xml | 41 +-----
metron-analytics/metron-profiler-client/pom.xml | 27 +---
metron-analytics/metron-profiler-common/pom.xml | 1 +
metron-analytics/metron-profiler/pom.xml | 70 +---------
metron-analytics/metron-statistics/pom.xml | 1 +
metron-analytics/pom.xml | 41 ------
metron-deployment/pom.xml | 1 +
metron-docker/pom.xml | 1 +
metron-interface/metron-config/pom.xml | 1 +
metron-interface/metron-rest-client/pom.xml | 1 +
metron-interface/metron-rest/pom.xml | 18 +--
metron-interface/pom.xml | 18 ---
metron-platform/elasticsearch-shaded/pom.xml | 2 +
metron-platform/metron-api/pom.xml | 2 +
metron-platform/metron-common/pom.xml | 27 +---
.../IPProtocolTransformation.java | 2 -
.../benchmark/StellarMicrobenchmark.java | 3 -
metron-platform/metron-data-management/pom.xml | 11 +-
metron-platform/metron-elasticsearch/pom.xml | 72 +---------
metron-platform/metron-enrichment/pom.xml | 43 +-----
.../integration/EnrichmentIntegrationTest.java | 2 +-
metron-platform/metron-hbase/pom.xml | 2 +
metron-platform/metron-indexing/pom.xml | 41 +-----
.../integration/IndexingIntegrationTest.java | 2 +-
metron-platform/metron-integration-test/pom.xml | 4 +-
.../metron/integration/ComponentRunner.java | 1 -
.../integration/processors/KafkaProcessor.java | 3 +
metron-platform/metron-management/pom.xml | 27 +---
metron-platform/metron-parsers/pom.xml | 29 +----
.../integration/ParserIntegrationTest.java | 2 +-
metron-platform/metron-pcap-backend/pom.xml | 28 +---
metron-platform/metron-pcap/pom.xml | 2 +
metron-platform/metron-solr/pom.xml | 51 +-------
metron-platform/metron-storm-kafka/pom.xml | 29 +----
metron-platform/metron-test-utilities/pom.xml | 7 +-
metron-platform/metron-writer/pom.xml | 27 +---
metron-platform/pom.xml | 41 ------
pom.xml | 130 ++++++++++++++++++-
site-book/README.md | 8 +-
site-book/pom.xml | 18 ++-
44 files changed, 213 insertions(+), 685 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/.github/PULL_REQUEST_TEMPLATE.md
----------------------------------------------------------------------
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 6c1f2a5..92f557e 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -34,8 +34,7 @@ In order to streamline the review of the contribution we ask you follow these gu
```
cd site-book
- bin/generate-md.sh
- mvn site:site
+ mvn site
```
#### Note:
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/.travis.yml
----------------------------------------------------------------------
diff --git a/.travis.yml b/.travis.yml
index 1632b6e..8fb218a 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -17,7 +17,7 @@ before_install:
- export PATH=$M2_HOME/bin:$PATH
script:
- |
- time mvn -q -T 2C -DskipTests install && time mvn -q -T 2C surefire:test@unit-tests && mvn -q surefire:test@integration-tests && time mvn -q test --projects metron-interface/metron-config && time build_utils/verify_licenses.sh
+ time mvn -q -T 2C -DskipTests install && time mvn -q -T 2C jacoco:prepare-agent surefire:test@unit-tests && mvn -q jacoco:prepare-agent surefire:test@integration-tests && time mvn -q jacoco:prepare-agent test --projects metron-interface/metron-config && time build_utils/verify_licenses.sh
cache:
directories:
- $HOME/.m2
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 09f72c3..53a84c4 100644
--- a/README.md
+++ b/README.md
@@ -83,6 +83,21 @@ $ mvn clean install -PHDP-2.5.0.0
You can swap "install" for "package" in the commands above if you don't want to deploy the artifacts to your local .m2 repo.
+# Build Metron Reporting
+
+To build and run reporting with code coverage:
+```
+$ mvn clean install
+$ mvn site site:stage-deploy site:deploy
+```
+
+Code coverage can be skipped by skipping tests:
+```
+$ mvn clean install -DskipTests site site:stage-deploy site:deploy
+```
+
+The staged site is deployed to /tmp/metron/site/index.html, and can be viewed by opening the file in a browser.
+
# Navigating the Architecture
Metron is at its core a Kappa architecture with Apache Storm as the processing
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/metron-maas-common/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-maas-common/pom.xml b/metron-analytics/metron-maas-common/pom.xml
index 20969df..ef51c62 100644
--- a/metron-analytics/metron-maas-common/pom.xml
+++ b/metron-analytics/metron-maas-common/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-maas-common</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -75,46 +76,6 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <systemProperties>
- <property>
- <name>mode</name>
- <value>global</value>
- </property>
- </systemProperties>
- </configuration>
- </plugin>
-
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/metron-maas-service/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-maas-service/pom.xml b/metron-analytics/metron-maas-service/pom.xml
index ac016d4..32a9e5d 100644
--- a/metron-analytics/metron-maas-service/pom.xml
+++ b/metron-analytics/metron-maas-service/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-maas-service</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -204,46 +205,6 @@
<scope>test</scope>
</dependency>
</dependencies>
- <reporting>
- <plugins>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <systemProperties>
- <property>
- <name>mode</name>
- <value>global</value>
- </property>
- </systemProperties>
- </configuration>
- </plugin>
-
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/metron-profiler-client/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/pom.xml b/metron-analytics/metron-profiler-client/pom.xml
index e87110a..812a6ec 100644
--- a/metron-analytics/metron-profiler-client/pom.xml
+++ b/metron-analytics/metron-profiler-client/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-profiler-client</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -248,32 +249,6 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/metron-profiler-common/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-common/pom.xml b/metron-analytics/metron-profiler-common/pom.xml
index b557e71..4f49ec2 100644
--- a/metron-analytics/metron-profiler-common/pom.xml
+++ b/metron-analytics/metron-profiler-common/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-profiler-common</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/metron-profiler/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler/pom.xml b/metron-analytics/metron-profiler/pom.xml
index 1746c51..cecf464 100644
--- a/metron-analytics/metron-profiler/pom.xml
+++ b/metron-analytics/metron-profiler/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-profiler</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -272,79 +273,10 @@
</exclusions>
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- <inherited>true</inherited>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
- <!-- Separates the unit tests from the integration tests. -->
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>2.12.4</version>
- <configuration>
- <!-- Skip the default running of this plug-in (or everything is run twice...see below) -->
- <argLine>-Xmx2048m</argLine>
- <skip>true</skip>
- <!-- Show 100% of the lines from the stack trace (doesn't work) -->
- <trimStackTrace>false</trimStackTrace>
- </configuration>
- <executions>
- <execution>
- <id>unit-tests</id>
- <phase>test</phase>
- <goals>
- <goal>test</goal>
- </goals>
- <configuration>
- <!-- Never skip running the tests when the test phase is invoked -->
- <skip>false</skip>
- <includes>
- <!-- Include unit tests within integration-test phase. -->
- <include>**/*Test.java</include>
- </includes>
- <excludes>
- <!-- Exclude integration tests within (unit) test phase. -->
- <exclude>**/*IntegrationTest.java</exclude>
- </excludes>
-
- </configuration>
- </execution>
- <execution>
- <id>integration-tests</id>
- <phase>integration-test</phase>
- <goals>
- <goal>test</goal>
- </goals>
- <configuration>
- <!-- Never skip running the tests when the integration-test phase is invoked -->
- <skip>false</skip>
- <includes>
- <!-- Include integration tests within integration-test phase. -->
- <include>**/*IntegrationTest.java</include>
- </includes>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${global_shade_version}</version>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/metron-statistics/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/pom.xml b/metron-analytics/metron-statistics/pom.xml
index d8db748..6489ae4 100644
--- a/metron-analytics/metron-statistics/pom.xml
+++ b/metron-analytics/metron-statistics/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-statistics</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-analytics/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/pom.xml b/metron-analytics/pom.xml
index 552de67..7670e8f 100644
--- a/metron-analytics/pom.xml
+++ b/metron-analytics/pom.xml
@@ -75,47 +75,6 @@
<plugins>
</plugins>
</build>
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>2.18</version>
- <configuration>
- <argLine>-Xmx2048m -XX:MaxPermSize=256m</argLine>
- <systemProperties>
- <property>
- <name>mode</name>
- <value>local</value>
- </property>
- </systemProperties>
- </configuration>
- </plugin>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <version>3.3</version>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- <inherited>true</inherited>
- </plugin>
- </plugins>
- </reporting>
<repositories>
<repository>
<id>clojars.org</id>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-deployment/pom.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/pom.xml b/metron-deployment/pom.xml
index fa7e1f9..e130ab4 100644
--- a/metron-deployment/pom.xml
+++ b/metron-deployment/pom.xml
@@ -24,6 +24,7 @@
<version>0.4.0</version>
</parent>
<description>Building and deploying Metron</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-docker/pom.xml
----------------------------------------------------------------------
diff --git a/metron-docker/pom.xml b/metron-docker/pom.xml
index 947e0bb..66ca5df 100644
--- a/metron-docker/pom.xml
+++ b/metron-docker/pom.xml
@@ -24,6 +24,7 @@
<version>0.4.0</version>
</parent>
<description>Metron Docker</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-interface/metron-config/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/pom.xml b/metron-interface/metron-config/pom.xml
index 646371d..2099817 100644
--- a/metron-interface/metron-config/pom.xml
+++ b/metron-interface/metron-config/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-config</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-interface/metron-rest-client/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest-client/pom.xml b/metron-interface/metron-rest-client/pom.xml
index 1c01acb..7bcd784 100644
--- a/metron-interface/metron-rest-client/pom.xml
+++ b/metron-interface/metron-rest-client/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-rest-client</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
</properties>
<dependencies>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-interface/metron-rest/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/pom.xml b/metron-interface/metron-rest/pom.xml
index cc53192..b11e999 100644
--- a/metron-interface/metron-rest/pom.xml
+++ b/metron-interface/metron-rest/pom.xml
@@ -21,6 +21,7 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-rest</artifactId>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -32,7 +33,6 @@
<spring.kerberos.version>1.0.1.RELEASE</spring.kerberos.version>
<swagger.version>2.5.0</swagger.version>
<mysql.client.version>5.1.40</mysql.client.version>
- <emma.version>1.0-alpha-3</emma.version>
</properties>
<dependencies>
<dependency>
@@ -254,22 +254,6 @@
</dependencies>
</dependencyManagement>
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>${emma.version}</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-interface/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/pom.xml b/metron-interface/pom.xml
index fa7e95b..58e066f 100644
--- a/metron-interface/pom.xml
+++ b/metron-interface/pom.xml
@@ -61,24 +61,6 @@
<plugins>
</plugins>
</build>
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <version>3.3</version>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- <inherited>true</inherited>
- </plugin>
- </plugins>
- </reporting>
<repositories>
<repository>
<id>multiline-release-repo</id>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/elasticsearch-shaded/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/elasticsearch-shaded/pom.xml b/metron-platform/elasticsearch-shaded/pom.xml
index b8dbf1c..ce5bb6b 100644
--- a/metron-platform/elasticsearch-shaded/pom.xml
+++ b/metron-platform/elasticsearch-shaded/pom.xml
@@ -21,6 +21,8 @@
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>elasticsearch-shaded</artifactId>
+ <name>elasticsearch-shaded</name>
+ <url>https://metron.apache.org/</url>
<dependencies>
<dependency>
<groupId>com.google.guava</groupId>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-api/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-api/pom.xml b/metron-platform/metron-api/pom.xml
index c9243e6..7fede58 100644
--- a/metron-platform/metron-api/pom.xml
+++ b/metron-platform/metron-api/pom.xml
@@ -20,7 +20,9 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-api</artifactId>
+ <name>metron-api</name>
<description>Metron API</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<flume.version>${global_flume_version}</flume.version>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-common/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/pom.xml b/metron-platform/metron-common/pom.xml
index 0aaf34c..04ac946 100644
--- a/metron-platform/metron-common/pom.xml
+++ b/metron-platform/metron-common/pom.xml
@@ -23,6 +23,7 @@
<artifactId>metron-common</artifactId>
<name>metron-common</name>
<description>Components common to all enrichments</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -360,32 +361,6 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/IPProtocolTransformation.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/IPProtocolTransformation.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/IPProtocolTransformation.java
index 2405c90..1488a9a 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/IPProtocolTransformation.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/transformation/IPProtocolTransformation.java
@@ -19,7 +19,6 @@
package org.apache.metron.common.field.transformation;
-import org.apache.hadoop.yarn.util.ConverterUtils;
import org.apache.metron.common.dsl.Context;
import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Stellar;
@@ -29,7 +28,6 @@ import org.apache.metron.common.utils.ConversionUtils;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.function.Function;
@Stellar(name="PROTOCOL_TO_NAME"
, description="Converts the IANA protocol number to the protocol name"
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/benchmark/StellarMicrobenchmark.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/benchmark/StellarMicrobenchmark.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/benchmark/StellarMicrobenchmark.java
index d674210..faa55e0 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/benchmark/StellarMicrobenchmark.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/benchmark/StellarMicrobenchmark.java
@@ -17,7 +17,6 @@
*/
package org.apache.metron.common.stellar.benchmark;
-import com.clearspring.analytics.util.Lists;
import com.fasterxml.jackson.core.type.TypeReference;
import com.google.common.base.Joiner;
import com.google.common.base.Splitter;
@@ -26,11 +25,9 @@ import org.apache.commons.cli.*;
import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
import org.apache.metron.common.dsl.Context;
import org.apache.metron.common.dsl.MapVariableResolver;
-import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.StellarFunctions;
import org.apache.metron.common.utils.JSONUtils;
import org.apache.metron.common.utils.cli.OptionHandler;
-import scala.testing.Benchmark;
import javax.annotation.Nullable;
import java.io.File;
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-data-management/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-data-management/pom.xml b/metron-platform/metron-data-management/pom.xml
index 9133eb6..2c4691f 100644
--- a/metron-platform/metron-data-management/pom.xml
+++ b/metron-platform/metron-data-management/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-data-management</artifactId>
+ <name>metron-data-management</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -366,15 +368,6 @@
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <argLine>-Xmx2048m</argLine>
- <skip>true</skip>
- <trimStackTrace>false</trimStackTrace>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${global_shade_version}</version>
<executions>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-elasticsearch/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-elasticsearch/pom.xml b/metron-platform/metron-elasticsearch/pom.xml
index 3f5b0b3..ba0f1ad 100644
--- a/metron-platform/metron-elasticsearch/pom.xml
+++ b/metron-platform/metron-elasticsearch/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-elasticsearch</artifactId>
+ <name>metron-elasticsearch</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -197,80 +199,10 @@
<scope>test</scope>
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- <inherited>true</inherited>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
- <!-- Separates the unit tests from the integration tests. -->
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>2.12.4</version>
- <configuration>
- <!-- Skip the default running of this plug-in (or everything is run twice...see below) -->
- <argLine>-Xmx2048m</argLine>
- <skip>true</skip>
- <!-- Show 100% of the lines from the stack trace (doesn't work) -->
- <trimStackTrace>false</trimStackTrace>
- </configuration>
- <executions>
- <execution>
- <id>unit-tests</id>
- <phase>test</phase>
- <goals>
- <goal>test</goal>
- </goals>
- <configuration>
- <!-- Never skip running the tests when the test phase is invoked -->
- <skip>false</skip>
- <includes>
- <!-- Include unit tests within integration-test phase. -->
- <include>**/*Test.java</include>
- </includes>
- <excludes>
- <!-- Exclude integration tests within (unit) test phase. -->
- <exclude>**/*IntegrationTest.java</exclude>
- </excludes>
- </configuration>
- </execution>
- <execution>
- <id>integration-tests</id>
- <phase>integration-test</phase>
- <goals>
- <goal>test</goal>
- </goals>
- <configuration>
- <!-- Never skip running the tests when the integration-test phase is invoked -->
- <skip>false</skip>
- <includes>
- <!-- Include integration tests within integration-test phase. -->
- <include>**/*IntegrationTest.java</include>
- </includes>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${global_shade_version}</version>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-enrichment/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/pom.xml b/metron-platform/metron-enrichment/pom.xml
index bebd774..29de74f 100644
--- a/metron-platform/metron-enrichment/pom.xml
+++ b/metron-platform/metron-enrichment/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-enrichment</artifactId>
+ <name>metron-enrichment</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -262,47 +264,6 @@
<version>${commons-compress.version}</version>
</dependency>
</dependencies>
- <reporting>
- <plugins>
-
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <systemProperties>
- <property>
- <name>mode</name>
- <value>global</value>
- </property>
- </systemProperties>
- </configuration>
- </plugin>
-
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
-
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
index 77b64dc..9634805 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
@@ -198,9 +198,9 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
.withCustomShutdownOrder(new String[]{"storm","config","kafka","zk"})
.withNumRetries(10)
.build();
- runner.start();
try {
+ runner.start();
fluxComponent.submitTopology();
kafkaComponent.writeMessages(Constants.ENRICHMENT_TOPIC, inputMessages);
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-hbase/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-hbase/pom.xml b/metron-platform/metron-hbase/pom.xml
index a2f4792..22d55b2 100644
--- a/metron-platform/metron-hbase/pom.xml
+++ b/metron-platform/metron-hbase/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-hbase</artifactId>
+ <name>metron-hbase</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-indexing/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/pom.xml b/metron-platform/metron-indexing/pom.xml
index ff34a97..6a18dc4 100644
--- a/metron-platform/metron-indexing/pom.xml
+++ b/metron-platform/metron-indexing/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-indexing</artifactId>
+ <name>metron-indexing</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -139,45 +141,6 @@
<scope>test</scope>
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <configuration>
- <systemProperties>
- <property>
- <name>mode</name>
- <value>global</value>
- </property>
- </systemProperties>
- </configuration>
- </plugin>
-
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java b/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
index ff786cf..da46d93 100644
--- a/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
+++ b/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/integration/IndexingIntegrationTest.java
@@ -182,9 +182,9 @@ public abstract class IndexingIntegrationTest extends BaseIntegrationTest {
.withMaxTimeMS(150000)
.withCustomShutdownOrder(new String[] {"search","storm","config","kafka","zk"})
.build();
- runner.start();
try {
+ runner.start();
while(!isLoaded.get()) {
Thread.sleep(100);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-integration-test/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-integration-test/pom.xml b/metron-platform/metron-integration-test/pom.xml
index 6e83f97..54f4a6b 100644
--- a/metron-platform/metron-integration-test/pom.xml
+++ b/metron-platform/metron-integration-test/pom.xml
@@ -21,7 +21,9 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-integration-test</artifactId>
+ <name>metron-integration-test</name>
<description>Metron Integration Test</description>
+ <url>https://metron.apache.org/</url>
<properties>
</properties>
<dependencies>
@@ -268,6 +270,4 @@
</plugin>
</plugins>
</build>
- <reporting>
- </reporting>
</project>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/ComponentRunner.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/ComponentRunner.java b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/ComponentRunner.java
index b1f581d..ce7cab8 100644
--- a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/ComponentRunner.java
+++ b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/ComponentRunner.java
@@ -121,7 +121,6 @@ public class ComponentRunner {
}
}
-
public <T> ProcessorResult<T> process(Processor<T> successState) {
int retryCount = 0;
long start = System.currentTimeMillis();
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/processors/KafkaProcessor.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/processors/KafkaProcessor.java b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/processors/KafkaProcessor.java
index 63f073d..50be8ae 100644
--- a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/processors/KafkaProcessor.java
+++ b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/processors/KafkaProcessor.java
@@ -58,6 +58,7 @@ public class KafkaProcessor<T> implements Processor<T> {
private Function<KafkaMessageSet, Boolean> validateReadMessages;
private Function<KafkaMessageSet,T> provideResult;
+ @Override
public ReadinessState process(ComponentRunner runner){
KafkaComponent kafkaComponent = runner.getComponent(kafkaComponentName, KafkaComponent.class);
LinkedList<byte[]> outputMessages = new LinkedList<>(kafkaComponent.readMessages(readTopic));
@@ -79,6 +80,8 @@ public class KafkaProcessor<T> implements Processor<T> {
}
return ReadinessState.NOT_READY;
}
+
+ @Override
@SuppressWarnings("unchecked")
public ProcessorResult<T> getResult(){
ProcessorResult.Builder<T> builder = new ProcessorResult.Builder();
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-management/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-management/pom.xml b/metron-platform/metron-management/pom.xml
index 1dad3af..638d65f 100644
--- a/metron-platform/metron-management/pom.xml
+++ b/metron-platform/metron-management/pom.xml
@@ -23,6 +23,7 @@
<artifactId>metron-management</artifactId>
<name>metron-management</name>
<description>Management Stellar functions</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -181,32 +182,6 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-parsers/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/pom.xml b/metron-platform/metron-parsers/pom.xml
index 97cfd49..cce975a 100644
--- a/metron-platform/metron-parsers/pom.xml
+++ b/metron-platform/metron-parsers/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-parsers</artifactId>
+ <name>metron-parsers</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -218,33 +220,6 @@
<scope>test</scope>
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>1.7</targetJdk>
- </configuration>
-
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
index a170a2c..defd815 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
@@ -74,8 +74,8 @@ public abstract class ParserIntegrationTest extends BaseIntegrationTest {
.withNumRetries(10)
.withCustomShutdownOrder(new String[] {"org/apache/storm","config","kafka","zk"})
.build();
- runner.start();
try {
+ runner.start();
kafkaComponent.writeMessages(sensorType, inputMessages);
ProcessorResult<List<byte[]>> result = runner.process(getProcessor());
List<byte[]> outputMessages = result.getResult();
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-pcap-backend/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/pom.xml b/metron-platform/metron-pcap-backend/pom.xml
index 106f9fb..3095e5b 100644
--- a/metron-platform/metron-pcap-backend/pom.xml
+++ b/metron-platform/metron-pcap-backend/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-pcap-backend</artifactId>
+ <name>metron-pcap-backend</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -198,32 +200,6 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-pcap/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/pom.xml b/metron-platform/metron-pcap/pom.xml
index 73f57f0..7b9c9e5 100644
--- a/metron-platform/metron-pcap/pom.xml
+++ b/metron-platform/metron-pcap/pom.xml
@@ -21,7 +21,9 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-pcap</artifactId>
+ <name>metron-pcap</name>
<description>Metron Pcap</description>
+ <url>https://metron.apache.org/</url>
<properties>
</properties>
<dependencies>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-solr/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-solr/pom.xml b/metron-platform/metron-solr/pom.xml
index c287dce..06f7822 100644
--- a/metron-platform/metron-solr/pom.xml
+++ b/metron-platform/metron-solr/pom.xml
@@ -21,6 +21,8 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-solr</artifactId>
+ <name>metron-solr</name>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -215,55 +217,6 @@
<build>
<plugins>
<plugin>
- <!-- Separates the unit tests from the integration tests. -->
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>2.12.4</version>
- <configuration>
- <!-- Skip the default running of this plug-in (or everything is run twice...see below) -->
- <argLine>-Xmx2048m</argLine>
- <skip>true</skip>
- <!-- Show 100% of the lines from the stack trace (doesn't work) -->
- <trimStackTrace>false</trimStackTrace>
- </configuration>
- <executions>
- <execution>
- <id>unit-tests</id>
- <phase>test</phase>
- <goals>
- <goal>test</goal>
- </goals>
- <configuration>
- <!-- Never skip running the tests when the test phase is invoked -->
- <skip>false</skip>
- <includes>
- <!-- Include unit tests within integration-test phase. -->
- <include>**/*Test.java</include>
- </includes>
- <excludes>
- <!-- Exclude integration tests within (unit) test phase. -->
- <exclude>**/*IntegrationTest.java</exclude>
- </excludes>
- </configuration>
- </execution>
- <execution>
- <id>integration-tests</id>
- <phase>integration-test</phase>
- <goals>
- <goal>test</goal>
- </goals>
- <configuration>
- <!-- Never skip running the tests when the integration-test phase is invoked -->
- <skip>false</skip>
- <includes>
- <!-- Include integration tests within integration-test phase. -->
- <include>**/*IntegrationTest.java</include>
- </includes>
- </configuration>
- </execution>
- </executions>
- </plugin>
- <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>${global_shade_version}</version>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-storm-kafka/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka/pom.xml b/metron-platform/metron-storm-kafka/pom.xml
index fe93f65..b8e3f8d 100644
--- a/metron-platform/metron-storm-kafka/pom.xml
+++ b/metron-platform/metron-storm-kafka/pom.xml
@@ -23,6 +23,7 @@
<artifactId>metron-storm-kafka</artifactId>
<name>metron-storm-kafka</name>
<description>Components that extend the Storm/Kafka spout</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -90,35 +91,7 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
- <plugins>
- </plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-test-utilities/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-test-utilities/pom.xml b/metron-platform/metron-test-utilities/pom.xml
index 8cfebe0..98ce767 100644
--- a/metron-platform/metron-test-utilities/pom.xml
+++ b/metron-platform/metron-test-utilities/pom.xml
@@ -21,7 +21,9 @@
<version>0.4.0</version>
</parent>
<artifactId>metron-test-utilities</artifactId>
+ <name>metron-test-utilities</name>
<description>Metron Test Utilities</description>
+ <url>https://metron.apache.org/</url>
<properties>
</properties>
<dependencies>
@@ -156,9 +158,4 @@
<version>0.1.2</version>
</dependency>
</dependencies>
-
- <build>
- </build>
- <reporting>
- </reporting>
</project>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/metron-writer/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/pom.xml b/metron-platform/metron-writer/pom.xml
index f984329..65b9528 100644
--- a/metron-platform/metron-writer/pom.xml
+++ b/metron-platform/metron-writer/pom.xml
@@ -23,6 +23,7 @@
<artifactId>metron-writer</artifactId>
<name>metron-writer</name>
<description>Components common to all enrichments</description>
+ <url>https://metron.apache.org/</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
@@ -200,32 +201,6 @@
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
-
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- </plugins>
- </reporting>
<build>
<plugins>
<plugin>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/metron-platform/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/pom.xml b/metron-platform/pom.xml
index 68d9be8..1376e5c 100644
--- a/metron-platform/pom.xml
+++ b/metron-platform/pom.xml
@@ -95,45 +95,4 @@
<scope>test</scope>
</dependency>
</dependencies>
- <reporting>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-surefire-plugin</artifactId>
- <version>2.18</version>
- <configuration>
- <argLine>-Xmx2048m -XX:MaxPermSize=256m</argLine>
- <systemProperties>
- <property>
- <name>mode</name>
- <value>local</value>
- </property>
- </systemProperties>
- </configuration>
- </plugin>
- <!-- Normally, dependency report takes time, skip it -->
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-project-info-reports-plugin</artifactId>
- <version>2.7</version>
- <configuration>
- <dependencyLocationsEnabled>false</dependencyLocationsEnabled>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-pmd-plugin</artifactId>
- <version>3.3</version>
- <configuration>
- <targetJdk>${global_java_version}</targetJdk>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>emma-maven-plugin</artifactId>
- <version>1.0-alpha-3</version>
- <inherited>true</inherited>
- </plugin>
- </plugins>
- </reporting>
</project>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 96885d6..bb32c64 100644
--- a/pom.xml
+++ b/pom.xml
@@ -24,12 +24,17 @@
<name>Metron</name>
<description>Metron Top Level Project</description>
<url>https://metron.apache.org/</url>
+ <organization>
+ <name>The Apache Software Foundation</name>
+ <url>https://www.apache.org</url>
+ </organization>
<modules>
<module>metron-analytics</module>
<module>metron-platform</module>
<module>metron-deployment</module>
<module>metron-docker</module>
<module>metron-interface</module>
+ <module>site-book</module>
</modules>
<repositories>
@@ -124,6 +129,7 @@
<global_jackson_version>2.7.4</global_jackson_version>
<global_errorprone_core_version>2.0.14</global_errorprone_core_version>
<global_jar_version>3.0.2</global_jar_version>
+ <global_surefire_version>2.18</global_surefire_version>
</properties>
<profiles>
@@ -159,10 +165,10 @@
<!-- Separates the unit tests from the integration tests. -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
- <version>2.18</version>
+ <version>${global_surefire_version}</version>
<configuration>
<!-- Skip the default running of this plug-in (or everything is run twice...see below) -->
- <argLine>-Xmx2048m</argLine>
+ <argLine>@{argLine} -Xmx2048m</argLine>
<skip>true</skip>
<!-- Show 100% of the lines from the stack trace (doesn't work) -->
<trimStackTrace>false</trimStackTrace>
@@ -216,6 +222,11 @@
</pluginManagement>
<plugins>
<plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-site-plugin</artifactId>
+ <version>3.6</version>
+ </plugin>
+ <plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
@@ -255,6 +266,25 @@
</configuration>
</plugin>
<plugin>
+ <groupId>org.jacoco</groupId>
+ <artifactId>jacoco-maven-plugin</artifactId>
+ <version>0.7.9</version>
+ <executions>
+ <execution>
+ <goals>
+ <goal>prepare-agent</goal>
+ </goals>
+ </execution>
+ <execution>
+ <id>report</id>
+ <phase>prepare-package</phase>
+ <goals>
+ <goal>report</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.12</version>
@@ -320,4 +350,100 @@
</plugin>
</plugins>
</build>
+
+ <distributionManagement>
+ <site>
+ <id>metron</id>
+ <name>Metron</name>
+ <url>file:///tmp/metron/site/</url>
+ </site>
+ </distributionManagement>
+ <reporting>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-jxr-plugin</artifactId>
+ <version>2.3</version>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-javadoc-plugin</artifactId>
+ <version>2.10.4</version>
+ <configuration>
+ <additionalparam>-Xdoclint:none</additionalparam>
+ </configuration>
+ <reportSets>
+ <reportSet>
+ <id>default</id>
+ <reports>
+ <report>javadoc</report>
+ <report>test-javadoc</report>
+ </reports>
+ </reportSet>
+ <reportSet>
+ <id>aggregate</id>
+ <reports>
+ <report>aggregate</report>
+ </reports>
+ </reportSet>
+ </reportSets>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-project-info-reports-plugin</artifactId>
+ <version>2.9</version>
+ <reportSets>
+ <reportSet>
+ <reports>
+ <!-- Normally, dependency report takes time, skip it -->
+ <!--<report>dependencies</report>-->
+ <report>index</report>
+ <report>cim</report>
+ <report>issue-tracking</report>
+ <report>license</report>
+ <report>mailing-list</report>
+ <report>plugins</report>
+ <report>project-team</report>
+ <report>scm</report>
+ <report>summary</report>
+ </reports>
+ </reportSet>
+ </reportSets>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-checkstyle-plugin</artifactId>
+ <version>2.17</version>
+ <reportSets>
+ <reportSet>
+ <reports>
+ <report>checkstyle</report>
+ <report>checkstyle-aggregate</report>
+ </reports>
+ </reportSet>
+ </reportSets>
+ </plugin>
+ <plugin>
+ <groupId>org.jacoco</groupId>
+ <artifactId>jacoco-maven-plugin</artifactId>
+ <version>0.7.9</version>
+ <reportSets>
+ <reportSet>
+ <reports>
+ <!-- select non-aggregate reports -->
+ <report>report</report>
+ </reports>
+ </reportSet>
+ </reportSets>
+ </plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-pmd-plugin</artifactId>
+ <version>3.7</version>
+ <configuration>
+ <targetJdk>${global_java_version}</targetJdk>
+ </configuration>
+ </plugin>
+ </plugins>
+ </reporting>
</project>
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/site-book/README.md
----------------------------------------------------------------------
diff --git a/site-book/README.md b/site-book/README.md
index d1060a0..6c6b7bd 100644
--- a/site-book/README.md
+++ b/site-book/README.md
@@ -8,15 +8,15 @@ Metron's Site Book is an attempt at producing documentation that is:
The idea is that a release manager would build the site-book (following the instructions below), then publish it from the public [Metron site](http://metron.apache.org/) as the docs for the new released version. Older site-book versions should remain available for users that need them.
+The site-book is also part of the Maven site lifecycle, and will be included by the full site from the top level. However, the site as a whole takes longer than just the site-book:
-To build the book, do the following:
+To build only the book, do the following:
In any git clone of incubator-metron containing the site-book subdirectory,
```
cd site-book
-bin/generate-md.sh
-mvn site:site
+mvn site
```
It only takes a few seconds. You may now view your copy of the book in a browser by opening
@@ -31,6 +31,7 @@ On a Mac, you can just type the following on the command line
open target/site/index.html
```
+
##Key Components:
###bin/generate-md.sh
@@ -38,6 +39,7 @@ open target/site/index.html
- Copies all .md files from the code directory tree into the site tree
- Performs some transformations on them
- Generates the nav tree structure and labels
+- Happens during the site:pre-site phase of Maven.
###bin/fix-md-dialect.py
http://git-wip-us.apache.org/repos/asf/metron/blob/29e7c70d/site-book/pom.xml
----------------------------------------------------------------------
diff --git a/site-book/pom.xml b/site-book/pom.xml
index 2f04c71..71e4879 100644
--- a/site-book/pom.xml
+++ b/site-book/pom.xml
@@ -51,6 +51,22 @@
<build>
<plugins>
+ <plugin>
+ <artifactId>exec-maven-plugin</artifactId>
+ <groupId>org.codehaus.mojo</groupId>
+ <executions>
+ <execution>
+ <id>Generate MD</id>
+ <phase>pre-site</phase>
+ <goals>
+ <goal>exec</goal>
+ </goals>
+ <configuration>
+ <executable>${basedir}/bin/generate-md.sh</executable>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
@@ -73,10 +89,8 @@
<configuration>
<generateProjectInfo>false</generateProjectInfo>
<generateReports>false</generateReports>
- <skip>false</skip>
</configuration>
</plugin>
</plugins>
</build>
-
</project>
[10/44] metron git commit: METRON-913 Create IP Report in Zeppelin
closes apache/incubator-metron#561
Posted by ma...@apache.org.
METRON-913 Create IP Report in Zeppelin closes apache/incubator-metron#561
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/8cf109da
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/8cf109da
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/8cf109da
Branch: refs/heads/Metron_0.4.0
Commit: 8cf109da180ee9f80984b0408afed799fa6c7c43
Parents: 082b218
Author: David Lyle <dl...@gmail.com>
Authored: Tue May 9 14:07:36 2017 -0400
Committer: David Lyle <dl...@gmail.com>
Committed: Tue May 9 14:07:36 2017 -0400
----------------------------------------------------------------------
.../docker/rpm-docker/SPECS/metron.spec | 9 +
.../zeppelin/metron/metron-ip-report.json | 439 +++++++++++++++++++
2 files changed, 448 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/8cf109da/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index 76129c3..da8f192 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -269,7 +269,11 @@ This package installs the Metron Indexing files
%{metron_home}/config/zookeeper/indexing/asa.json
%{metron_home}/config/zookeeper/indexing/error.json
%{metron_home}/config/zeppelin/metron/metron-yaf-telemetry.json
+<<<<<<< HEAD
%{metron_home}/config/zeppelin/metron/metron-connection-report.json
+=======
+%{metron_home}/config/zeppelin/metron/metron-ip-report.json
+>>>>>>> 1cace9ff29f31301d74fa6a7b2630d471452e985
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -399,10 +403,15 @@ This package installs the Metron Management UI %{metron_home}
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%changelog
+<<<<<<< HEAD
* Thu May 4 2017 Ryan Merriman <me...@gmail.com> - 0.4.0
- Added REST
* Fri Apr 28 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
- Add Zeppelin Connection Report Dashboard
+=======
+* Tue May 2 2017 David Lyle <dl...@gmail.com> - 0.4.0
+- Add Metron IP Report
+>>>>>>> 1cace9ff29f31301d74fa6a7b2630d471452e985
* Thu Jan 19 2017 Justin Leet <ju...@gmail.com> - 0.3.1
- Replace GeoIP files with new implementation
* Thu Nov 03 2016 David Lyle <dl...@gmail.com> - 0.2.1
http://git-wip-us.apache.org/repos/asf/metron/blob/8cf109da/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-ip-report.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-ip-report.json b/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-ip-report.json
new file mode 100644
index 0000000..cf226c9
--- /dev/null
+++ b/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-ip-report.json
@@ -0,0 +1,439 @@
+{
+ "paragraphs": [
+ {
+ "text": "%md\n\n# Metron IP Report\n\nFor a given Source IP Address, produces the following reports:\n\n* Most Frequent Connections (defaults to 24 hours)\n* Recent Connections (defaults to 1 hour)\n* Top DNS Queries (defaults to 24 hours)\n* All Ports Used (defaults to 24 hours)\n* HTTP User Agents (defaults to 24 hours)\n\nAll reports default to a 50 row limit.\n\n\nTo execute the reports, select the Source IP Adress in the box below and click on 'Run All Paragraphs' in the top tool bar.\n\nReports may be executed individually, but be sure to execute the 'Setup' SQL Context and Tables first.",
+ "dateUpdated": "2017-05-02T15:40:11+0000",
+ "config": {
+ "colWidth": 12,
+ "editorMode": "ace/mode/scala",
+ "editorHide": true,
+ "graph": {
+ "mode": "table",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [],
+ "values": [],
+ "groups": [],
+ "scatter": {}
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {},
+ "forms": {}
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-152938_356593846",
+ "result": {
+ "code": "SUCCESS",
+ "type": "HTML",
+ "msg": "<h1>Metron IP Report</h1>\n<p>For a given Source IP Address, produces the following reports:</p>\n<ul>\n<li>Most Frequent Connections (defaults to 24 hours)</li>\n<li>Recent Connections (defaults to 1 hour)</li>\n<li>Top DNS Queries (defaults to 24 hours)</li>\n<li>All Ports Used (defaults to 24 hours)</li>\n<li>HTTP User Agents (defaults to 24 hours)</li>\n</ul>\n<p>All reports default to a 50 row limit.</p>\n<p>To execute the reports, select the Source IP Adress in the box below and click on 'Run All Paragraphs' in the top tool bar.</p>\n<p>Reports may be executed individually, but be sure to execute the 'Setup' SQL Context and Tables first.</p>\n"
+ },
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19401"
+ },
+ {
+ "title": "Setup SQL Context and Tables",
+ "text": "%spark\n/**\n* date_sub_hours: Subtract an hours worth of seconds from a timestamp\n*/\ndef date_sub_hours(t:Long, h:Int) : Long = {\n t - h*3600\n}\n\n/**\n* source_ip_address: Provides the same Source IP Address to all reports\n*/\nvar srcIp = z.input(\"Source IP Address\")\ndef source_ip_address() : String = {\n srcIp.toString\n}\n\n//Register udfs\nsqlc.udf.register(\"source_ip_address\", source_ip_address _)\nsqlc.udf.register(\"date_sub_hours\", date_sub_hours _)\n\n//Load Yaf Data \nval yaf = sqlc.read.json(\"hdfs:///apps/metron/indexing/indexed/yaf\")\nyaf.registerTempTable(\"yaf\")\n\n//Load Bro Data\nval bro = sqlc.read.json(\"hdfs:///apps/metron/indexing/indexed/bro\")\nbro.registerTempTable(\"bro\")",
+ "dateUpdated": "2017-05-02T15:40:11+0000",
+ "config": {
+ "tableHide": false,
+ "colWidth": 12,
+ "editorMode": "ace/mode/scala",
+ "editorHide": true,
+ "title": true,
+ "graph": {
+ "mode": "table",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [],
+ "values": [],
+ "groups": [],
+ "scatter": {}
+ },
+ "enabled": true
+ },
+ "settings": {
+ "params": {
+ "Source IP Address": ""
+ },
+ "forms": {
+ "Source IP Address": {
+ "name": "Source IP Address",
+ "displayName": "Source IP Address",
+ "type": "input",
+ "defaultValue": "",
+ "hidden": false
+ }
+ }
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-140420_1010901147",
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19402"
+ },
+ {
+ "title": "Most Frequent Connections - YAF",
+ "text": "%spark.sql\nSELECT\n ip_dst_addr,\n COUNT(*) AS count\nFROM yaf\nWHERE\n ip_src_addr = source_ip_address() AND\n timestamp BETWEEN date_sub_hours(unix_timestamp(),${Number of Hours=24}) * 1000 AND unix_timestamp() * 1000\nGROUP BY ip_dst_addr\nORDER BY count DESC\nLIMIT ${Number of Rows=50}",
+ "dateUpdated": "2017-05-02T15:40:18+0000",
+ "config": {
+ "colWidth": 12,
+ "editorMode": "ace/mode/sql",
+ "title": true,
+ "graph": {
+ "mode": "multiBarChart",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [
+ {
+ "name": "ip_dst_addr",
+ "index": 0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "scatter": {
+ "xAxis": {
+ "name": "ip_dst_addr",
+ "index": 0,
+ "aggr": "sum"
+ },
+ "yAxis": {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ }
+ },
+ "enabled": true,
+ "editorHide": true
+ },
+ "settings": {
+ "params": {
+ "Number of Rows": "50",
+ "Number of Hours": "24",
+ "Source IP Address": "192.168.138.158"
+ },
+ "forms": {
+ "Number of Rows": {
+ "name": "Number of Rows",
+ "defaultValue": "50",
+ "hidden": false
+ },
+ "Number of Hours": {
+ "name": "Number of Hours",
+ "defaultValue": "24",
+ "hidden": false
+ }
+ }
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-140443_302943667",
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19403"
+ },
+ {
+ "title": "Recent Connections - YAF",
+ "text": "%spark.sql\nSELECT\n ip_dst_addr,\n COUNT(*) as count\nFROM yaf\nWHERE\n ip_src_addr = source_ip_address() AND\n timestamp BETWEEN date_sub_hours(unix_timestamp(),${Number of Hours=1}) * 1000 AND unix_timestamp() * 1000\nGROUP BY ip_dst_addr\nORDER BY count DESC\nLIMIT ${Number of Rows=50}",
+ "dateUpdated": "2017-05-02T15:40:21+0000",
+ "config": {
+ "colWidth": 12,
+ "editorMode": "ace/mode/sql",
+ "title": true,
+ "graph": {
+ "mode": "multiBarChart",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [
+ {
+ "name": "ip_dst_addr",
+ "index": 0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "scatter": {
+ "xAxis": {
+ "name": "ip_dst_addr",
+ "index": 0,
+ "aggr": "sum"
+ },
+ "yAxis": {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ }
+ },
+ "enabled": true,
+ "editorHide": true
+ },
+ "settings": {
+ "params": {
+ "Number of Rows": "50",
+ "Number of Hours": "1",
+ "Source IP Address": "192.168.138.158"
+ },
+ "forms": {
+ "Number of Rows": {
+ "name": "Number of Rows",
+ "defaultValue": "50",
+ "hidden": false
+ },
+ "Number of Hours": {
+ "name": "Number of Hours",
+ "defaultValue": "1",
+ "hidden": false
+ }
+ }
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-140621_1919836666",
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19404"
+ },
+ {
+ "title": "Top DNS Queries - BRO",
+ "text": "%spark.sql\nSELECT \n query as domain_name,\n COUNT(*) as count\nFROM bro \nWHERE\n ip_src_addr = source_ip_address()\n AND\n timestamp BETWEEN date_sub_hours(unix_timestamp(),${Number of Hours=24}) * 1000 AND unix_timestamp() * 1000 \n AND\n protocol = 'dns' \n AND \n query is not null\nGROUP BY query\nORDER BY count DESC\nLIMIT ${Number of Rows=50}\n ",
+ "dateUpdated": "2017-05-02T15:40:23+0000",
+ "config": {
+ "colWidth": 12,
+ "editorMode": "ace/mode/sql",
+ "title": true,
+ "graph": {
+ "mode": "table",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [
+ {
+ "name": "domain_name",
+ "index": 0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "scatter": {
+ "xAxis": {
+ "name": "domain_name",
+ "index": 0,
+ "aggr": "sum"
+ },
+ "yAxis": {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ }
+ },
+ "enabled": true,
+ "editorHide": true
+ },
+ "settings": {
+ "params": {
+ "Number of Rows": "50",
+ "Number of Hours": "24",
+ "Source IP Address": "192.168.138.158"
+ },
+ "forms": {
+ "Number of Rows": {
+ "name": "Number of Rows",
+ "defaultValue": "50",
+ "hidden": false
+ },
+ "Number of Hours": {
+ "name": "Number of Hours",
+ "defaultValue": "24",
+ "hidden": false
+ }
+ }
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-141019_193020479",
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19405"
+ },
+ {
+ "title": "All Ports Used - YAF",
+ "text": "%spark.sql\nSELECT\n ip_dst_port,\n COUNT(*) as count\nFROM yaf \nWHERE\n ip_src_addr = source_ip_address()\n AND\n timestamp BETWEEN date_sub_hours(unix_timestamp(),${Number of Hours=24}) * 1000 AND unix_timestamp() * 1000 \nGROUP BY ip_dst_port\nORDER BY count DESC\nLIMIT ${Number of Rows=50}",
+ "dateUpdated": "2017-05-02T15:40:25+0000",
+ "config": {
+ "colWidth": 12,
+ "editorMode": "ace/mode/sql",
+ "title": true,
+ "graph": {
+ "mode": "table",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [
+ {
+ "name": "ip_dst_port",
+ "index": 0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "scatter": {
+ "xAxis": {
+ "name": "ip_dst_port",
+ "index": 0,
+ "aggr": "sum"
+ },
+ "yAxis": {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ }
+ },
+ "enabled": true,
+ "editorHide": true
+ },
+ "settings": {
+ "params": {
+ "Number of Rows": "50",
+ "Number of Hours": "24"
+ },
+ "forms": {
+ "Number of Rows": {
+ "name": "Number of Rows",
+ "defaultValue": "50",
+ "hidden": false
+ },
+ "Number of Hours": {
+ "name": "Number of Hours",
+ "defaultValue": "24",
+ "hidden": false
+ }
+ }
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-141412_380718762",
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19406"
+ },
+ {
+ "title": "HTTP User Agents - BRO",
+ "text": "%spark.sql\nSELECT\n user_agent,\n COUNT(*) as count\nFROM bro\nWHERE \n ip_src_addr = source_ip_address()\n AND\n timestamp BETWEEN date_sub_hours(unix_timestamp(),${Number of Hours=24}) * 1000 AND unix_timestamp() * 1000 \n AND\n protocol = 'http' \nGROUP BY user_agent\nORDER BY count DESC\nLIMIT ${Number of Rows=50}",
+ "dateUpdated": "2017-05-02T15:40:27+0000",
+ "config": {
+ "colWidth": 12,
+ "editorMode": "ace/mode/sql",
+ "title": true,
+ "graph": {
+ "mode": "table",
+ "height": 300,
+ "optionOpen": false,
+ "keys": [
+ {
+ "name": "user_agent",
+ "index": 0,
+ "aggr": "sum"
+ }
+ ],
+ "values": [
+ {
+ "name": "count",
+ "index": 1,
+ "aggr": "sum"
+ }
+ ],
+ "groups": [],
+ "scatter": {
+ "xAxis": {
+ "name": "user_agent",
+ "index": 0,
+ "aggr": "sum"
+ }
+ }
+ },
+ "enabled": true,
+ "editorHide": true
+ },
+ "settings": {
+ "params": {
+ "Number of Hours": "24",
+ "Number of Rows": "50"
+ },
+ "forms": {
+ "Number of Rows": {
+ "name": "Number of Rows",
+ "defaultValue": "50",
+ "hidden": false
+ },
+ "Number of Hours": {
+ "name": "Number of Hours",
+ "defaultValue": "24",
+ "hidden": false
+ }
+ }
+ },
+ "jobName": "paragraph_1493739611136_-165301953",
+ "id": "20170502-143830_1892618095",
+ "dateCreated": "2017-05-02T15:40:11+0000",
+ "status": "READY",
+ "errorMessage": "",
+ "progressUpdateIntervalMs": 500,
+ "$$hashKey": "object:19407"
+ }
+ ],
+ "name": "Metron IP Report",
+ "id": "2CG7BF9JT",
+ "angularObjects": {
+ "2CEKEU5T4:shared_process": [],
+ "2CGU6FFBF:shared_process": [],
+ "2CF6U7R6S:shared_process": [],
+ "2CG2SK9DG:shared_process": [],
+ "2CFDEBF6X:shared_process": [],
+ "2CFJGFKFD:shared_process": []
+ },
+ "config": {
+ "looknfeel": "default"
+ },
+ "info": {}
+}
\ No newline at end of file
[14/44] metron git commit: METRON-931: Stellar REDUCE incorrectly
returns null for fewer than 3 items in list (mmiklavc) closes
apache/incubator-metron#565
Posted by ma...@apache.org.
METRON-931: Stellar REDUCE incorrectly returns null for fewer than 3 items in list (mmiklavc) closes apache/incubator-metron#565
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/0d5fdf41
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/0d5fdf41
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/0d5fdf41
Branch: refs/heads/Metron_0.4.0
Commit: 0d5fdf419fdb2ad863eb531fa101f0f4ecae3497
Parents: 4662c77
Author: mmiklavc <mi...@gmail.com>
Authored: Tue May 9 14:51:01 2017 -0600
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Tue May 9 14:51:01 2017 -0600
----------------------------------------------------------------------
.../dsl/functions/FunctionalFunctions.java | 2 +-
.../dsl/functions/FunctionalFunctionsTest.java | 43 +++++++++++++++++++-
2 files changed, 42 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/0d5fdf41/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java
index 0f39d1d..40e8863 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/FunctionalFunctions.java
@@ -94,7 +94,7 @@ public class FunctionalFunctions {
@Override
public Object apply(List<Object> args) {
List<Object> input = (List<Object>) args.get(0);
- if(input == null || input.size() < 3) {
+ if(input == null || args.size() < 3) {
return null;
}
LambdaExpression expression = (LambdaExpression) args.get(1);
http://git-wip-us.apache.org/repos/asf/metron/blob/0d5fdf41/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
index 2e1859e..bd70a33 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
@@ -20,6 +20,7 @@ package org.apache.metron.common.dsl.functions;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
+import org.hamcrest.CoreMatchers;
import org.junit.Assert;
import org.junit.Test;
@@ -205,8 +206,8 @@ public class FunctionalFunctionsTest {
@Test
public void testReduce() {
- for (String expr : ImmutableList.of("REDUCE([ 1, 2, 3], (x, y) -> x + y , 0 )"
- ,"REDUCE([ foo, bar, 3], (x, y) -> x + y , 0 )"
+ for (String expr : ImmutableList.of("REDUCE([ 1, 2, 3 ], (x, y) -> x + y , 0 )"
+ ,"REDUCE([ foo, bar, 3 ], (x, y) -> x + y , 0 )"
)
)
{
@@ -218,6 +219,31 @@ public class FunctionalFunctionsTest {
}
@Test
+ public void testReduce_on_various_list_sizes() {
+ {
+ String expr = "REDUCE([ 1, 2, 3, 4 ], (x, y) -> x + y , 0 )";
+ Object o = run(expr, ImmutableMap.of());
+ Assert.assertTrue(o instanceof Number);
+ Number result = (Number) o;
+ Assert.assertEquals(10, result.intValue());
+ }
+ {
+ String expr = "REDUCE([ 1, 2 ], (x, y) -> x + y , 0 )";
+ Object o = run(expr, ImmutableMap.of());
+ Assert.assertTrue(o instanceof Number);
+ Number result = (Number) o;
+ Assert.assertEquals(3, result.intValue());
+ }
+ {
+ String expr = "REDUCE([ 1 ], (x, y) -> x + y , 0 )";
+ Object o = run(expr, ImmutableMap.of());
+ Assert.assertTrue(o instanceof Number);
+ Number result = (Number) o;
+ Assert.assertEquals(1, result.intValue());
+ }
+ }
+
+ @Test
public void testReduce_NonNumeric() {
for (String expr : ImmutableList.of("REDUCE([ 'foo', 'bar', 'grok'], (x, y) -> LIST_ADD(x, y), [] )"
)
@@ -232,4 +258,17 @@ public class FunctionalFunctionsTest {
Assert.assertEquals("grok", result.get(2));
}
}
+
+ @Test
+ public void testReduce_returns_null_when_less_than_3_args() {
+ {
+ String expr = "REDUCE([ 1, 2, 3 ], (x, y) -> LIST_ADD(x, y))";
+ Assert.assertThat(run(expr, ImmutableMap.of()), CoreMatchers.equalTo(null));
+ }
+ {
+ String expr = "REDUCE([ 1, 2, 3 ])";
+ Assert.assertThat(run(expr, ImmutableMap.of()), CoreMatchers.equalTo(null));
+ }
+ }
+
}
[30/44] metron git commit: METRON-948 License field in npm
package.json files is incorrect (simonellistonball via merrimanr) closes
apache/metron#582
Posted by ma...@apache.org.
METRON-948 License field in npm package.json files is incorrect (simonellistonball via merrimanr) closes apache/metron#582
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/ce381f3e
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/ce381f3e
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/ce381f3e
Branch: refs/heads/Metron_0.4.0
Commit: ce381f3e0391c15d0ec36b2e383dea270e7d613f
Parents: 5e0cce2
Author: simonellistonball <si...@simonellistonball.com>
Authored: Fri May 19 10:25:35 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Fri May 19 10:25:35 2017 -0500
----------------------------------------------------------------------
metron-interface/metron-config/package.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/ce381f3e/metron-interface/metron-config/package.json
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/package.json b/metron-interface/metron-config/package.json
index 41d188f..008814e 100644
--- a/metron-interface/metron-config/package.json
+++ b/metron-interface/metron-config/package.json
@@ -1,7 +1,7 @@
{
"name": "metron-management-ui",
"version": "0.4.0",
- "license": "MIT",
+ "license": "Apache 2.0",
"config": {
"node_ace": "node_modules/ace-builds/src-min-noconflict/"
},
[37/44] metron git commit: METRON-953: Complete migration away from
incubator-metron to metron for links and docs (justinleet via mmiklavc)
closes apache/incubator-metron#590
Posted by ma...@apache.org.
METRON-953: Complete migration away from incubator-metron to metron for links and docs (justinleet via mmiklavc) closes apache/incubator-metron#590
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/ae05f38d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/ae05f38d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/ae05f38d
Branch: refs/heads/Metron_0.4.0
Commit: ae05f38d8728bcc8737b0a4261ab7df35710b97f
Parents: 08f95dc
Author: justinleet <ju...@gmail.com>
Authored: Thu May 25 22:53:29 2017 -0600
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Thu May 25 22:53:29 2017 -0600
----------------------------------------------------------------------
README.md | 4 ++--
Upgrading.md | 2 +-
metron-analytics/pom.xml | 6 +++---
metron-deployment/Kerberos-manual-setup.md | 2 +-
metron-deployment/README.md | 4 ++--
.../packaging/docker/ansible-docker/README.md | 4 ++--
metron-interface/metron-config/README.md | 2 +-
metron-interface/metron-rest/README.md | 2 +-
.../metron-rest/src/test/resources/README.vm | 2 +-
metron-interface/pom.xml | 6 +++---
.../src/main/config/zeppelin/metron/metron-pcap.json | 2 +-
metron-platform/pom.xml | 6 +++---
metron-sensors/fastcapa/README.md | 2 +-
metron-sensors/pycapa/README.md | 4 ++--
site-book/README.md | 4 ++--
site-book/pom.xml | 6 +++---
site/BUILD.md | 2 +-
site/README.md | 2 +-
site/_config.yml | 2 +-
site/community/index.md | 4 ++--
site/develop/index.md | 12 ++++++------
site/help/index.md | 4 ++--
site/index.html | 2 +-
23 files changed, 43 insertions(+), 43 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 06e5471..dcea8fa 100644
--- a/README.md
+++ b/README.md
@@ -58,8 +58,8 @@ This repository is a collection of submodules for convenience which is regularly
updated to point to the latest versions. Github provides multiple ways to obtain
Metron's code:
-1. git clone --recursive https://github.com/apache/incubator-metron
-2. [Download ZIP](https://github.com/apache/incubator-metron/archive/master.zip)
+1. git clone --recursive https://github.com/apache/metron
+2. [Download ZIP](https://github.com/apache/metron/archive/master.zip)
3. Clone or download each repository individually
Option 3 is more likely to have the latest code.
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/Upgrading.md
----------------------------------------------------------------------
diff --git a/Upgrading.md b/Upgrading.md
index b05acbc..3864ae3 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -15,7 +15,7 @@ As of 0.3.0 the indexing configuration
In the 0.3.1 release, this configuration has been broken out
and control for individual writers are separated.
-Please see the description of the configurations in the indexing [README](https://github.com/apache/incubator-metron/tree/Metron_0.3.1/metron-platform/metron-indexing#sensor-indexing-configuration)
+Please see the description of the configurations in the indexing [README](https://github.com/apache/metron/tree/Metron_0.3.1/metron-platform/metron-indexing#sensor-indexing-configuration)
#### Migration
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-analytics/pom.xml
----------------------------------------------------------------------
diff --git a/metron-analytics/pom.xml b/metron-analytics/pom.xml
index 7670e8f..2377702 100644
--- a/metron-analytics/pom.xml
+++ b/metron-analytics/pom.xml
@@ -27,10 +27,10 @@
<description>Stream analytics for Metron</description>
<url>https://metron.apache.org/</url>
<scm>
- <connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
- <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
+ <connection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</connection>
+ <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</developerConnection>
<tag>HEAD</tag>
- <url>https://git-wip-us.apache.org/repos/asf/incubator-metron</url>
+ <url>https://git-wip-us.apache.org/repos/asf/metron</url>
</scm>
<licenses>
<license>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index fc46572..76d3701 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -379,7 +379,7 @@ Metron should be ready to receive data.
Push Data
---------
-1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
+1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index 012b7b6..f2f4b43 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -42,10 +42,10 @@ Use quick-dev for testing out changes to core Metron services.
installed
### Full-Dev
-Navigate to `incubator-metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.
+Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`.
### Quick-Dev
-Navigate to `incubator-metron/metron-deployment/vagrant/quick-dev-platform` and run `vagrant up`.
+Navigate to `metron/metron-deployment/vagrant/quick-dev-platform` and run `vagrant up`.
## Ambari Management Pack
An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-deployment/packaging/docker/ansible-docker/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/ansible-docker/README.md b/metron-deployment/packaging/docker/ansible-docker/README.md
index 6ee7618..ebd594a 100644
--- a/metron-deployment/packaging/docker/ansible-docker/README.md
+++ b/metron-deployment/packaging/docker/ansible-docker/README.md
@@ -12,6 +12,6 @@ Full instructions are found on the wiki [https://cwiki.apache.org/confluence/pag
tl;dr
-1. docker run -it -v \<project-directory\>:/root/incubator-metron ansible-docker:2.0.0.2 bash
-2. cd /root/incubator-metron
+1. docker run -it -v \<project-directory\>:/root/metron ansible-docker:2.0.0.2 bash
+2. cd /root/metron
3. mvn clean package -DskipTests
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-interface/metron-config/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/README.md b/metron-interface/metron-config/README.md
index 92878b5..070668f 100644
--- a/metron-interface/metron-config/README.md
+++ b/metron-interface/metron-config/README.md
@@ -13,7 +13,7 @@ This module provides a user interface for management functions in Metron.
mvn clean package -DskipTests
```
-1. Copy `incubator-metron/metron-interface/metron-config/target/metron-config-METRON_VERSION-archive.tar.gz` to the desired host.
+1. Copy `metron/metron-interface/metron-config/target/metron-config-METRON_VERSION-archive.tar.gz` to the desired host.
1. Untar the archive in the target directory. The directory structure will look like:
```
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index 674f524..3309691 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -32,7 +32,7 @@ lib
### From Package Manager
-1. Deploy the RPM at `/incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
+1. Deploy the RPM at `/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
1. Install the RPM with:
```
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-interface/metron-rest/src/test/resources/README.vm
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/resources/README.vm b/metron-interface/metron-rest/src/test/resources/README.vm
index 2d8fd75..e6e6472 100644
--- a/metron-interface/metron-rest/src/test/resources/README.vm
+++ b/metron-interface/metron-rest/src/test/resources/README.vm
@@ -32,7 +32,7 @@ lib
#[[###]]# From Package Manager
-1. Deploy the RPM at `/incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
+1. Deploy the RPM at `/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
1. Install the RPM with:
```
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-interface/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/pom.xml b/metron-interface/pom.xml
index 58e066f..4f3edea 100644
--- a/metron-interface/pom.xml
+++ b/metron-interface/pom.xml
@@ -26,10 +26,10 @@
<description>Interfaces for Metron</description>
<url>https://metron.apache.org/</url>
<scm>
- <connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
- <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
+ <connection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</connection>
+ <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</developerConnection>
<tag>HEAD</tag>
- <url>https://git-wip-us.apache.org/repos/asf/incubator-metron</url>
+ <url>https://git-wip-us.apache.org/repos/asf/metron</url>
</scm>
<licenses>
<license>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
index 8c853a5..c18b5fb 100644
--- a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
+++ b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
@@ -1 +1 @@
-{"paragraphs":[{"text":"%md\n# Execute Packet Capture Queries\n\nSpecify the following to filter the packet capture query:\n* *end time* - The ending time of the query in yyyyMMdd format (e.g. 20170428)\n* *start time* - The starting time of the query in yyyyMMdd format (e.g. 20170428)\n* *query* - The [Stellar](https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-language) query (i.e. a Stellar expression that returns `true` or `false`) to specify the packets.\n\nThe available fields to use in the queries are as follows:\n* `ip_src_addr` - The source IP address of the packets filtered\n* `ip_src_port` - The source port of the packets filtered\n* `ip_dst_addr` - The destination IP address of the packets filtered\n* `ip_dst_port` - The destination port of the packets filtered\n* `packet` - The raw packet (for use with the `BYTEARRAY_MATCHER` function)\n\nYou can use any [Stellar function](https://github.com/apache/incubator-metron/tree/maste
r/metron-platform/metron-common#stellar-core-functions)\n\n## Simple Boolean Expressions\n\nFor example:\n* `ip_dst_port == 8080` would return all packets where the destination port is `8080`\n* `ip_dst_port in [ 8080, 80 ]` would return all packets where the destination port is either `8080` or `80`\n\n## Common Network Functions on Metadata\n\nFor example:\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24')` would return all packets whose destination conforms to the CIDR `192.168.0.0/24`\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080` would return all packets matching the CIDR and whose destination port is `8080`\n \n## Filtering based on the Packet Contents\n\nWe use byteseek regular expressions to filter packets. The syntax for these is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).\n* `BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` in them anywhere.\n* `ip_ds
t_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` and have a destination port of `8080`\n* `BYTEARRAY_MATCHER('ff(.){5}ff', packet)` would return all packets containing a binary regex with `0xff` followed by any 5 bytes and then `0xff`\n\n# The Output\nThe output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.","dateUpdated":"2017-05-01T17:36:27+0000","config":{"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"query":"ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\\\'/api/v1/\\\\\\'', packet)"},"forms":{}},"jobName":"paragraph_1493658709556_1010155640","id":"20170428-183346_777875025","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Execute Packet Capture Queries</h1>\n<p>Specify the fo
llowing to filter the packet capture query:</p>\n<ul>\n<li><em>end time</em> - The ending time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>start time</em> - The starting time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>query</em> - The <a href=\"https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-language\">Stellar</a> query (i.e. a Stellar expression that returns <code>true</code> or <code>false</code>) to specify the packets.</li>\n</ul>\n<p>The available fields to use in the queries are as follows:</p>\n<ul>\n<li><code>ip_src_addr</code> - The source IP address of the packets filtered</li>\n<li><code>ip_src_port</code> - The source port of the packets filtered</li>\n<li><code>ip_dst_addr</code> - The destination IP address of the packets filtered</li>\n<li><code>ip_dst_port</code> - The destination port of the packets filtered</li>\n<li><code>packet</code> - The raw packet (for use with the <code>BYTEARR
AY_MATCHER</code> function)</li>\n</ul>\n<p>You can use any <a href=\"https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-core-functions\">Stellar function</a></p>\n<h2>Simple Boolean Expressions</h2>\n<p>For example:</p>\n<ul>\n<li><code>ip_dst_port == 8080</code> would return all packets where the destination port is <code>8080</code></li>\n<li><code>ip_dst_port in [ 8080, 80 ]</code> would return all packets where the destination port is either <code>8080</code> or <code>80</code></li>\n</ul>\n<h2>Common Network Functions on Metadata</h2>\n<p>For example:</p>\n<ul>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24')</code> would return all packets whose destination conforms to the CIDR <code>192.168.0.0/24</code></li>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080</code> would return all packets matching the CIDR and whose destination port is <code>8080</code></li>\n</ul>\n<h2>Filtering based on the Pack
et Contents</h2>\n<p>We use byteseek regular expressions to filter packets. The syntax for these is described <a href=\"https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md\">here</a>.</p>\n<ul>\n<li><code>BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> in them anywhere.</li>\n<li><code>ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> and have a destination port of <code>8080</code></li>\n<li><code>BYTEARRAY_MATCHER('ff(.){5}ff', packet)</code> would return all packets containing a binary regex with <code>0xff</code> followed by any 5 bytes and then <code>0xff</code></li>\n</ul>\n<h1>The Output</h1>\n<p>The output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.</p>\n"},"dateCreated":"2017-05-01T17:11:49+0000","stat
us":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1724","dateFinished":"2017-05-01T17:36:26+0000","dateStarted":"2017-05-01T17:36:26+0000","focus":true},{"text":"%sh\nexport PCAP_ZEPPELIN_RUN=$(find /usr -name pcap_zeppelin_run.sh)\nexport RECORDS_PER_FILE=10000\nexport NUMBER_OF_REDUCERS=10\nexport DATE_FORMAT=\"yyyyMMdd\"\nexport PCAP_DATA_PATH=\"/apps/metron/pcap\"\n\n$PCAP_ZEPPELIN_RUN \"${query}\" \"${start time}\" \"${optional end time}\"","dateUpdated":"2017-05-01T17:35:04+0000","config":{"colWidth":12,"editorMode":"ace/mode/sh","editorHide":false,"graph":{"mode":"table","height":164,"optionOpen":false,"keys":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"values":[],"groups":[],"scatter":{"xAxis":{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}}},"e
nabled":true},"settings":{"params":{"start time":"20170428","end time":"","query":"BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet) && ip_dst_port == 8080","optional end time":""},"forms":{"optional end time":{"name":"optional end time","defaultValue":"","hidden":false},"query":{"name":"query","defaultValue":"","hidden":false},"start time":{"name":"start time","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493658709562_1009386142","id":"20170428-181957_829993114","result":{"code":"SUCCESS","type":"TABLE","msg":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now\n%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>\n","comment":"","msgTable":[[{"value":"%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578
%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>"}]],"columnNames":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"rows":[["%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>"]]},"dateCreated":"2017-05-01T17:11:49+0000","dateStarted":"2017-05-01T17:35:04+0000","dateFinished":"2017-05-01T17:34:55+0000","status":"RUNNING","progressUpdateIntervalMs":500,"$$hashKey":"object:1725","focus":true},{"dateUpdated":"2017-05-01T17:34:55+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493658709563_1009001393","id":"20170428-184335_
1604096389","dateCreated":"2017-05-01T17:11:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1726","dateFinished":"2017-05-01T17:34:54+0000","dateStarted":"2017-05-01T17:34:54+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Troubleshooting</h1>\n<p>If you are having problems with the above form, the following may help.</p>\n<h2>I see <code>Terminated by SIGINTERRUPT</code> or something similar in the output!</h2>\n<p>PCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\n<br />They can do this by changing the <code>shell.command.timeout.millisecs</code> property for the <code>sh</code> interpreter to a larger value, likely <code>100000</code>.</p>\n<h2>I do not see a table of URLs to pcap files in my output, what happened?</h2>\n<p>If an error happens, the log of the pcap querying utility will be displayed instead of an output. Please co
ntact an administrator with this output to debug further.</p>\n"},"text":"%md\n# Troubleshooting\n\nIf you are having problems with the above form, the following may help.\n\n## I see `Terminated by SIGINTERRUPT` or something similar in the output!\n\nPCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\nThey can do this by changing the `shell.command.timeout.millisecs` property for the `sh` interpreter to a larger value, likely `100000`.\n\n## I do not see a table of URLs to pcap files in my output, what happened?\n\nIf an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.","focus":true},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobN
ame":"paragraph_1493659778398_1082698332","id":"20170501-172938_213921861","dateCreated":"2017-05-01T17:29:38+0000","status":"READY","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:1972"}],"name":"metron/pcap","id":"2CEEXVR1W","angularObjects":{"2CFNGE6TP:shared_process":[],"2CGUX5TSW:shared_process":[],"2CETVR2AB:shared_process":[],"2CF163WFX:shared_process":[],"2CG4YXKUV:shared_process":[],"2CG6QDFF7:shared_process":[]},"config":{"looknfeel":"default"},"info":{}}
\ No newline at end of file
+{"paragraphs":[{"text":"%md\n# Execute Packet Capture Queries\n\nSpecify the following to filter the packet capture query:\n* *end time* - The ending time of the query in yyyyMMdd format (e.g. 20170428)\n* *start time* - The starting time of the query in yyyyMMdd format (e.g. 20170428)\n* *query* - The [Stellar](https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language) query (i.e. a Stellar expression that returns `true` or `false`) to specify the packets.\n\nThe available fields to use in the queries are as follows:\n* `ip_src_addr` - The source IP address of the packets filtered\n* `ip_src_port` - The source port of the packets filtered\n* `ip_dst_addr` - The destination IP address of the packets filtered\n* `ip_dst_port` - The destination port of the packets filtered\n* `packet` - The raw packet (for use with the `BYTEARRAY_MATCHER` function)\n\nYou can use any [Stellar function](https://github.com/apache/metron/tree/master/metron-platform/me
tron-common#stellar-core-functions)\n\n## Simple Boolean Expressions\n\nFor example:\n* `ip_dst_port == 8080` would return all packets where the destination port is `8080`\n* `ip_dst_port in [ 8080, 80 ]` would return all packets where the destination port is either `8080` or `80`\n\n## Common Network Functions on Metadata\n\nFor example:\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24')` would return all packets whose destination conforms to the CIDR `192.168.0.0/24`\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080` would return all packets matching the CIDR and whose destination port is `8080`\n \n## Filtering based on the Packet Contents\n\nWe use byteseek regular expressions to filter packets. The syntax for these is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).\n* `BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` in them anywhere.\n* `ip_dst_port==8080 && BYTE
ARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` and have a destination port of `8080`\n* `BYTEARRAY_MATCHER('ff(.){5}ff', packet)` would return all packets containing a binary regex with `0xff` followed by any 5 bytes and then `0xff`\n\n# The Output\nThe output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.","dateUpdated":"2017-05-01T17:36:27+0000","config":{"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"query":"ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\\\'/api/v1/\\\\\\'', packet)"},"forms":{}},"jobName":"paragraph_1493658709556_1010155640","id":"20170428-183346_777875025","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Execute Packet Capture Queries</h1>\n<p>Specify the following to filter th
e packet capture query:</p>\n<ul>\n<li><em>end time</em> - The ending time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>start time</em> - The starting time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>query</em> - The <a href=\"https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language\">Stellar</a> query (i.e. a Stellar expression that returns <code>true</code> or <code>false</code>) to specify the packets.</li>\n</ul>\n<p>The available fields to use in the queries are as follows:</p>\n<ul>\n<li><code>ip_src_addr</code> - The source IP address of the packets filtered</li>\n<li><code>ip_src_port</code> - The source port of the packets filtered</li>\n<li><code>ip_dst_addr</code> - The destination IP address of the packets filtered</li>\n<li><code>ip_dst_port</code> - The destination port of the packets filtered</li>\n<li><code>packet</code> - The raw packet (for use with the <code>BYTEARRAY_MATCHER</code> function)</l
i>\n</ul>\n<p>You can use any <a href=\"https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-core-functions\">Stellar function</a></p>\n<h2>Simple Boolean Expressions</h2>\n<p>For example:</p>\n<ul>\n<li><code>ip_dst_port == 8080</code> would return all packets where the destination port is <code>8080</code></li>\n<li><code>ip_dst_port in [ 8080, 80 ]</code> would return all packets where the destination port is either <code>8080</code> or <code>80</code></li>\n</ul>\n<h2>Common Network Functions on Metadata</h2>\n<p>For example:</p>\n<ul>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24')</code> would return all packets whose destination conforms to the CIDR <code>192.168.0.0/24</code></li>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080</code> would return all packets matching the CIDR and whose destination port is <code>8080</code></li>\n</ul>\n<h2>Filtering based on the Packet Contents</h2>\n<p>We use byteseek reg
ular expressions to filter packets. The syntax for these is described <a href=\"https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md\">here</a>.</p>\n<ul>\n<li><code>BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> in them anywhere.</li>\n<li><code>ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> and have a destination port of <code>8080</code></li>\n<li><code>BYTEARRAY_MATCHER('ff(.){5}ff', packet)</code> would return all packets containing a binary regex with <code>0xff</code> followed by any 5 bytes and then <code>0xff</code></li>\n</ul>\n<h1>The Output</h1>\n<p>The output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.</p>\n"},"dateCreated":"2017-05-01T17:11:49+0000","status":"FINISHED","progressUpdateIntervalMs
":500,"$$hashKey":"object:1724","dateFinished":"2017-05-01T17:36:26+0000","dateStarted":"2017-05-01T17:36:26+0000","focus":true},{"text":"%sh\nexport PCAP_ZEPPELIN_RUN=$(find /usr -name pcap_zeppelin_run.sh)\nexport RECORDS_PER_FILE=10000\nexport NUMBER_OF_REDUCERS=10\nexport DATE_FORMAT=\"yyyyMMdd\"\nexport PCAP_DATA_PATH=\"/apps/metron/pcap\"\n\n$PCAP_ZEPPELIN_RUN \"${query}\" \"${start time}\" \"${optional end time}\"","dateUpdated":"2017-05-01T17:35:04+0000","config":{"colWidth":12,"editorMode":"ace/mode/sh","editorHide":false,"graph":{"mode":"table","height":164,"optionOpen":false,"keys":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"values":[],"groups":[],"scatter":{"xAxis":{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}}},"enabled":true},"settings":{"params":{"sta
rt time":"20170428","end time":"","query":"BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet) && ip_dst_port == 8080","optional end time":""},"forms":{"optional end time":{"name":"optional end time","defaultValue":"","hidden":false},"query":{"name":"query","defaultValue":"","hidden":false},"start time":{"name":"start time","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493658709562_1009386142","id":"20170428-181957_829993114","result":{"code":"SUCCESS","type":"TABLE","msg":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now\n%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>\n","comment":"","msgTable":[[{"value":"%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-2017050
1173442578+0000.pcap</a>"}]],"columnNames":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"rows":[["%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>"]]},"dateCreated":"2017-05-01T17:11:49+0000","dateStarted":"2017-05-01T17:35:04+0000","dateFinished":"2017-05-01T17:34:55+0000","status":"RUNNING","progressUpdateIntervalMs":500,"$$hashKey":"object:1725","focus":true},{"dateUpdated":"2017-05-01T17:34:55+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493658709563_1009001393","id":"20170428-184335_1604096389","dateCreated":"2017-05-01T17
:11:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1726","dateFinished":"2017-05-01T17:34:54+0000","dateStarted":"2017-05-01T17:34:54+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Troubleshooting</h1>\n<p>If you are having problems with the above form, the following may help.</p>\n<h2>I see <code>Terminated by SIGINTERRUPT</code> or something similar in the output!</h2>\n<p>PCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\n<br />They can do this by changing the <code>shell.command.timeout.millisecs</code> property for the <code>sh</code> interpreter to a larger value, likely <code>100000</code>.</p>\n<h2>I do not see a table of URLs to pcap files in my output, what happened?</h2>\n<p>If an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output
to debug further.</p>\n"},"text":"%md\n# Troubleshooting\n\nIf you are having problems with the above form, the following may help.\n\n## I see `Terminated by SIGINTERRUPT` or something similar in the output!\n\nPCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\nThey can do this by changing the `shell.command.timeout.millisecs` property for the `sh` interpreter to a larger value, likely `100000`.\n\n## I do not see a table of URLs to pcap files in my output, what happened?\n\nIf an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.","focus":true},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493659778398_1082698332
","id":"20170501-172938_213921861","dateCreated":"2017-05-01T17:29:38+0000","status":"READY","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:1972"}],"name":"metron/pcap","id":"2CEEXVR1W","angularObjects":{"2CFNGE6TP:shared_process":[],"2CGUX5TSW:shared_process":[],"2CETVR2AB:shared_process":[],"2CF163WFX:shared_process":[],"2CG4YXKUV:shared_process":[],"2CG6QDFF7:shared_process":[]},"config":{"looknfeel":"default"},"info":{}}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-platform/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/pom.xml b/metron-platform/pom.xml
index cc48851..718c7be 100644
--- a/metron-platform/pom.xml
+++ b/metron-platform/pom.xml
@@ -27,10 +27,10 @@
<description>Stream analytics for Metron</description>
<url>https://metron.apache.org/</url>
<scm>
- <connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
- <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
+ <connection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</connection>
+ <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</developerConnection>
<tag>HEAD</tag>
- <url>https://git-wip-us.apache.org/repos/asf/incubator-metron</url>
+ <url>https://git-wip-us.apache.org/repos/asf/metron</url>
</scm>
<licenses>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-sensors/fastcapa/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/fastcapa/README.md b/metron-sensors/fastcapa/README.md
index 5efdb3f..707b491 100644
--- a/metron-sensors/fastcapa/README.md
+++ b/metron-sensors/fastcapa/README.md
@@ -181,7 +181,7 @@ The probe has been tested with [Librdkafka 0.9.4](https://github.com/edenhill/li
2. Build Fastcapa. The resulting binary will be placed at `build/app/fastcapa`.
```
- cd incubator-metron/metron-sensors/fastcapa
+ cd metron/metron-sensors/fastcapa
make
```
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/metron-sensors/pycapa/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md
index 1e8c2c5..6f41d63 100644
--- a/metron-sensors/pycapa/README.md
+++ b/metron-sensors/pycapa/README.md
@@ -12,7 +12,7 @@ Pycapa
Overview
========
-Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of [Apache Metron](https://github.com/apache/incubator-metron). It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat) or even to feed directly into a tool like Wireshark to monitor ongoing activity.
+Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of [Apache Metron](https://github.com/apache/metron). It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat) or even to feed directly into a tool like Wireshark to monitor ongoing activity.
Installation
============
@@ -55,7 +55,7 @@ General notes on the installation of Pycapa.
1. Install Pycapa. This assumes that you already have the Metron source code on the host.
```
- cd incubator-metron/metron-sensors/pycapa
+ cd metron/metron-sensors/pycapa
pip install -r requirements.txt
python setup.py install
```
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site-book/README.md
----------------------------------------------------------------------
diff --git a/site-book/README.md b/site-book/README.md
index bf579de..52aae51 100644
--- a/site-book/README.md
+++ b/site-book/README.md
@@ -12,7 +12,7 @@ The site-book is also part of the Maven site lifecycle, and will be included by
To build only the book, do the following:
-In any git clone of incubator-metron containing the site-book subdirectory,
+In any git clone of metron containing the site-book subdirectory,
```
cd site-book
@@ -22,7 +22,7 @@ mvn site
It only takes a few seconds. You may now view your copy of the book in a browser by opening
```
-file:///your/path/to/incubator-metron/site-book/target/site/index.html
+file:///your/path/to/metron/site-book/target/site/index.html
```
On a Mac, you can just type the following on the command line
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site-book/pom.xml
----------------------------------------------------------------------
diff --git a/site-book/pom.xml b/site-book/pom.xml
index 71e4879..a6e4873 100644
--- a/site-book/pom.xml
+++ b/site-book/pom.xml
@@ -27,10 +27,10 @@
<description>User Documentation for Metron</description>
<url>https://metron.apache.org/</url>
<scm>
- <connection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</connection>
- <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-metron.git</developerConnection>
+ <connection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</connection>
+ <developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/metron.git</developerConnection>
<tag>HEAD</tag>
- <url>https://git-wip-us.apache.org/repos/asf/incubator-metron</url>
+ <url>https://git-wip-us.apache.org/repos/asf/metron</url>
</scm>
<licenses>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/BUILD.md
----------------------------------------------------------------------
diff --git a/site/BUILD.md b/site/BUILD.md
index 20c0f5e..6475d55 100644
--- a/site/BUILD.md
+++ b/site/BUILD.md
@@ -6,7 +6,7 @@ This directory contains the code for the Apache Metron web site,
## Setup
1. `cd site`
-2. `git clone https://git-wip-us.apache.org/repos/asf/incubator-metron.git -b asf-site target`
+2. `git clone https://git-wip-us.apache.org/repos/asf/metron.git -b asf-site target`
3. `sudo gem install bundler`
4. `sudo gem install github-pages jekyll`
4. `bundle install`
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/README.md
----------------------------------------------------------------------
diff --git a/site/README.md b/site/README.md
index fd2d8cc..2d3d943 100644
--- a/site/README.md
+++ b/site/README.md
@@ -4,4 +4,4 @@ This branch contains the generated HTML for the
[Metron website](https://metron.apache.org).
You are probably looking for the
-[Metron master branch](https://github.com/apache/incubator-metron/tree/master).
+[Metron master branch](https://github.com/apache/metron/tree/master).
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/_config.yml
----------------------------------------------------------------------
diff --git a/site/_config.yml b/site/_config.yml
index cfc3054..0ddd560 100644
--- a/site/_config.yml
+++ b/site/_config.yml
@@ -3,7 +3,7 @@ highlighter: rouge
permalink: /news/:year/:month/:day/:title/
excerpt_separator: ""
-repository: https://github.com/apache/incubator-metron
+repository: https://github.com/apache/metron
destination: target
exclude: [BUILD.md, Gemfile*]
keep_files: [.git]
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/community/index.md
----------------------------------------------------------------------
diff --git a/site/community/index.md b/site/community/index.md
index ac61708..5797ef2 100644
--- a/site/community/index.md
+++ b/site/community/index.md
@@ -117,11 +117,11 @@ title: Apache Metron Community
<h4> General & Public Discussion </h4>
[<a href="mailto:user-subscribe@metron.apache.org">Subscribe</a>]
[<a href="mailto:user-unsubscribe@metron.apache.org">Unsubscribe</a>]
- [<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-user/">Archives</a>]
+ [<a href="http://mail-archives.apache.org/mod_mbox/metron-user/">Archives</a>]
<h4> Code & Documentation Change </h4>
[<a href="mailto:dev-subscribe@metron.apache.org">Subscribe</a>]
[<a href="mailto:dev-unsubscribe@metron.apache.org">Unsubscribe</a>]
- [<a href="http://mail-archives.apache.org/mod_mbox/incubator-metron-dev/">Archives</a>]
+ [<a href="http://mail-archives.apache.org/mod_mbox/metron-dev/">Archives</a>]
</div>
</section>
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/develop/index.md
----------------------------------------------------------------------
diff --git a/site/develop/index.md b/site/develop/index.md
index ca917fa..7db04fa 100644
--- a/site/develop/index.md
+++ b/site/develop/index.md
@@ -30,11 +30,11 @@ George Vetticaden | gvetticaden | PMC
There are several development mailing lists for Metron
* [dev@metron.apache.org](mailto:dev@metron.apache.org) - Development discussions
- with archive [here](https://mail-archives.apache.org/mod_mbox/incubator-metron-dev/)
+ with archive [here](https://mail-archives.apache.org/mod_mbox/metron-dev/)
* [issues@metron.apache.org](mailto:issues@metron.apache.org) - Bug tracking
- with archive [here](https://mail-archives.apache.org/mod_mbox/incubator-metron-issues/)
+ with archive [here](https://mail-archives.apache.org/mod_mbox/metron-issues/)
* [commits@metron.apache.org](mailto:commits@metron.apache.org) - Git tracking
- with archive [here](https://mail-archives.apache.org/mod_mbox/incubator-metron-commits/)
+ with archive [here](https://mail-archives.apache.org/mod_mbox/metron-commits/)
You can subscribe to the lists by sending email to
*list*-subscribe@metron.apache.org and unsubscribe by sending email to
@@ -44,13 +44,13 @@ You can subscribe to the lists by sending email to
Metron uses git for version control. Get the source code:
-`% git clone https://git-wip-us.apache.org/repos/asf/incubator-metron.git`
+`% git clone https://git-wip-us.apache.org/repos/asf/metron.git`
The important branches are:
-* [master](https://github.com/apache/incubator-metron/tree/master) -
+* [master](https://github.com/apache/metron/tree/master) -
The trunk for all developement
-* [asf-site](https://github.com/apache/incubator-metron/tree/asf-site) -
+* [asf-site](https://github.com/apache/metron/tree/asf-site) -
The pages that are deployed to https://metron.apache.org/
Please check our [coding guidelines](/develop/coding.html).
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/help/index.md
----------------------------------------------------------------------
diff --git a/site/help/index.md b/site/help/index.md
index c91f4e1..00a3f4c 100644
--- a/site/help/index.md
+++ b/site/help/index.md
@@ -10,7 +10,7 @@ Need help with Metron? Try these resources.
The best option is to send email to the user list
[user@metron.apache.org](mailto:user@metron.apache.org), which is for user
discussions. All of the historic traffic is available in the
-[archive](http://mail-archives.apache.org/mod_mbox/incubator-metron-user/). To
+[archive](http://mail-archives.apache.org/mod_mbox/metron-user/). To
subscribe to the user list, please send email to
[user-subscribe@metron.apache.org](mailto:user-subscribe@metron.apache.org).
@@ -29,4 +29,4 @@ your question.
## Browse the code
One of the advantages of open source software is that you can browse the code.
-The code is available on [github](https://github.com/apache/incubator-metron/tree/master).
+The code is available on [github](https://github.com/apache/metron/tree/master).
http://git-wip-us.apache.org/repos/asf/metron/blob/ae05f38d/site/index.html
----------------------------------------------------------------------
diff --git a/site/index.html b/site/index.html
index 4800e4d..0157ea7 100644
--- a/site/index.html
+++ b/site/index.html
@@ -12,7 +12,7 @@ overview: true
<div class="v-middle-inner">
<div class="v-middle">
<h1>real-time big data security </h1>
- <a href="https://github.com/apache/incubator-metron" target="_blank"><button type="button" class="button-default">GitHub</button></a>
+ <a href="https://github.com/apache/metron" target="_blank"><button type="button" class="button-default">GitHub</button></a>
<a href="https://cwiki.apache.org/confluence/display/METRON/Community+Resources" target="_blank"><button type="button" class="button-default">Community Home</button></a>
[28/44] metron git commit: METRON-891 KafkaConsumer should not be
shared among threads (jjmeyer via merrimanr) closes apache/metron#567
Posted by ma...@apache.org.
METRON-891 KafkaConsumer should not be shared among threads (jjmeyer via merrimanr) closes apache/metron#567
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/47e2b735
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/47e2b735
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/47e2b735
Branch: refs/heads/Metron_0.4.0
Commit: 47e2b735ea682e7d647e3eaafac89a192a783e86
Parents: c0b0825
Author: jjmeyer <jj...@gmail.com>
Authored: Fri May 19 10:10:25 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Fri May 19 10:10:25 2017 -0500
----------------------------------------------------------------------
dependencies_with_url.csv | 1 +
metron-interface/metron-rest/pom.xml | 34 +++-
.../apache/metron/rest/config/KafkaConfig.java | 52 ++++++-
.../metron/rest/controller/KafkaController.java | 104 +++++++------
.../metron/rest/service/KafkaService.java | 45 +++++-
.../rest/service/impl/KafkaServiceImpl.java | 155 +++++++++++--------
.../apache/metron/rest/config/TestConfig.java | 32 ++--
.../rest/service/impl/KafkaServiceImplTest.java | 15 +-
8 files changed, 297 insertions(+), 141 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/dependencies_with_url.csv
----------------------------------------------------------------------
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index 93a19b7..3752366 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -296,3 +296,4 @@ org.htrace:htrace-core:jar:3.0.4:compile,ASLv2,http://htrace.incubator.apache.or
net.byteseek:byteseek:jar:2.0.3:compile,BSD,https://github.com/nishihatapalmer/byteseek
org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos
org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos
+org.springframework.kafka:spring-kafka:jar:1.1.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-kafka
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/pom.xml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/pom.xml b/metron-interface/metron-rest/pom.xml
index b11e999..1c3ff92 100644
--- a/metron-interface/metron-rest/pom.xml
+++ b/metron-interface/metron-rest/pom.xml
@@ -33,9 +33,37 @@
<spring.kerberos.version>1.0.1.RELEASE</spring.kerberos.version>
<swagger.version>2.5.0</swagger.version>
<mysql.client.version>5.1.40</mysql.client.version>
+ <spring-kafka.version>1.1.1.RELEASE</spring-kafka.version>
</properties>
<dependencies>
<dependency>
+ <groupId>org.springframework.kafka</groupId>
+ <artifactId>spring-kafka</artifactId>
+ <version>${spring-kafka.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka-clients</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.springframework.retry</groupId>
+ <artifactId>spring-retry</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.springframework</groupId>
+ <artifactId>spring-messaging</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
@@ -75,7 +103,7 @@
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
- </exclusions>
+ </exclusions>
</dependency>
<dependency>
<groupId>com.googlecode.json-simple</groupId>
@@ -112,8 +140,8 @@
<artifactId>jackson-databind</artifactId>
</exclusion>
<exclusion>
- <groupId>org.reflections</groupId>
- <artifactId>reflections</artifactId>
+ <groupId>org.reflections</groupId>
+ <artifactId>reflections</artifactId>
</exclusion>
</exclusions>
</dependency>
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
index 309a549..247264b 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java
@@ -27,33 +27,56 @@ import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
import org.springframework.core.env.Environment;
+import org.springframework.kafka.core.ConsumerFactory;
+import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
-import java.util.Properties;
+import java.util.HashMap;
+import java.util.Map;
import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
+/**
+ * Configuration used for connecting to Kafka.
+ */
@Configuration
@Profile("!" + TEST_PROFILE)
public class KafkaConfig {
-
+ /**
+ * The Spring environment.
+ */
private Environment environment;
+ /**
+ * Construvtor used to inject {@link Environment}.
+ * @param environment Spring environment to inject.
+ */
@Autowired
- public KafkaConfig(Environment environment) {
+ public KafkaConfig(final Environment environment) {
this.environment = environment;
}
+ /**
+ * The client used for ZooKeeper.
+ */
@Autowired
private ZkClient zkClient;
+ /**
+ * Bean for ZooKeeper
+ */
@Bean
public ZkUtils zkUtils() {
return ZkUtils.apply(zkClient, false);
}
- @Bean(destroyMethod = "close")
- public KafkaConsumer<String, String> kafkaConsumer() {
- Properties props = new Properties();
+ /**
+ * Create properties that will be used by {@link this#createConsumerFactory()}
+ *
+ * @return Configurations used by {@link this#createConsumerFactory()}.
+ */
+ @Bean
+ public Map<String, Object> consumerProperties() {
+ final Map<String, Object> props = new HashMap<>();
props.put("bootstrap.servers", environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY));
props.put("group.id", "metron-rest");
props.put("enable.auto.commit", "false");
@@ -64,9 +87,24 @@ public class KafkaConfig {
if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) {
props.put("security.protocol", "SASL_PLAINTEXT");
}
- return new KafkaConsumer<>(props);
+ return props;
+ }
+
+ /**
+ * Create a {@link ConsumerFactory} which will be used for certain Kafka interactions within config API.
+ *
+ * @return a {@link ConsumerFactory} used to create {@link KafkaConsumer} for interactions with Kafka.
+ */
+ @Bean
+ public ConsumerFactory<String, String> createConsumerFactory() {
+ return new DefaultKafkaConsumerFactory<>(consumerProperties());
}
+ /**
+ * Create a bean for {@link AdminUtils$}. This is primarily done to make testing a bit easier.
+ *
+ * @return {@link AdminUtils$} is written in scala. We return a reference to this class.
+ */
@Bean
public AdminUtils$ adminUtils() {
return AdminUtils$.MODULE$;
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/KafkaController.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/KafkaController.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/KafkaController.java
index 0cd4d54..2787504 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/KafkaController.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/KafkaController.java
@@ -35,62 +35,78 @@ import org.springframework.web.bind.annotation.RestController;
import java.util.Set;
+/**
+ * The API resource that is use to interact with Kafka.
+ */
@RestController
@RequestMapping("/api/v1/kafka")
public class KafkaController {
- @Autowired
- private KafkaService kafkaService;
+ /**
+ * Service used to interact with Kafka.
+ */
+ @Autowired
+ private KafkaService kafkaService;
- @ApiOperation(value = "Creates a new Kafka topic")
+ @ApiOperation(value = "Creates a new Kafka topic")
+ @ApiResponses({
@ApiResponse(message = "Returns saved Kafka topic", code = 200)
- @RequestMapping(value = "/topic", method = RequestMethod.POST)
- ResponseEntity<KafkaTopic> save(@ApiParam(name="topic", value="Kafka topic", required=true)@RequestBody KafkaTopic topic) throws RestException {
- return new ResponseEntity<>(kafkaService.createTopic(topic), HttpStatus.CREATED);
- }
+ })
+ @RequestMapping(value = "/topic", method = RequestMethod.POST)
+ ResponseEntity<KafkaTopic> save(final @ApiParam(name = "topic", value = "Kafka topic", required = true) @RequestBody KafkaTopic topic) throws RestException {
+ return new ResponseEntity<>(kafkaService.createTopic(topic), HttpStatus.CREATED);
+ }
- @ApiOperation(value = "Retrieves a Kafka topic")
- @ApiResponses(value = { @ApiResponse(message = "Returns Kafka topic", code = 200),
- @ApiResponse(message = "Kafka topic is missing", code = 404) })
- @RequestMapping(value = "/topic/{name}", method = RequestMethod.GET)
- ResponseEntity<KafkaTopic> get(@ApiParam(name="name", value="Kafka topic name", required=true)@PathVariable String name) throws RestException {
- KafkaTopic kafkaTopic = kafkaService.getTopic(name);
- if (kafkaTopic != null) {
- return new ResponseEntity<>(kafkaTopic, HttpStatus.OK);
- } else {
- return new ResponseEntity<>(HttpStatus.NOT_FOUND);
- }
+ @ApiOperation(value = "Retrieves a Kafka topic")
+ @ApiResponses(value = {
+ @ApiResponse(message = "Returns Kafka topic", code = 200),
+ @ApiResponse(message = "Kafka topic is missing", code = 404)
+ })
+ @RequestMapping(value = "/topic/{name}", method = RequestMethod.GET)
+ ResponseEntity<KafkaTopic> get(final @ApiParam(name = "name", value = "Kafka topic name", required = true) @PathVariable String name) throws RestException {
+ KafkaTopic kafkaTopic = kafkaService.getTopic(name);
+ if (kafkaTopic != null) {
+ return new ResponseEntity<>(kafkaTopic, HttpStatus.OK);
+ } else {
+ return new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
+ }
- @ApiOperation(value = "Retrieves all Kafka topics")
+ @ApiOperation(value = "Retrieves all Kafka topics")
+ @ApiResponses({
@ApiResponse(message = "Returns a list of all Kafka topics", code = 200)
- @RequestMapping(value = "/topic", method = RequestMethod.GET)
- ResponseEntity<Set<String>> list() throws Exception {
- return new ResponseEntity<>(kafkaService.listTopics(), HttpStatus.OK);
- }
+ })
+ @RequestMapping(value = "/topic", method = RequestMethod.GET)
+ ResponseEntity<Set<String>> list() throws Exception {
+ return new ResponseEntity<>(kafkaService.listTopics(), HttpStatus.OK);
+ }
- @ApiOperation(value = "Deletes a Kafka topic")
- @ApiResponses(value = { @ApiResponse(message = "Kafka topic was deleted", code = 200),
- @ApiResponse(message = "Kafka topic is missing", code = 404) })
- @RequestMapping(value = "/topic/{name}", method = RequestMethod.DELETE)
- ResponseEntity<Void> delete(@ApiParam(name="name", value="Kafka topic name", required=true)@PathVariable String name) throws RestException {
- if (kafkaService.deleteTopic(name)) {
- return new ResponseEntity<>(HttpStatus.OK);
- } else {
- return new ResponseEntity<>(HttpStatus.NOT_FOUND);
- }
+ @ApiOperation(value = "Deletes a Kafka topic")
+ @ApiResponses(value = {
+ @ApiResponse(message = "Kafka topic was deleted", code = 200),
+ @ApiResponse(message = "Kafka topic is missing", code = 404)
+ })
+ @RequestMapping(value = "/topic/{name}", method = RequestMethod.DELETE)
+ ResponseEntity<Void> delete(final @ApiParam(name = "name", value = "Kafka topic name", required = true) @PathVariable String name) throws RestException {
+ if (kafkaService.deleteTopic(name)) {
+ return new ResponseEntity<>(HttpStatus.OK);
+ } else {
+ return new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
+ }
- @ApiOperation(value = "Retrieves a sample message from a Kafka topic using the most recent offset")
- @ApiResponses(value = { @ApiResponse(message = "Returns sample message", code = 200),
- @ApiResponse(message = "Either Kafka topic is missing or contains no messages", code = 404) })
- @RequestMapping(value = "/topic/{name}/sample", method = RequestMethod.GET)
- ResponseEntity<String> getSample(@ApiParam(name="name", value="Kafka topic name", required=true)@PathVariable String name) throws RestException {
- String sampleMessage = kafkaService.getSampleMessage(name);
- if (sampleMessage != null) {
- return new ResponseEntity<>(sampleMessage, HttpStatus.OK);
- } else {
- return new ResponseEntity<>(HttpStatus.NOT_FOUND);
- }
+ @ApiOperation(value = "Retrieves a sample message from a Kafka topic using the most recent offset")
+ @ApiResponses(value = {
+ @ApiResponse(message = "Returns sample message", code = 200),
+ @ApiResponse(message = "Either Kafka topic is missing or contains no messages", code = 404)
+ })
+ @RequestMapping(value = "/topic/{name}/sample", method = RequestMethod.GET)
+ ResponseEntity<String> getSample(final @ApiParam(name = "name", value = "Kafka topic name", required = true) @PathVariable String name) throws RestException {
+ String sampleMessage = kafkaService.getSampleMessage(name);
+ if (sampleMessage != null) {
+ return new ResponseEntity<>(sampleMessage, HttpStatus.OK);
+ } else {
+ return new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/KafkaService.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/KafkaService.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/KafkaService.java
index f3cd901..bee00f2 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/KafkaService.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/KafkaService.java
@@ -22,16 +22,49 @@ import org.apache.metron.rest.model.KafkaTopic;
import java.util.Set;
+/**
+ * This is a set of operations created to interact with Kafka.
+ */
public interface KafkaService {
- String CONSUMER_OFFSETS_TOPIC = "__consumer_offsets";
+ /**
+ * Please see the following for documentation.
+ *
+ * @see <a href="https://kafka.apache.org/documentation/#impl_offsettracking">Kafka offset tracking documentation</a>.
+ */
+ String CONSUMER_OFFSETS_TOPIC = "__consumer_offsets";
- KafkaTopic createTopic(KafkaTopic topic) throws RestException;
+ /**
+ * Create a topic in Kafka for given information.
+ * @param topic The information used to create a Kafka topic.
+ * @return The Kafka topic created.
+ * @throws RestException If exceptions occur when creating a topic they should be wrapped in a {@link RestException}.
+ */
+ KafkaTopic createTopic(KafkaTopic topic) throws RestException;
- boolean deleteTopic(String name);
+ /**
+ * Delete a topic for a given name.
+ * @param name The name of the topic to delete.
+ * @return If topic was deleted true; otherwise false.
+ */
+ boolean deleteTopic(String name);
- KafkaTopic getTopic(String name);
+ /**
+ * Retrieves the Kafka topic for a given name.
+ * @param name The name of the Kafka topic to retrieve.
+ * @return A {@link KafkaTopic} with the name of {@code name}. Null if topic with name, {@code name}, doesn't exist.
+ */
+ KafkaTopic getTopic(String name);
- Set<String> listTopics();
+ /**
+ * Returns a set of all topics.
+ * @return A set of all topics in Kafka.
+ */
+ Set<String> listTopics();
- String getSampleMessage(String topic);
+ /**
+ * Return a single sample message from a given topic.
+ * @param topic The name of the topic to retrieve a sample message from.
+ * @return A string representation of the sample message retrieved. If topic doesn't exist null will be returned.
+ */
+ String getSampleMessage(String topic);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/KafkaServiceImpl.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/KafkaServiceImpl.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/KafkaServiceImpl.java
index 33cb2e3..61e2618 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/KafkaServiceImpl.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/KafkaServiceImpl.java
@@ -20,8 +20,8 @@ package org.apache.metron.rest.service.impl;
import kafka.admin.AdminOperationException;
import kafka.admin.AdminUtils$;
import kafka.admin.RackAwareMode;
-import kafka.admin.RackAwareMode$;
import kafka.utils.ZkUtils;
+import org.apache.kafka.clients.consumer.Consumer;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.PartitionInfo;
@@ -30,6 +30,7 @@ import org.apache.metron.rest.RestException;
import org.apache.metron.rest.model.KafkaTopic;
import org.apache.metron.rest.service.KafkaService;
import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.stereotype.Service;
import java.util.HashSet;
@@ -38,88 +39,106 @@ import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
+/**
+ * The default service layer implementation of {@link KafkaService}.
+ *
+ * @see KafkaService
+ */
@Service
public class KafkaServiceImpl implements KafkaService {
- private ZkUtils zkUtils;
- private KafkaConsumer<String, String> kafkaConsumer;
- private AdminUtils$ adminUtils;
+ /**
+ * The timeout used when polling Kafka.
+ */
+ private static final int KAFKA_CONSUMER_TIMEOUT = 100;
- @Autowired
- public KafkaServiceImpl(ZkUtils zkUtils, KafkaConsumer<String, String> kafkaConsumer, AdminUtils$ adminUtils) {
- this.zkUtils = zkUtils;
- this.kafkaConsumer = kafkaConsumer;
- this.adminUtils = adminUtils;
- }
+ private final ZkUtils zkUtils;
+ private final ConsumerFactory<String, String> kafkaConsumerFactory;
+ private final AdminUtils$ adminUtils;
- @Override
- public KafkaTopic createTopic(KafkaTopic topic) throws RestException {
- if (!listTopics().contains(topic.getName())) {
- try {
- adminUtils.createTopic(zkUtils, topic.getName(), topic.getNumPartitions(), topic.getReplicationFactor(), topic.getProperties(),RackAwareMode.Disabled$.MODULE$ );
- } catch (AdminOperationException e) {
- throw new RestException(e);
- }
- }
- return topic;
+ /**
+ * @param zkUtils A utility class used to interact with ZooKeeper.
+ * @param kafkaConsumerFactory A class used to create {@link KafkaConsumer} in order to interact with Kafka.
+ * @param adminUtils A utility class used to do administration operations on Kafka.
+ */
+ @Autowired
+ public KafkaServiceImpl(final ZkUtils zkUtils,
+ final ConsumerFactory<String, String> kafkaConsumerFactory,
+ final AdminUtils$ adminUtils) {
+ this.zkUtils = zkUtils;
+ this.kafkaConsumerFactory = kafkaConsumerFactory;
+ this.adminUtils = adminUtils;
+ }
+
+ @Override
+ public KafkaTopic createTopic(final KafkaTopic topic) throws RestException {
+ if (!listTopics().contains(topic.getName())) {
+ try {
+ adminUtils.createTopic(zkUtils, topic.getName(), topic.getNumPartitions(), topic.getReplicationFactor(), topic.getProperties(), RackAwareMode.Disabled$.MODULE$);
+ } catch (AdminOperationException e) {
+ throw new RestException(e);
+ }
}
+ return topic;
+ }
- @Override
- public boolean deleteTopic(String name) {
- Set<String> topics = listTopics();
- if (topics != null && topics.contains(name)) {
- adminUtils.deleteTopic(zkUtils, name);
- return true;
- } else {
- return false;
- }
+ @Override
+ public boolean deleteTopic(final String name) {
+ final Set<String> topics = listTopics();
+ if (topics != null && topics.contains(name)) {
+ adminUtils.deleteTopic(zkUtils, name);
+ return true;
+ } else {
+ return false;
}
+ }
- @Override
- public KafkaTopic getTopic(String name) {
- KafkaTopic kafkaTopic = null;
- if (listTopics().contains(name)) {
- List<PartitionInfo> partitionInfos = kafkaConsumer.partitionsFor(name);
- if (partitionInfos.size() > 0) {
- PartitionInfo partitionInfo = partitionInfos.get(0);
- kafkaTopic = new KafkaTopic();
- kafkaTopic.setName(name);
- kafkaTopic.setNumPartitions(partitionInfos.size());
- kafkaTopic.setReplicationFactor(partitionInfo.replicas().length);
- }
+ @Override
+ public KafkaTopic getTopic(final String name) {
+ KafkaTopic kafkaTopic = null;
+ if (listTopics().contains(name)) {
+ try (Consumer<String, String> consumer = kafkaConsumerFactory.createConsumer()) {
+ final List<PartitionInfo> partitionInfos = consumer.partitionsFor(name);
+ if (partitionInfos.size() > 0) {
+ final PartitionInfo partitionInfo = partitionInfos.get(0);
+ kafkaTopic = new KafkaTopic();
+ kafkaTopic.setName(name);
+ kafkaTopic.setNumPartitions(partitionInfos.size());
+ kafkaTopic.setReplicationFactor(partitionInfo.replicas().length);
}
- return kafkaTopic;
+ }
}
+ return kafkaTopic;
+ }
- @Override
- public Set<String> listTopics() {
- Set<String> topics;
- synchronized (this) {
- Map<String, List<PartitionInfo>> topicsInfo = kafkaConsumer.listTopics();
- topics = topicsInfo == null ? new HashSet<>() : topicsInfo.keySet();
- topics.remove(CONSUMER_OFFSETS_TOPIC);
- }
- return topics;
+ @Override
+ public Set<String> listTopics() {
+ try (Consumer<String, String> consumer = kafkaConsumerFactory.createConsumer()) {
+ final Map<String, List<PartitionInfo>> topicsInfo = consumer.listTopics();
+ final Set<String> topics = topicsInfo == null ? new HashSet<>() : topicsInfo.keySet();
+ topics.remove(CONSUMER_OFFSETS_TOPIC);
+ return topics;
}
+ }
- @Override
- public String getSampleMessage(String topic) {
- String message = null;
- if (listTopics().contains(topic)) {
- synchronized (this) {
- kafkaConsumer.assign(kafkaConsumer.partitionsFor(topic).stream()
- .map(partitionInfo -> new TopicPartition(topic, partitionInfo.partition()))
- .collect(Collectors.toList()));
+ @Override
+ public String getSampleMessage(final String topic) {
+ String message = null;
+ if (listTopics().contains(topic)) {
+ try (Consumer<String, String> kafkaConsumer = kafkaConsumerFactory.createConsumer()) {
+ kafkaConsumer.assign(kafkaConsumer.partitionsFor(topic).stream()
+ .map(partitionInfo -> new TopicPartition(topic, partitionInfo.partition()))
+ .collect(Collectors.toList()));
- kafkaConsumer.assignment().stream()
- .filter(p -> (kafkaConsumer.position(p) -1) >= 0)
- .forEach(p -> kafkaConsumer.seek(p, kafkaConsumer.position(p) - 1));
+ kafkaConsumer.assignment().stream()
+ .filter(p -> (kafkaConsumer.position(p) - 1) >= 0)
+ .forEach(p -> kafkaConsumer.seek(p, kafkaConsumer.position(p) - 1));
- ConsumerRecords<String, String> records = kafkaConsumer.poll(100);
- message = records.isEmpty() ? null : records.iterator().next().value();
- kafkaConsumer.unsubscribe();
- }
- }
- return message;
+ final ConsumerRecords<String, String> records = kafkaConsumer.poll(KAFKA_CONSUMER_TIMEOUT);
+ message = records.isEmpty() ? null : records.iterator().next().value();
+ kafkaConsumer.unsubscribe();
+ }
}
+ return message;
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/TestConfig.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/TestConfig.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/TestConfig.java
index edfd542..adfe056 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/TestConfig.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/TestConfig.java
@@ -25,7 +25,6 @@ import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
-import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.metron.integration.ComponentRunner;
import org.apache.metron.integration.UnableToStartException;
import org.apache.metron.integration.components.KafkaComponent;
@@ -36,8 +35,12 @@ import org.apache.metron.rest.service.impl.StormCLIWrapper;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Profile;
+import org.springframework.kafka.core.ConsumerFactory;
+import org.springframework.kafka.core.DefaultKafkaConsumerFactory;
import org.springframework.web.client.RestTemplate;
+import java.util.HashMap;
+import java.util.Map;
import java.util.Properties;
import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE;
@@ -54,7 +57,7 @@ public class TestConfig {
@Bean
public ZKServerComponent zkServerComponent(Properties zkProperties) {
return new ZKServerComponent()
- .withPostStartCallback((zkComponent) -> zkProperties.setProperty(ZKServerComponent.ZOOKEEPER_PROPERTY, zkComponent.getConnectionString()));
+ .withPostStartCallback((zkComponent) -> zkProperties.setProperty(ZKServerComponent.ZOOKEEPER_PROPERTY, zkComponent.getConnectionString()));
}
@Bean
@@ -66,10 +69,10 @@ public class TestConfig {
@Bean
public ComponentRunner componentRunner(ZKServerComponent zkServerComponent, KafkaComponent kafkaWithZKComponent) {
ComponentRunner runner = new ComponentRunner.Builder()
- .withComponent("zk", zkServerComponent)
- .withComponent("kafka", kafkaWithZKComponent)
- .withCustomShutdownOrder(new String[] {"kafka","zk"})
- .build();
+ .withComponent("zk", zkServerComponent)
+ .withComponent("kafka", kafkaWithZKComponent)
+ .withCustomShutdownOrder(new String[]{"kafka", "zk"})
+ .build();
try {
runner.start();
} catch (UnableToStartException e) {
@@ -78,14 +81,14 @@ public class TestConfig {
return runner;
}
- @Bean(initMethod = "start", destroyMethod="close")
+ @Bean(initMethod = "start", destroyMethod = "close")
public CuratorFramework client(ComponentRunner componentRunner) {
RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
ZKServerComponent zkServerComponent = componentRunner.getComponent("zk", ZKServerComponent.class);
return CuratorFrameworkFactory.newClient(zkServerComponent.getConnectionString(), retryPolicy);
}
- @Bean(destroyMethod="close")
+ @Bean(destroyMethod = "close")
public ZkClient zkClient(ComponentRunner componentRunner) {
ZKServerComponent zkServerComponent = componentRunner.getComponent("zk", ZKServerComponent.class);
return new ZkClient(zkServerComponent.getConnectionString(), 10000, 10000, ZKStringSerializer$.MODULE$);
@@ -96,9 +99,9 @@ public class TestConfig {
return ZkUtils.apply(zkClient, false);
}
- @Bean(destroyMethod="close")
- public KafkaConsumer<String, String> kafkaConsumer(KafkaComponent kafkaWithZKComponent) {
- Properties props = new Properties();
+ @Bean
+ public Map<String, Object> kafkaConsumer(KafkaComponent kafkaWithZKComponent) {
+ Map<String, Object> props = new HashMap<>();
props.put("bootstrap.servers", kafkaWithZKComponent.getBrokerList());
props.put("group.id", "metron-config");
props.put("enable.auto.commit", "false");
@@ -106,7 +109,12 @@ public class TestConfig {
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
- return new KafkaConsumer<>(props);
+ return props;
+ }
+
+ @Bean
+ public ConsumerFactory<String, String> createConsumerFactory() {
+ return new DefaultKafkaConsumerFactory<>(kafkaConsumer(kafkaWithZKComponent(zkProperties())));
}
@Bean
http://git-wip-us.apache.org/repos/asf/metron/blob/47e2b735/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/KafkaServiceImplTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/KafkaServiceImplTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/KafkaServiceImplTest.java
index c7d42b3..c92feab 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/KafkaServiceImplTest.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/KafkaServiceImplTest.java
@@ -41,6 +41,7 @@ import org.junit.runner.RunWith;
import org.powermock.core.classloader.annotations.PowerMockIgnore;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
+import org.springframework.kafka.core.ConsumerFactory;
import java.util.ArrayList;
import java.util.HashMap;
@@ -71,6 +72,7 @@ public class KafkaServiceImplTest {
private ZkUtils zkUtils;
private KafkaConsumer<String, String> kafkaConsumer;
+ private ConsumerFactory<String, String> kafkaConsumerFactory;
private AdminUtils$ adminUtils;
private KafkaService kafkaService;
@@ -86,10 +88,13 @@ public class KafkaServiceImplTest {
@Before
public void setUp() throws Exception {
zkUtils = mock(ZkUtils.class);
+ kafkaConsumerFactory = mock(ConsumerFactory.class);
kafkaConsumer = mock(KafkaConsumer.class);
adminUtils = mock(AdminUtils$.class);
- kafkaService = new KafkaServiceImpl(zkUtils, kafkaConsumer, adminUtils);
+ when(kafkaConsumerFactory.createConsumer()).thenReturn(kafkaConsumer);
+
+ kafkaService = new KafkaServiceImpl(zkUtils, kafkaConsumerFactory, adminUtils);
}
@Test
@@ -104,6 +109,7 @@ public class KafkaServiceImplTest {
verifyZeroInteractions(zkUtils);
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verifyNoMoreInteractions(kafkaConsumer, zkUtils, adminUtils);
}
@@ -119,6 +125,7 @@ public class KafkaServiceImplTest {
verifyZeroInteractions(zkUtils);
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verifyNoMoreInteractions(kafkaConsumer, zkUtils);
}
@@ -137,6 +144,7 @@ public class KafkaServiceImplTest {
verifyZeroInteractions(zkUtils);
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verifyNoMoreInteractions(kafkaConsumer, zkUtils);
}
@@ -156,6 +164,7 @@ public class KafkaServiceImplTest {
verifyZeroInteractions(zkUtils);
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verifyNoMoreInteractions(kafkaConsumer, zkUtils);
}
@@ -167,6 +176,7 @@ public class KafkaServiceImplTest {
verifyZeroInteractions(zkUtils);
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verifyNoMoreInteractions(kafkaConsumer, zkUtils);
}
@@ -180,6 +190,7 @@ public class KafkaServiceImplTest {
assertTrue(kafkaService.deleteTopic("non_existent_topic"));
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verify(adminUtils).deleteTopic(zkUtils, "non_existent_topic");
verifyNoMoreInteractions(kafkaConsumer);
}
@@ -193,6 +204,7 @@ public class KafkaServiceImplTest {
assertFalse(kafkaService.deleteTopic("non_existent_topic"));
verify(kafkaConsumer).listTopics();
+ verify(kafkaConsumer).close();
verifyNoMoreInteractions(kafkaConsumer);
}
@@ -230,6 +242,7 @@ public class KafkaServiceImplTest {
verify(kafkaConsumer).listTopics();
verify(kafkaConsumer, times(0)).partitionsFor("t");
+ verify(kafkaConsumer).close();
verifyZeroInteractions(zkUtils);
verifyNoMoreInteractions(kafkaConsumer);
}
[16/44] metron git commit: METRON-946 Full/Quick dev should default
network_host to [ _local_,
_site_ ] (mattf-horton) closes apache/incubator-metron#578
Posted by ma...@apache.org.
METRON-946 Full/Quick dev should default network_host to [ _local_, _site_ ] (mattf-horton) closes apache/incubator-metron#578
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/64b66f28
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/64b66f28
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/64b66f28
Branch: refs/heads/Metron_0.4.0
Commit: 64b66f28a5410b095c2a64e7e66b9ee9776c4949
Parents: 8532e28
Author: mattf-horton <mf...@hortonworks.com>
Authored: Wed May 10 10:07:02 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Wed May 10 10:07:02 2017 -0700
----------------------------------------------------------------------
metron-deployment/amazon-ec2/conf/defaults.yml | 3 +-
.../inventory/devimage-vagrant/hosts | 52 --------------------
.../inventory/full-dev-platform/group_vars/all | 3 +-
.../inventory/quick-dev-platform/group_vars/all | 3 +-
.../2.3.3/configuration/elastic-site.xml | 4 +-
.../roles/ambari_config/defaults/main.yml | 1 +
.../roles/ambari_config/vars/single_node_vm.yml | 2 +-
.../roles/ambari_config/vars/small_cluster.yml | 2 +-
8 files changed, 11 insertions(+), 59 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/amazon-ec2/conf/defaults.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/conf/defaults.yml b/metron-deployment/amazon-ec2/conf/defaults.yml
index 7a2e350..fd07fd8 100644
--- a/metron-deployment/amazon-ec2/conf/defaults.yml
+++ b/metron-deployment/amazon-ec2/conf/defaults.yml
@@ -102,5 +102,6 @@ elasticsearch_data_dir: "/data1/elasticsearch,/data2/elasticsearch"
install_elasticsearch: True
install_solr: False
elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth0
+## The elasticsearch_network_interface must be in the form specified for Elasticsearch, with leading and trailing underscores.
+elasticsearch_network_interface: _eth0_
elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/inventory/devimage-vagrant/hosts
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/devimage-vagrant/hosts b/metron-deployment/inventory/devimage-vagrant/hosts
deleted file mode 100644
index 98e8681..0000000
--- a/metron-deployment/inventory/devimage-vagrant/hosts
+++ /dev/null
@@ -1,52 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-[ambari_master]
-node1
-
-[ambari_slave]
-node1
-
-[metron_hbase_tables]
-node1
-
-[metron_kafka_topics]
-node1
-
-[enrichment]
-node1
-
-[search]
-node1
-
-[web]
-node1
-
-[sensors]
-node1
-
-[pcap_server]
-node1
-
-[metron:children]
-enrichment
-search
-web
-sensors
-metron_kafka_topics
-metron_hbase_tables
-pcap_server
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/inventory/full-dev-platform/group_vars/all
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/full-dev-platform/group_vars/all b/metron-deployment/inventory/full-dev-platform/group_vars/all
index 7e68cc0..2f097b6 100644
--- a/metron-deployment/inventory/full-dev-platform/group_vars/all
+++ b/metron-deployment/inventory/full-dev-platform/group_vars/all
@@ -85,5 +85,6 @@ solr_collection_name: Metron
solr_number_shards: 1
solr_replication_factor: 1
elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
+## The elasticsearch_network_interface must be in the form specified for Elasticsearch, with leading and trailing underscores.
+elasticsearch_network_interface: _eth1_
elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/inventory/quick-dev-platform/group_vars/all
----------------------------------------------------------------------
diff --git a/metron-deployment/inventory/quick-dev-platform/group_vars/all b/metron-deployment/inventory/quick-dev-platform/group_vars/all
index 7d0703a..a8bba0e 100644
--- a/metron-deployment/inventory/quick-dev-platform/group_vars/all
+++ b/metron-deployment/inventory/quick-dev-platform/group_vars/all
@@ -84,5 +84,6 @@ solr_collection_name: Metron
solr_number_shards: 1
solr_replication_factor: 1
elasticsearch_transport_port: 9300
-elasticsearch_network_interface: eth1
+## The elasticsearch_network_interface must be in the form specified for Elasticsearch, with leading and trailing underscores.
+elasticsearch_network_interface: _eth1_
elasticsearch_web_port: 9200
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
index 33e01bf..7369f92 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml
@@ -189,8 +189,8 @@
</property>
<property>
<name>network_host</name>
- <value>[ _local_ ]</value>
- <description>Network interface(s) ES will bind to within each node. Use [ _local_ ] for single-node install; change to [ _local_, _site_ ] (including square brackets) for typical cluster install. See https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html for ES documentation.</description>
+ <value>[ _local_, _site_ ]</value>
+ <description>Network interface(s) ES will bind to within each node. "_site_" or a more specific external address is required for all multi-node clusters, and also recommended for single-node installs to allow access to ES reports from non-local hosts. Always include the square brackets. See https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html for ES documentation.</description>
</property>
<property>
<name>network_publish_host</name>
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/roles/ambari_config/defaults/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/defaults/main.yml b/metron-deployment/roles/ambari_config/defaults/main.yml
index 9ac0342..e0de145 100644
--- a/metron-deployment/roles/ambari_config/defaults/main.yml
+++ b/metron-deployment/roles/ambari_config/defaults/main.yml
@@ -35,3 +35,4 @@ mapred_map_mem_mb : 1229
mapred_reduce_mem_mb : 1229
topology_classpath: '/etc/hbase/conf:/etc/hadoop/conf'
hdp_stack: "2.5"
+elasticsearch_network_interface: _site_
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
index b24c153..3de1af7 100644
--- a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
+++ b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
@@ -97,7 +97,7 @@ configurations:
index_number_of_replicas: 0
zen_discovery_ping_unicast_hosts: "[ {{ groups.search | join(', ') }} ]"
gateway_recover_after_data_nodes: 1
- network_host: "[ _local_ ]"
+ network_host: "[ _local_, {{ elasticsearch_network_interface }} ]"
masters_also_are_datanodes: "1"
required_configurations:
http://git-wip-us.apache.org/repos/asf/metron/blob/64b66f28/metron-deployment/roles/ambari_config/vars/small_cluster.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/small_cluster.yml b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
index 70c31ea..7803dec 100644
--- a/metron-deployment/roles/ambari_config/vars/small_cluster.yml
+++ b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
@@ -91,7 +91,7 @@ configurations:
index_number_of_replicas: 1
zen_discovery_ping_unicast_hosts: "[ {{ groups.web[0] }}, {{ groups.search | join(', ') }} ]"
gateway_recover_after_data_nodes: 1
- network_host: "[ _local_, _{{ elasticsearch_network_interface }}_ ]"
+ network_host: "[ _local_, {{ elasticsearch_network_interface }} ]"
required_configurations:
- metron-env:
[39/44] metron git commit: METRON-844 Install Metron Management UI
with Ambari MPack (merrimanr) closes apache/metron#581
Posted by ma...@apache.org.
METRON-844 Install Metron Management UI with Ambari MPack (merrimanr) closes apache/metron#581
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/61105c71
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/61105c71
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/61105c71
Branch: refs/heads/Metron_0.4.0
Commit: 61105c71520909674985e90acc0c33a54bcc559e
Parents: 1a0e8f2
Author: merrimanr <me...@gmail.com>
Authored: Wed May 31 08:55:24 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Wed May 31 08:55:24 2017 -0500
----------------------------------------------------------------------
metron-deployment/README.md | 1 +
.../METRON/CURRENT/role_command_order.json | 1 +
.../METRON/CURRENT/configuration/metron-env.xml | 6 +
.../common-services/METRON/CURRENT/metainfo.xml | 50 +++++++
.../package/scripts/management_ui_commands.py | 46 ++++++
.../package/scripts/management_ui_master.py | 90 ++++++++++++
.../package/scripts/params/params_linux.py | 4 +
.../package/templates/management_ui.yml.j2 | 23 +++
.../METRON/CURRENT/quicklinks/quicklinks.json | 15 ++
.../docker/rpm-docker/SPECS/metron.spec | 12 +-
.../roles/ambari_common/tasks/main.yml | 4 +
.../roles/ambari_config/vars/single_node_vm.yml | 3 +-
.../roles/ambari_config/vars/small_cluster.yml | 3 +-
metron-interface/metron-config/README.md | 104 +++++++++-----
metron-interface/metron-config/assembly.xml | 2 +-
.../metron-config/management_ui.yml | 21 +++
.../metron-config/scripts/metron-management-ui | 139 +++++++++++++++++++
.../metron-config/scripts/package.json | 3 +-
.../metron-config/scripts/server.js | 32 ++---
.../scripts/start_management_ui.sh | 24 ----
20 files changed, 497 insertions(+), 86 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/README.md b/metron-deployment/README.md
index f2f4b43..75af150 100644
--- a/metron-deployment/README.md
+++ b/metron-deployment/README.md
@@ -61,6 +61,7 @@ This will set up
### Prerequisites
- A cluster managed by Ambari 2.4.2+
- Metron RPMs available on the cluster in the /localrepo directory. See [RPM](#rpm) for further information.
+- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host
### Building Management Pack
From `metron-deployment` run
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
index 829758f..1c9affc 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
@@ -11,6 +11,7 @@
"METRON_ENRICHMENT_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_ENRICHMENT_MASTER-START"],
"METRON_INDEXING-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_PARSERS-START"],
"METRON_REST-START": ["KAFKA_BROKER-START","STORM_REST_API-START","ZOOKEEPER_SERVER-START","NAMENODE-START","METRON_PARSERS-INSTALL","METRON_INDEXING-INSTALL","METRON_ENRICHMENT-INSTALL"],
+ "METRON_MANAGEMENT_UI-START": ["METRON_REST-START"],
"METRON_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_PARSERS-START","METRON_INDEXING-START"]
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index ebc2d73..3e5a405 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -83,6 +83,12 @@
<display-name>Metron REST port</display-name>
</property>
<property>
+ <name>metron_management_ui_port</name>
+ <value>4200</value>
+ <description>Port the Management UI runs on</description>
+ <display-name>Management UI port</display-name>
+ </property>
+ <property>
<name>metron_jvm_flags</name>
<description>JVM flags passed to Metron scripts</description>
<display-name>Metron JVM flags</display-name>
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index e0fac4c..35828c4 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -220,6 +220,50 @@
</component>
<component>
+ <name>METRON_MANAGEMENT_UI</name>
+ <displayName>Metron Management UI</displayName>
+ <category>MASTER</category>
+ <cardinality>1</cardinality>
+ <versionAdvertised>false</versionAdvertised>
+ <reassignAllowed>false</reassignAllowed>
+ <clientsToUpdateConfigs></clientsToUpdateConfigs>
+ <dependencies>
+ <dependency>
+ <name>HDFS/HDFS_CLIENT</name>
+ <scope>host</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ <dependency>
+ <name>ZOOKEEPER/ZOOKEEPER_SERVER</name>
+ <scope>cluster</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ <dependency>
+ <name>STORM/SUPERVISOR</name>
+ <scope>host</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ <dependency>
+ <name>KAFKA/KAFKA_BROKER</name>
+ <scope>cluster</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ </dependencies>
+ <commandScript>
+ <script>scripts/management_ui_master.py</script>
+ <scriptType>PYTHON</scriptType>
+ </commandScript>
+ </component>
+
+ <component>
<name>METRON_CLIENT</name>
<displayName>Metron Client</displayName>
<category>CLIENT</category>
@@ -269,6 +313,12 @@
<package>
<name>metron-rest</name>
</package>
+ <package>
+ <name>nodejs</name>
+ </package>
+ <package>
+ <name>metron-config</name>
+ </package>
</packages>
</osSpecific>
<osSpecific>
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_commands.py
new file mode 100644
index 0000000..de67f64
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_commands.py
@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute, File
+
+# Wrap major operations and functionality in this class
+class ManagementUICommands:
+ __params = None
+
+ def __init__(self, params):
+ if params is None:
+ raise ValueError("params argument is required for initialization")
+ self.__params = params
+
+ def start_management_ui(self):
+ Logger.info('Starting Management UI')
+ Execute("service metron-management-ui start")
+ Logger.info('Done starting Management UI')
+
+ def stop_management_ui(self):
+ Logger.info('Stopping Management UI')
+ Execute("service metron-management-ui stop")
+ Logger.info('Done stopping Management UI')
+
+ def restart_management_ui(self, env):
+ Logger.info('Restarting the Management UI')
+ Execute('service metron-management-ui restart')
+ Logger.info('Done restarting the Management UI')
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_master.py
new file mode 100644
index 0000000..28e6b55
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/management_ui_master.py
@@ -0,0 +1,90 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.core.exceptions import ComponentIsNotRunning
+from resource_management.core.exceptions import ExecutionFailed
+from resource_management.core.resources.system import Directory
+from resource_management.core.resources.system import File
+from resource_management.core.source import Template
+from resource_management.libraries.functions.format import format
+from resource_management.libraries.script import Script
+from resource_management.core.resources.system import Execute
+
+from resource_management.core.logger import Logger
+
+from management_ui_commands import ManagementUICommands
+
+
+class ManagementUIMaster(Script):
+
+ def install(self, env):
+ from params import params
+ env.set_params(params)
+ self.install_packages(env)
+ Execute('npm --prefix ' + params.metron_home + '/web/expressjs/ install')
+
+ def configure(self, env, upgrade_type=None, config_dir=None):
+ print 'configure managment_ui'
+ from params import params
+ env.set_params(params)
+
+ File(format("{metron_config_path}/management_ui.yml"),
+ mode=0755,
+ content=Template("management_ui.yml.j2"),
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+ Directory('/var/run/metron',
+ create_parents=False,
+ mode=0755,
+ owner=params.metron_user,
+ group=params.metron_group
+ )
+
+ def start(self, env, upgrade_type=None):
+ from params import params
+ env.set_params(params)
+ self.configure(env)
+ commands = ManagementUICommands(params)
+ commands.start_management_ui()
+
+ def stop(self, env, upgrade_type=None):
+ from params import params
+ env.set_params(params)
+ commands = ManagementUICommands(params)
+ commands.stop_management_ui()
+
+ def status(self, env):
+ status_cmd = format('service metron-management-ui status')
+ try:
+ Execute(status_cmd)
+ except ExecutionFailed:
+ raise ComponentIsNotRunning()
+
+ def restart(self, env):
+ from params import params
+ env.set_params(params)
+ self.configure(env)
+ commands = ManagementUICommands(params)
+ commands.restart_management_ui(env)
+
+
+if __name__ == "__main__":
+ ManagementUIMaster().execute()
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index 7f34703..d473325 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -47,6 +47,7 @@ metron_group = config['configurations']['metron-env']['metron_group']
metron_log_dir = config['configurations']['metron-env']['metron_log_dir']
metron_pid_dir = config['configurations']['metron-env']['metron_pid_dir']
metron_rest_port = config['configurations']['metron-env']['metron_rest_port']
+metron_management_ui_port = config['configurations']['metron-env']['metron_management_ui_port']
metron_jvm_flags = ''
metron_spring_profiles_active = config['configurations']['metron-env']['metron_spring_profiles_active']
metron_jdbc_driver = config['configurations']['metron-env']['metron_jdbc_driver']
@@ -219,3 +220,6 @@ if security_enabled:
kafka_keytab_path = config['configurations']['kafka-env']['kafka_keytab']
nimbus_seeds = config['configurations']['storm-site']['nimbus.seeds']
+
+# Management UI
+metron_rest_host = default("/clusterHostInfo/metron_rest_hosts", ['localhost'])[0]
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/management_ui.yml.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/management_ui.yml.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/management_ui.yml.j2
new file mode 100644
index 0000000..e88a9f9
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/management_ui.yml.j2
@@ -0,0 +1,23 @@
+{#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#}
+
+port: {{metron_management_ui_port}}
+
+rest:
+ host: {{metron_rest_host}}
+ port: {{metron_rest_port}}
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
index f9b7091..31fde1f 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
@@ -37,6 +37,21 @@
"regex": "^(\\d+)$",
"site": "metron-env"
}
+ },
+ {
+ "name": "metron_management_ui",
+ "label": "Management UI",
+ "requires_user_name": "false",
+ "component_name": "METRON_MANAGEMENT_UI",
+ "url": "%@://%@:%@/",
+ "port": {
+ "http_property": "metron_management_ui_port",
+ "http_default_port": "4200",
+ "https_property": "metron_management_ui_port",
+ "https_default_port": "4200",
+ "regex": "^(\\d+)$",
+ "site": "metron-env"
+ }
}
]
}
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index b49e0e6..2706e05 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -87,6 +87,7 @@ tar -xzf %{SOURCE9} -C %{buildroot}%{metron_home}
tar -xzf %{SOURCE10} -C %{buildroot}%{metron_home}
install %{buildroot}%{metron_home}/bin/metron-rest %{buildroot}/etc/init.d/
+install %{buildroot}%{metron_home}/bin/metron-management-ui %{buildroot}/etc/init.d/
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -378,7 +379,8 @@ This package installs the Metron Management UI %{metron_home}
%dir %{metron_home}/web/management-ui/assets/fonts/Roboto
%dir %{metron_home}/web/management-ui/assets/images
%dir %{metron_home}/web/management-ui/license
-%{metron_home}/bin/start_management_ui.sh
+%{metron_home}/bin/metron-management-ui
+/etc/init.d/metron-management-ui
%attr(0755,root,root) %{metron_home}/web/expressjs/server.js
%attr(0644,root,root) %{metron_home}/web/expressjs/package.json
%attr(0644,root,root) %{metron_home}/web/management-ui/favicon.ico
@@ -398,9 +400,17 @@ This package installs the Metron Management UI %{metron_home}
%attr(0644,root,root) %{metron_home}/web/management-ui/assets/images/*
%attr(0644,root,root) %{metron_home}/web/management-ui/license/*
+%post config
+chkconfig --add metron-management-ui
+
+%preun config
+chkconfig --del metron-management-ui
+
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%changelog
+* Thu May 15 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
+- Added Management UI
* Tue May 9 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
- Add Zeppelin Connection Volume Report Dashboard
* Thu May 4 2017 Ryan Merriman <me...@gmail.com> - 0.4.0
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/roles/ambari_common/tasks/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_common/tasks/main.yml b/metron-deployment/roles/ambari_common/tasks/main.yml
index ea809d6..011c05f 100644
--- a/metron-deployment/roles/ambari_common/tasks/main.yml
+++ b/metron-deployment/roles/ambari_common/tasks/main.yml
@@ -48,3 +48,7 @@
dest: "/etc/logrotate.d/metron-ambari"
mode: 0644
+- name: Install Nodejs repository
+ shell: curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
+ args:
+ warn: false
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
index 3de1af7..7e54db3 100644
--- a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
+++ b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
@@ -34,11 +34,12 @@ metron_indexing: [METRON_INDEXING]
metron_enrichment_master : [METRON_ENRICHMENT_MASTER]
metron_parsers : [METRON_PARSERS]
metron_rest: [METRON_REST]
+metron_management_ui: [METRON_MANAGEMENT_UI]
metron_components: >
{{ hadoop_master | union(zookeeper_master) | union(storm_master) | union(hbase_master) | union(hadoop_slave) | union(zookeeper_slave) |
union(storm_slave) | union(kafka_broker) | union(hbase_slave) | union(kibana_master) | union(metron_indexing) |
- union(metron_enrichment_master) | union(metron_parsers) | union(metron_rest) | union(es_master) }}
+ union(metron_enrichment_master) | union(metron_parsers) | union(metron_rest) | union(metron_management_ui) | union(es_master) }}
cluster_name: "metron_cluster"
blueprint_name: "metron_blueprint"
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-deployment/roles/ambari_config/vars/small_cluster.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/small_cluster.yml b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
index 7803dec..dbcfb60 100644
--- a/metron-deployment/roles/ambari_config/vars/small_cluster.yml
+++ b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
@@ -34,6 +34,7 @@ metron_indexing: [METRON_INDEXING]
metron_enrichment_master : [METRON_ENRICHMENT_MASTER]
metron_parsers : [METRON_PARSERS]
metron_rest: [METRON_REST]
+metron_management_ui: [METRON_MANAGEMENT_UI]
master_1_components: "{{ hadoop_master | union(hadoop_clients) | union(es_slave) }}"
master_1_host:
@@ -42,7 +43,7 @@ master_2_components: "{{ zookeeper_master | union(storm_master) | union(spark_ma
master_2_host:
- "{{groups.ambari_slave[1]}}"
metron_components: >
- {{ metron_indexing | union(metron_enrichment_master) | union(metron_parsers) | union(metron_rest) | union(hadoop_slave) | union(storm_slave) |
+ {{ metron_indexing | union(metron_enrichment_master) | union(metron_parsers) | union(metron_rest) | union(metron_management_ui) | union(hadoop_slave) | union(storm_slave) |
union(kafka_broker) | union(hbase_slave) | union(hadoop_clients) }}
metron_host:
- "{{ groups.metron[0] }}"
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/README.md b/metron-interface/metron-config/README.md
index 070668f..241adff 100644
--- a/metron-interface/metron-config/README.md
+++ b/metron-interface/metron-config/README.md
@@ -8,69 +8,97 @@ This module provides a user interface for management functions in Metron.
* nodejs v6.9+ (nodejs can be installed on quick dev with `curl --silent --location https://rpm.nodesource.com/setup_6.x | bash - && yum install -y nodejs`)
## Installation
-1. Build Metron:
- ```
- mvn clean package -DskipTests
- ```
-
-1. Copy `metron/metron-interface/metron-config/target/metron-config-METRON_VERSION-archive.tar.gz` to the desired host.
-
-1. Untar the archive in the target directory. The directory structure will look like:
- ```
- bin
- start_management_ui.sh
- web
- expressjs
+
+### From Source
+
+1. Package the application with Maven:
+```
+cd metron-interface/metron-config
+mvn clean package
+```
+
+1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
+```
+bin
+ metron-management-ui
+web
+ expressjs
package.json
server.js
- management-ui
+ management-ui
web assets (html, css, js, ...)
- ```
+```
+
+1. Copy the `$METRON_HOME/bin/metron-management-ui` script to `/etc/init.d/metron-management-ui`
+
+1. Install the [Express](https://expressjs.com/) web framework from the `package.json` file in `$METRON_HOME/web/expressjs`:
+```
+npm --prefix $METRON_HOME/web/expressjs/ install
+```
-1. For production use, the contents of the `./web/management-ui` directory should be deployed to a web server with paths `/api/v1` and `/logout` mapped to the REST application url.
+### From Package Manager
-1. As an example, a convenience script is included that will install a simple [expressjs](https://github.com/expressjs/express) webserver.
+1. Deploy the RPM at `/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-config-$METRON_VERSION-*.noarch.rpm`
-1. Then start the application with the script:
- ```
- ./bin/start_management_ui.sh
- Usage: server.js -p [port] -r [restUrl]
- Options:
- -p Port to run metron management ui [required]
- -r, --resturl Url where metron rest application is available [required]
- ```
+1. Install the RPM with:
+```
+rpm -ih metron-config-$METRON_VERSION-*.noarch.rpm
+```
+
+1. Install the [Express](https://expressjs.com/) web framework from the `package.json` file in `$METRON_HOME/web/expressjs`:
+```
+npm --prefix $METRON_HOME/web/expressjs/ install
+```
+
+## Configuration
+
+The Managment UI is configured in the `$METRON_HOME/config/management_ui.yml` file. Create this file and set the values to match your environment:
+```
+port: port the managment UI will run on
+
+rest:
+ host: REST application host
+ port: REST applciation port
+```
## Usage
-The application will be available at http://host:4200 with credentials `user/password`, assuming the default port is configured and the `dev` profile is included when starting the REST application. See the [REST application](../metron-rest#security) documentation for more information about security configuration for production.
+After configuration is complete, the Management UI can be managed as a service:
+```
+service metron-management-ui start
+```
+
+The application will be available at http://host:4200 assuming the port is set to `4200`. Logs can be found at `/var/log/metron/metron-management-ui.log`.
## Development
The Management UI can also be started in development mode. This allows changes to web assets to be seen interactively.
1. Install the application with dev dependencies:
- ```
- npm install
- ```
+```
+cd metron-interface/metron-config
+npm install
+```
1. Start the application:
- ```
- ./scripts/start_dev.sh
- ```
+```
+./scripts/start_dev.sh
+```
The application will be available at http://localhost:4200/. The REST application url defaults to `http://localhost:8080` but can be changed in the `proxy.conf.json` file.
## Testing
1. Install the application with dev dependencies:
- ```
- npm install
- ```
+```
+cd metron-interface/metron-config
+npm install
+```
1. Unit tests can be run with:
- ```
- npm test
- ```
+```
+npm test
+```
## License
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/assembly.xml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/assembly.xml b/metron-interface/metron-config/assembly.xml
index 86eedd4..39c00e5 100644
--- a/metron-interface/metron-config/assembly.xml
+++ b/metron-interface/metron-config/assembly.xml
@@ -40,7 +40,7 @@
<outputDirectory>/bin</outputDirectory>
<useDefaultExcludes>true</useDefaultExcludes>
<includes>
- <include>start_management_ui.sh</include>
+ <include>metron-management-ui</include>
</includes>
<fileMode>0755</fileMode>
<lineEnding>unix</lineEnding>
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/management_ui.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/management_ui.yml b/metron-interface/metron-config/management_ui.yml
new file mode 100644
index 0000000..395628d
--- /dev/null
+++ b/metron-interface/metron-config/management_ui.yml
@@ -0,0 +1,21 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+port: 4200
+
+rest:
+ host: localhost
+ port: 8080
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/scripts/metron-management-ui
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/scripts/metron-management-ui b/metron-interface/metron-config/scripts/metron-management-ui
new file mode 100644
index 0000000..32b4e9e
--- /dev/null
+++ b/metron-interface/metron-config/scripts/metron-management-ui
@@ -0,0 +1,139 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# metron management UI service
+# chkconfig: - 20 80
+# description: Management UI
+# processname: metron-management-ui
+#
+if [ -f /etc/rc.d/init.d/functions ]; then
+ . /etc/rc.d/init.d/functions
+fi
+
+NAME=metron-management-ui
+DESC="Metron Management UI"
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+METRON_LOG_DIR="/var/log/metron"
+METRON_PID_DIR="/var/run/metron"
+METRON_USER="metron"
+METRON_GROUP="metron"
+
+PIDFILE="$METRON_PID_DIR/$NAME.pid"
+LOCKFILE=/var/lock/subsys/$NAME
+
+DAEMON="node $METRON_HOME/web/expressjs/server.js -c $METRON_HOME/config/management_ui.yml"
+
+#
+# start the rest application
+#
+start() {
+
+ # if pidfile exists, do not start another
+ if [ -f $PIDFILE ]; then
+ PID=`cat $PIDFILE`
+ printf "OK [$PID]\n"
+ return
+ fi
+
+ if [ ! -d "$METRON_LOG_DIR" ]; then
+ mkdir -p "$METRON_LOG_DIR" && chown "$METRON_USER":"$METRON_GROUP" "$METRON_LOG_DIR"
+ fi
+
+ if [ ! -d "$METRON_PID_DIR" ]; then
+ mkdir -p "$METRON_PID_DIR" && chown "$METRON_USER":"$METRON_GROUP" "$METRON_PID_DIR"
+ fi
+
+ # kick-off the daemon
+ PID=`runuser -s /bin/bash $METRON_USER -c "$DAEMON" >> $METRON_LOG_DIR/$NAME.log 2>&1 & echo $!`
+
+ if [ -z $PID ]; then
+ printf "Fail\n"
+ else
+ echo $PID > $PIDFILE && touch $LOCKFILE
+ printf "Ok [$PID]\n"
+ fi
+}
+
+#
+# stop the rest application
+#
+stop() {
+ if [ -f $PIDFILE ]; then
+ PID=`cat $PIDFILE`
+ killproc -p $PIDFILE metron-rest
+ RETVAL=$?
+ echo
+ [ $RETVAL -eq 0 ] && rm -f $LOCKFILE
+ printf "Stopped [$PID]\n"
+ else
+ printf "Not running\n"
+ fi
+}
+
+#
+# status check of the rest application
+#
+management_ui_status() {
+ status -p $PIDFILE metron-managment-ui
+}
+
+case "$1" in
+
+ ##############################################################################
+ # start
+ #
+ start)
+ printf "%-50s \n" "Starting $NAME..."
+ start
+ ;;
+
+ ##############################################################################
+ # status
+ #
+ status)
+ printf "%-50s \n" "Checking $NAME..."
+ management_ui_status
+ ;;
+
+ ##############################################################################
+ # stop
+ #
+ stop)
+ printf "%-50s \n" "Stopping $NAME..."
+ stop
+ ;;
+
+ ##############################################################################
+ # restart
+ #
+ restart)
+ $0 stop
+ $0 start
+ ;;
+
+ ##############################################################################
+ # reload
+ #
+ reload)
+ ;;
+
+ *)
+ echo "Usage: $0 {status|start|stop|restart}"
+ exit 1
+esac
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/scripts/package.json
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/scripts/package.json b/metron-interface/metron-config/scripts/package.json
index 6c7b23d..aca8ed7 100644
--- a/metron-interface/metron-config/scripts/package.json
+++ b/metron-interface/metron-config/scripts/package.json
@@ -9,7 +9,8 @@
"http-proxy-middleware": "0.17.4",
"optimist": "0.6.1",
"serve-favicon": "2.4.2",
- "serve-static": "1.12.1"
+ "serve-static": "1.12.1",
+ "yamljs": "0.2.9"
},
"devDependencies": {},
"scripts": {
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/scripts/server.js
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/scripts/server.js b/metron-interface/metron-config/scripts/server.js
old mode 100644
new mode 100755
index 7fb1728..7c8ee9e
--- a/metron-interface/metron-config/scripts/server.js
+++ b/metron-interface/metron-config/scripts/server.js
@@ -22,34 +22,27 @@
var os = require('os');
var app = require('express')();
var path = require('path');
-var compression = require('compression')
+var compression = require('compression');
var serveStatic = require('serve-static');
var favicon = require('serve-favicon');
var proxy = require('http-proxy-middleware');
var argv = require('optimist')
- .demand(['p', 'r'])
- .alias('r', 'resturl')
- .usage('Usage: server.js -p [port] -r [restUrl]')
- .describe('p', 'Port to run metron management ui')
- .describe('r', 'Url where metron rest application is available')
+ .demand(['c'])
+ .alias('c', 'config_file')
+ .usage('Usage: server.js -c [config_file]')
+ .describe('c', 'Path to management_ui.yml')
.argv;
+var YAML = require('yamljs');
-var port = argv.p;
var metronUIAddress = '';
var ifaces = os.networkInterfaces();
-var restUrl = argv.r || argv.resturl;
-var conf = {
- "rest": {
- "target": restUrl,
- "secure": false
- }
-};
+var uiConfig = YAML.load(argv.c);
Object.keys(ifaces).forEach(function (dev) {
ifaces[dev].forEach(function (details) {
if (details.family === 'IPv4') {
metronUIAddress += '\n';
- metronUIAddress += 'http://' + details.address + ':' + port;
+ metronUIAddress += 'http://' + details.address + ':' + uiConfig.port;
}
});
});
@@ -63,8 +56,9 @@ function setCustomCacheControl (res, path) {
app.use(compression());
-app.use('/api/v1', proxy(conf.rest));
-app.use('/logout', proxy(conf.rest));
+var restUrl = 'http://' + uiConfig.rest.host + ':' + uiConfig.rest.port;
+app.use('/api/v1', proxy(restUrl));
+app.use('/logout', proxy(restUrl));
app.use(favicon(path.join(__dirname, '../management-ui/favicon.ico')));
@@ -74,9 +68,9 @@ app.use(serveStatic(path.join(__dirname, '../management-ui'), {
}));
app.get('*', function(req, res){
- res.sendFile(path.resolve('../management-ui/index.html'));
+ res.sendFile(path.join(__dirname, '../management-ui/index.html'));
});
-app.listen(port, function(){
+app.listen(uiConfig.port, function(){
console.log("Metron server listening on " + metronUIAddress);
});
http://git-wip-us.apache.org/repos/asf/metron/blob/61105c71/metron-interface/metron-config/scripts/start_management_ui.sh
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/scripts/start_management_ui.sh b/metron-interface/metron-config/scripts/start_management_ui.sh
deleted file mode 100755
index cfd055b..0000000
--- a/metron-interface/metron-config/scripts/start_management_ui.sh
+++ /dev/null
@@ -1,24 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-METRON_VERSION=${project.version}
-METRON_HOME=/usr/metron/$METRON_VERSION
-
-cd $METRON_HOME/web/expressjs
-npm install
-node $METRON_HOME/web/expressjs/server.js $*
[07/44] metron git commit: METRON-795 Install Metron REST with Ambari
MPack (merrimanr) closes apache/incubator-metron#500
Posted by ma...@apache.org.
METRON-795 Install Metron REST with Ambari MPack (merrimanr) closes apache/incubator-metron#500
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/716bda32
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/716bda32
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/716bda32
Branch: refs/heads/Metron_0.4.0
Commit: 716bda326f26b60c98318cdc1cc949b351a0b71a
Parents: 38d26d4
Author: merrimanr <me...@gmail.com>
Authored: Thu May 4 15:57:14 2017 -0500
Committer: merrimanr <me...@gmail.com>
Committed: Thu May 4 15:57:14 2017 -0500
----------------------------------------------------------------------
.../METRON/CURRENT/role_command_order.json | 2 +
.../METRON/CURRENT/configuration/metron-env.xml | 103 ++++++++
.../common-services/METRON/CURRENT/metainfo.xml | 47 ++++
.../package/scripts/params/params_linux.py | 17 ++
.../package/scripts/params/status_params.py | 1 +
.../CURRENT/package/scripts/rest_commands.py | 69 ++++++
.../CURRENT/package/scripts/rest_master.py | 83 +++++++
.../METRON/CURRENT/package/templates/metron.j2 | 39 +++
.../METRON/CURRENT/quicklinks/quicklinks.json | 15 ++
.../METRON/CURRENT/service_advisor.py | 5 +
.../docker/rpm-docker/SPECS/metron.spec | 35 +++
.../packaging/docker/rpm-docker/pom.xml | 6 +
.../roles/ambari_config/vars/single_node_vm.yml | 9 +-
.../roles/ambari_config/vars/small_cluster.yml | 3 +-
metron-interface/metron-rest/README.md | 237 ++++++++++++-------
.../src/main/config/rest_application.yml | 54 +++++
.../metron/rest/controller/HdfsController.java | 4 +-
.../src/main/resources/application-vagrant.yml | 2 +-
.../src/main/resources/schema-mysql.sql | 3 -
.../metron-rest/src/main/scripts/metron-rest | 160 +++++++++++++
.../src/main/scripts/start_metron_rest.sh | 25 --
.../metron-rest/src/test/resources/README.vm | 223 +++++++++++------
22 files changed, 953 insertions(+), 189 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
index 9c71887..829758f 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json
@@ -5,10 +5,12 @@
"_comment" : "dependencies for all cases",
"METRON_INDEXING-INSTALL" : ["METRON_PARSERS-INSTALL"],
"METRON_ENRICHMENT-INSTALL": ["METRON_INDEXING-INSTALL"],
+ "METRON_REST-INSTALL": ["METRON_PARSERS-INSTALL"],
"METRON_PARSERS-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_ENRICHMENT_MASTER-START"],
"METRON_ENRICHMENT_MASTER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START"],
"METRON_ENRICHMENT_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_ENRICHMENT_MASTER-START"],
"METRON_INDEXING-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_PARSERS-START"],
+ "METRON_REST-START": ["KAFKA_BROKER-START","STORM_REST_API-START","ZOOKEEPER_SERVER-START","NAMENODE-START","METRON_PARSERS-INSTALL","METRON_INDEXING-INSTALL","METRON_ENRICHMENT-INSTALL"],
"METRON_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_PARSERS-START","METRON_INDEXING-START"]
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index d09614b..ebc2d73 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -65,6 +65,109 @@
</value-attributes>
</property>
<property>
+ <name>metron_log_dir</name>
+ <value>/var/log/metron</value>
+ <description>Log directory for metron</description>
+ <display-name>Metron log dir</display-name>
+ </property>
+ <property>
+ <name>metron_pid_dir</name>
+ <value>/var/run/metron</value>
+ <description>The directory for metron pid files</description>
+ <display-name>Metron pid dir</display-name>
+ </property>
+ <property>
+ <name>metron_rest_port</name>
+ <value>8082</value>
+ <description>Port the REST application runs on</description>
+ <display-name>Metron REST port</display-name>
+ </property>
+ <property>
+ <name>metron_jvm_flags</name>
+ <description>JVM flags passed to Metron scripts</description>
+ <display-name>Metron JVM flags</display-name>
+ <value/>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
+ </property>
+ <property>
+ <name>metron_spring_profiles_active</name>
+ <description>Active Spring profiles</description>
+ <display-name>Active Spring profiles</display-name>
+ <value/>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
+ </property>
+ <property require-input="true">
+ <name>metron_jdbc_driver</name>
+ <value></value>
+ <description>Class name of the JDBC Driver used by Metron</description>
+ <display-name>Metron JDBC Driver</display-name>
+ </property>
+ <property require-input="true">
+ <name>metron_jdbc_url</name>
+ <value></value>
+ <description>JDBC Connection URL used by Metron</description>
+ <display-name>Metron JDBC URL</display-name>
+ </property>
+ <property require-input="true">
+ <name>metron_jdbc_username</name>
+ <value></value>
+ <description>Metron JDBC Username</description>
+ <display-name>Metron JDBC username</display-name>
+ </property>
+ <property require-input="true">
+ <name>metron_jdbc_password</name>
+ <value></value>
+ <property-type>PASSWORD</property-type>
+ <description>Metron JDBC Password</description>
+ <display-name>Metron JDBC password</display-name>
+ <value-attributes>
+ <type>password</type>
+ <overridable>false</overridable>
+ <hidden>METRON_CLIENT</hidden>
+ </value-attributes>
+ <on-ambari-upgrade add="true"/>
+ </property>
+ <property require-input="true">
+ <name>metron_jdbc_platform</name>
+ <value></value>
+ <description>Database platform used by Metron. One of: hsqldb, h2, oracle, mysql, postgresql</description>
+ <display-name>Metron JDBC platform</display-name>
+ </property>
+ <property>
+ <name>metron_jdbc_client_path</name>
+ <description>Path to JDBC jar for selected platform</description>
+ <display-name>Metron JDBC client path</display-name>
+ <value/>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
+ </property>
+ <property>
+ <name>metron_temp_grok_path</name>
+ <description>Temporary local file path where grok patterns are written during testing</description>
+ <value>./patterns/temp</value>
+ <display-name>Metron temp grok path</display-name>
+ </property>
+ <property>
+ <name>metron_default_grok_path</name>
+ <description>Default HDFS directory path used when storing Grok patterns</description>
+ <value>/apps/metron/patterns</value>
+ <display-name>Metron default grok path</display-name>
+ </property>
+ <property>
+ <name>metron_spring_options</name>
+ <description>Additional Spring options not included in the rest_application.yml file</description>
+ <display-name>Metron Spring options</display-name>
+ <value/>
+ <value-attributes>
+ <empty-value-valid>true</empty-value-valid>
+ </value-attributes>
+ </property>
+ <property>
<name>metron_topic_retention</name>
<description>Kafka Retention in GB</description>
<value>10</value>
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
index a4e82e2..e0fac4c 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/metainfo.xml
@@ -176,6 +176,50 @@
</component>
<component>
+ <name>METRON_REST</name>
+ <displayName>Metron REST</displayName>
+ <category>MASTER</category>
+ <cardinality>1</cardinality>
+ <versionAdvertised>false</versionAdvertised>
+ <reassignAllowed>false</reassignAllowed>
+ <clientsToUpdateConfigs></clientsToUpdateConfigs>
+ <dependencies>
+ <dependency>
+ <name>HDFS/HDFS_CLIENT</name>
+ <scope>host</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ <dependency>
+ <name>ZOOKEEPER/ZOOKEEPER_SERVER</name>
+ <scope>cluster</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ <dependency>
+ <name>STORM/SUPERVISOR</name>
+ <scope>host</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ <dependency>
+ <name>KAFKA/KAFKA_BROKER</name>
+ <scope>cluster</scope>
+ <auto-deploy>
+ <enabled>true</enabled>
+ </auto-deploy>
+ </dependency>
+ </dependencies>
+ <commandScript>
+ <script>scripts/rest_master.py</script>
+ <scriptType>PYTHON</scriptType>
+ </commandScript>
+ </component>
+
+ <component>
<name>METRON_CLIENT</name>
<displayName>Metron Client</displayName>
<category>CLIENT</category>
@@ -222,6 +266,9 @@
<package>
<name>metron-pcap</name>
</package>
+ <package>
+ <name>metron-rest</name>
+ </package>
</packages>
</osSpecific>
<osSpecific>
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index f2ab12b..7f34703 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -44,11 +44,26 @@ geoip_hdfs_dir = "/apps/metron/geo/default/"
metron_indexing_topology = status_params.metron_indexing_topology
metron_user = status_params.metron_user
metron_group = config['configurations']['metron-env']['metron_group']
+metron_log_dir = config['configurations']['metron-env']['metron_log_dir']
+metron_pid_dir = config['configurations']['metron-env']['metron_pid_dir']
+metron_rest_port = config['configurations']['metron-env']['metron_rest_port']
+metron_jvm_flags = ''
+metron_spring_profiles_active = config['configurations']['metron-env']['metron_spring_profiles_active']
+metron_jdbc_driver = config['configurations']['metron-env']['metron_jdbc_driver']
+metron_jdbc_url = config['configurations']['metron-env']['metron_jdbc_url']
+metron_jdbc_username = config['configurations']['metron-env']['metron_jdbc_username']
+metron_jdbc_password = config['configurations']['metron-env']['metron_jdbc_password']
+metron_jdbc_platform = config['configurations']['metron-env']['metron_jdbc_platform']
+metron_jdbc_client_path = config['configurations']['metron-env']['metron_jdbc_client_path']
+metron_temp_grok_path = config['configurations']['metron-env']['metron_temp_grok_path']
+metron_default_grok_path = config['configurations']['metron-env']['metron_default_grok_path']
+metron_spring_options = config['configurations']['metron-env']['metron_spring_options']
metron_config_path = metron_home + '/config'
metron_zookeeper_config_dir = status_params.metron_zookeeper_config_dir
metron_zookeeper_config_path = status_params.metron_zookeeper_config_path
parsers_configured_flag_file = status_params.parsers_configured_flag_file
parsers_acl_configured_flag_file = status_params.parsers_acl_configured_flag_file
+rest_acl_configured_flag_file = status_params.rest_acl_configured_flag_file
enrichment_kafka_configured_flag_file = status_params.enrichment_kafka_configured_flag_file
enrichment_kafka_acl_configured_flag_file = status_params.enrichment_kafka_acl_configured_flag_file
enrichment_hbase_configured_flag_file = status_params.enrichment_hbase_configured_flag_file
@@ -160,6 +175,8 @@ threatintel_cf = status_params.threatintel_cf
metron_enrichment_topology = status_params.metron_enrichment_topology
metron_enrichment_topic = status_params.metron_enrichment_topic
metron_error_topic = 'indexing'
+ambari_kafka_service_check_topic = 'ambari_kafka_service_check'
+consumer_offsets_topic = '__consumer_offsets'
# ES Templates
bro_index_path = tmp_dir + "/bro_index.template"
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
index 154b3de..690c301 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
@@ -34,6 +34,7 @@ metron_zookeeper_config_dir = config['configurations']['metron-env']['metron_zoo
metron_zookeeper_config_path = format('{metron_home}/{metron_zookeeper_config_dir}')
parsers_configured_flag_file = metron_zookeeper_config_path + '/../metron_parsers_configured'
parsers_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_parsers_acl_configured'
+rest_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_rest_acl_configured'
# Enrichment
metron_enrichment_topology = 'enrichment'
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
new file mode 100755
index 0000000..fe5fa6e
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_commands.py
@@ -0,0 +1,69 @@
+#!/usr/bin/env python
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+import os
+
+from resource_management.core.logger import Logger
+from resource_management.core.resources.system import Execute, File
+from resource_management.libraries.functions.format import format
+
+import metron_service
+
+# Wrap major operations and functionality in this class
+class RestCommands:
+ __params = None
+ __acl_configured = False
+
+ def __init__(self, params):
+ if params is None:
+ raise ValueError("params argument is required for initialization")
+ self.__params = params
+ self.__acl_configured = os.path.isfile(self.__params.rest_acl_configured_flag_file)
+
+ def is_acl_configured(self):
+ return self.__acl_configured
+
+ def set_acl_configured(self):
+ File(self.__params.rest_acl_configured_flag_file,
+ content="",
+ owner=self.__params.metron_user,
+ mode=0755)
+
+ def init_kafka_acls(self):
+ Logger.info('Creating Kafka ACLs for rest')
+ # The following topics must be permissioned for the rest application list operation
+ topics = [self.__params.ambari_kafka_service_check_topic, self.__params.consumer_offsets_topic]
+ metron_service.init_kafka_acls(self.__params, topics, ['metron-rest'])
+
+ def start_rest_application(self):
+ Logger.info('Starting REST application')
+ command = format("service metron-rest start {metron_jdbc_password!p}")
+ Execute(command)
+ Logger.info('Done starting REST application')
+
+ def stop_rest_application(self):
+ Logger.info('Stopping REST application')
+ Execute("service metron-rest stop")
+ Logger.info('Done stopping REST application')
+
+ def restart_rest_application(self, env):
+ Logger.info('Restarting the REST application')
+ command = format("service metron-rest restart {metron_jdbc_password!p}")
+ Execute(command)
+ Logger.info('Done restarting the REST application')
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py
new file mode 100755
index 0000000..a097dff
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/rest_master.py
@@ -0,0 +1,83 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+"""
+
+from resource_management.core.exceptions import ComponentIsNotRunning
+from resource_management.core.exceptions import ExecutionFailed
+from resource_management.core.resources.system import Directory
+from resource_management.core.resources.system import File
+from resource_management.core.source import Template
+from resource_management.libraries.functions.format import format
+from resource_management.libraries.script import Script
+from resource_management.core.resources.system import Execute
+
+from rest_commands import RestCommands
+
+
+class RestMaster(Script):
+
+ def install(self, env):
+ from params import params
+ env.set_params(params)
+ self.install_packages(env)
+
+ def configure(self, env, upgrade_type=None, config_dir=None):
+ from params import params
+ if params.security_enabled:
+ params.metron_jvm_flags = format('-Djava.security.auth.login.config={client_jaas_path}')
+
+ env.set_params(params)
+ File(format("/etc/sysconfig/metron"),
+ content=Template("metron.j2")
+ )
+
+ commands = RestCommands(params)
+ if params.security_enabled and not commands.is_acl_configured():
+ commands.init_kafka_acls()
+ commands.set_acl_configured()
+
+ def start(self, env, upgrade_type=None):
+ from params import params
+ env.set_params(params)
+ self.configure(env)
+ commands = RestCommands(params)
+ commands.start_rest_application()
+
+ def stop(self, env, upgrade_type=None):
+ from params import params
+ env.set_params(params)
+ commands = RestCommands(params)
+ commands.stop_rest_application()
+
+ def status(self, env):
+ status_cmd = format('service metron-rest status')
+ try:
+ Execute(status_cmd)
+ except ExecutionFailed:
+ raise ComponentIsNotRunning()
+
+ def restart(self, env):
+ from params import params
+ env.set_params(params)
+ self.configure(env)
+ commands = RestCommands(params)
+ commands.restart_rest_application(env)
+
+
+if __name__ == "__main__":
+ RestMaster().execute()
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2 b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
new file mode 100644
index 0000000..7543a67
--- /dev/null
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+METRON_HOME="{{metron_home}}"
+METRON_USER="{{metron_user}}"
+METRON_LOG_DIR="{{metron_log_dir}}"
+METRON_PID_DIR="{{metron_pid_dir}}"
+METRON_REST_PORT={{metron_rest_port}}
+METRON_JVMFLAGS="{{metron_jvm_flags}}"
+METRON_SPRING_PROFILES_ACTIVE="{{metron_spring_profiles_active}}"
+METRON_JDBC_DRIVER="{{metron_jdbc_driver}}"
+METRON_JDBC_URL="{{metron_jdbc_url}}"
+METRON_JDBC_USERNAME="{{metron_jdbc_username}}"
+METRON_JDBC_PLATFORM="{{metron_jdbc_platform}}"
+METRON_JDBC_CLIENT_PATH="{{metron_jdbc_client_path}}"
+METRON_TEMP_GROK_PATH="{{metron_temp_grok_path}}"
+METRON_DEFAULT_GROK_PATH="{{metron_default_grok_path}}"
+METRON_SPRING_OPTIONS="{{metron_spring_options}}"
+ZOOKEEPER="{{zookeeper_quorum}}"
+BROKERLIST="{{kafka_brokers}}"
+HDFS_URL="{{default_fs}}"
+STORM_REST_URL="{{storm_rest_addr}}"
+SECURITY_ENABLED={{security_enabled|lower}}
+{% if metron_principal_name is defined %}METRON_PRINCIPAL_NAME="{{metron_principal_name}}"
+{% endif %}
+{% if metron_keytab_path is defined %}METRON_SERVICE_KEYTAB="{{metron_keytab_path}}"
+{% endif %}
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
index ee1b225..f9b7091 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/quicklinks/quicklinks.json
@@ -22,6 +22,21 @@
"regex": "^(\\d+)$",
"site": "storm-site"
}
+ },
+ {
+ "name": "metron_rest_ui",
+ "label": "Swagger UI",
+ "requires_user_name": "false",
+ "component_name": "METRON_REST",
+ "url":"%@://%@:%@/swagger-ui.html",
+ "port":{
+ "http_property": "metron_rest_port",
+ "http_default_port": "8082",
+ "https_property": "metron_rest_port",
+ "https_default_port": "8082",
+ "regex": "^(\\d+)$",
+ "site": "metron-env"
+ }
}
]
}
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index 23aebb4..464857a 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -44,6 +44,7 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
metronParsersHost = self.getHosts(componentsList, "METRON_PARSERS")[0]
metronEnrichmentMaster = self.getHosts(componentsList, "METRON_ENRICHMENT_MASTER")[0]
metronIndexingHost = self.getHosts(componentsList, "METRON_INDEXING")[0]
+ metronRESTHost = self.getHosts(componentsList, "METRON_REST")[0]
hbaseClientHosts = self.getHosts(componentsList, "HBASE_CLIENT")
hdfsClientHosts = self.getHosts(componentsList, "HDFS_CLIENT")
@@ -63,6 +64,10 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
message = "Metron must be colocated with an instance of STORM SUPERVISOR"
items.append({ "type": 'host-component', "level": 'WARN', "message": message, "component-name": 'METRON_PARSERS', "host": metronParsersHost })
+ if metronRESTHost not in stormSupervisors:
+ message = "Metron REST must be colocated with an instance of STORM SUPERVISOR"
+ items.append({ "type": 'host-component', "level": 'ERROR', "message": message, "component-name": 'METRON_REST', "host": metronRESTHost })
+
if metronParsersHost != metronEnrichmentMaster:
message = "Metron Enrichment Master must be co-located with Metron Parsers on {0}".format(metronParsersHost)
items.append({ "type": 'host-component', "level": 'ERROR', "message": message, "component-name": 'METRON_ENRICHMENT_MASTER', "host": metronEnrichmentMaster })
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index eb30cac..76129c3 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -50,6 +50,7 @@ Source5: metron-enrichment-%{full_version}-archive.tar.gz
Source6: metron-indexing-%{full_version}-archive.tar.gz
Source7: metron-pcap-backend-%{full_version}-archive.tar.gz
Source8: metron-profiler-%{full_version}-archive.tar.gz
+Source9: metron-rest-%{full_version}-archive.tar.gz
Source10: metron-config-%{full_version}-archive.tar.gz
%description
@@ -70,6 +71,7 @@ rm -rf %{_builddir}/*
%install
rm -rf %{buildroot}
mkdir -p %{buildroot}%{metron_home}
+mkdir -p %{buildroot}/etc/init.d
# copy source files and untar
tar -xzf %{SOURCE0} -C %{buildroot}%{metron_home}
@@ -81,8 +83,11 @@ tar -xzf %{SOURCE5} -C %{buildroot}%{metron_home}
tar -xzf %{SOURCE6} -C %{buildroot}%{metron_home}
tar -xzf %{SOURCE7} -C %{buildroot}%{metron_home}
tar -xzf %{SOURCE8} -C %{buildroot}%{metron_home}
+tar -xzf %{SOURCE9} -C %{buildroot}%{metron_home}
tar -xzf %{SOURCE10} -C %{buildroot}%{metron_home}
+install %{buildroot}%{metron_home}/bin/metron-rest %{buildroot}/etc/init.d/
+
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%package common
@@ -320,6 +325,34 @@ This package installs the Metron Profiler %{metron_home}
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+%package rest
+Summary: Metron Rest
+Group: Applications/Internet
+Provides: rest = %{version}
+
+%description rest
+This package installs the Metron Rest %{metron_home}
+
+%files rest
+%defattr(-,root,root,755)
+%dir %{metron_root}
+%dir %{metron_home}
+%dir %{metron_home}/config
+%dir %{metron_home}/bin
+%dir %{metron_home}/lib
+%{metron_home}/config/rest_application.yml
+%{metron_home}/bin/metron-rest
+/etc/init.d/metron-rest
+%attr(0644,root,root) %{metron_home}/lib/metron-rest-%{full_version}.jar
+
+%post rest
+chkconfig --add metron-rest
+
+%preun rest
+chkconfig --del metron-rest
+
+# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
%package config
Summary: Metron Management UI
Group: Applications/Internet
@@ -366,6 +399,8 @@ This package installs the Metron Management UI %{metron_home}
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%changelog
+* Thu May 4 2017 Ryan Merriman <me...@gmail.com> - 0.4.0
+- Added REST
* Fri Apr 28 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
- Add Zeppelin Connection Report Dashboard
* Thu Jan 19 2017 Justin Leet <ju...@gmail.com> - 0.3.1
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/packaging/docker/rpm-docker/pom.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/pom.xml b/metron-deployment/packaging/docker/rpm-docker/pom.xml
index ec22223..8feeb2b 100644
--- a/metron-deployment/packaging/docker/rpm-docker/pom.xml
+++ b/metron-deployment/packaging/docker/rpm-docker/pom.xml
@@ -156,6 +156,12 @@
</includes>
</resource>
<resource>
+ <directory>${metron_dir}/metron-interface/metron-rest/target/</directory>
+ <includes>
+ <include>*.tar.gz</include>
+ </includes>
+ </resource>
+ <resource>
<directory>${metron_dir}/metron-interface/metron-config/target/</directory>
<includes>
<include>*.tar.gz</include>
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
index 46a486e..fe25eed 100644
--- a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
+++ b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
@@ -33,11 +33,12 @@ kibana_master: [KIBANA_MASTER]
metron_indexing: [METRON_INDEXING]
metron_enrichment_master : [METRON_ENRICHMENT_MASTER]
metron_parsers : [METRON_PARSERS]
+metron_rest: [METRON_REST]
metron_components: >
{{ hadoop_master | union(zookeeper_master) | union(storm_master) | union(hbase_master) | union(hadoop_slave) | union(zookeeper_slave) |
union(storm_slave) | union(kafka_broker) | union(hbase_slave) | union(kibana_master) | union(metron_indexing) |
- union(metron_enrichment_master) | union(metron_parsers) | union(es_master) }}
+ union(metron_enrichment_master) | union(metron_parsers) | union(metron_rest) | union(es_master) }}
cluster_name: "metron_cluster"
blueprint_name: "metron_blueprint"
@@ -90,6 +91,7 @@ configurations:
delete.topic.enable: "true"
- metron-env:
parsers: "bro,snort"
+ metron_spring_profiles_active: "dev"
- elastic-site:
index_number_of_shards: 1
index_number_of_replicas: 0
@@ -103,6 +105,11 @@ required_configurations:
storm_rest_addr: "{{ groups.ambari_slave[0] }}:8744"
es_hosts: "{{ groups.search | join(',') }}"
zeppelin_server_url: "{{ groups.zeppelin[0] }}:9995"
+ metron_jdbc_driver: "org.h2.Driver"
+ metron_jdbc_url: "jdbc:h2:file:~/metrondb"
+ metron_jdbc_username: "root"
+ metron_jdbc_password: "root"
+ metron_jdbc_platform: "h2"
- kibana-env:
kibana_pid_dir: /var/run/kibana
kibana_es_url: http://{{ groups.search[0] }}:9200
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-deployment/roles/ambari_config/vars/small_cluster.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/small_cluster.yml b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
index 2f15740..76d7761 100644
--- a/metron-deployment/roles/ambari_config/vars/small_cluster.yml
+++ b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
@@ -33,6 +33,7 @@ kibana_master: [KIBANA_MASTER]
metron_indexing: [METRON_INDEXING]
metron_enrichment_master : [METRON_ENRICHMENT_MASTER]
metron_parsers : [METRON_PARSERS]
+metron_rest: [METRON_REST]
master_1_components: "{{ hadoop_master | union(hadoop_clients) | union(es_slave) }}"
master_1_host:
@@ -41,7 +42,7 @@ master_2_components: "{{ zookeeper_master | union(storm_master) | union(spark_ma
master_2_host:
- "{{groups.ambari_slave[1]}}"
metron_components: >
- {{ metron_indexing | union(metron_enrichment_master) | union(metron_parsers) | union(hadoop_slave) | union(storm_slave) |
+ {{ metron_indexing | union(metron_enrichment_master) | union(metron_parsers) | union(metron_rest) | union(hadoop_slave) | union(storm_slave) |
union(kafka_broker) | union(hbase_slave) | union(hadoop_clients) }}
metron_host:
- "{{ groups.metron[0] }}"
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index 89984ed..93beb0c 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -7,76 +7,165 @@ This module provides a RESTful API for interacting with Metron.
* A running Metron cluster
* Java 8 installed
* Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
+* A relational database
## Installation
+
+### From Source
+
1. Package the application with Maven:
- ```
- mvn clean package
- ```
+```
+mvn clean package
+```
+
+1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
+```
+config
+ rest_application.yml
+bin
+ metron-rest
+lib
+ metron-rest-$METRON_VERSION.jar
+```
+
+1. Copy the `$METRON_HOME/bin/metron-rest` script to `/etc/init.d/metron-rest`
+
+### From Package Manager
+
+1. Deploy the RPM at `/incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
+
+1. Install the RPM with:
+ ```
+ rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
+ ```
+
+## Configuration
+
+The REST application depends on several configuration parameters:
+
+| Environment Variable | Description | Required | Default |
+| ------------------------------------- | ----------------------------------------------------------------- | -------- | ---------------------- |
+| METRON_USER | Run the application as this user | Optional | metron |
+| METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
+| METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
+| METRON_REST_PORT | REST application port | Optional | 8082 |
+| METRON_JVMFLAGS | JVM flags added to the start command | Optional | |
+| METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see [below](#spring-profiles)) | Optional | |
+| METRON_JDBC_DRIVER | JDBC driver class | Required | |
+| METRON_JDBC_URL | JDBC url | Required | |
+| METRON_JDBC_USERNAME | JDBC username | Required | |
+| METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle | Required | |
+| METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
+| METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
+| METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
+| METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional | |
+| ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) | Required | |
+| BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) | Required | |
+| HDFS_URL | HDFS url or `fs.defaultFS` Hadoop setting (ex. hdfs://node1:8020) | Required | |
+| SECURITY_ENABLED | Enables Kerberos support | Optional | false |
+| METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional | |
+| METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional | |
+
+These are set in the `/etc/sysconfig/metron` file.
+
+## Database setup
+
+The REST application persists data in a relational database and requires a dedicated database user and database (see https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html for more detail).
+
+### Development
+
+The REST application comes with embedded database support for development purposes (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support).
+
+For example, edit these variables in `/etc/sysconfig/metron` before starting the application to configure H2:
+```
+METRON_JDBC_DRIVER="org.h2.Driver"
+METRON_JDBC_URL="jdbc:h2:file:~/metrondb"
+METRON_JDBC_USERNAME="root"
+METRON_JDBC_PASSWORD='root"
+METRON_JDBC_PLATFORM="h2"
+```
+
+### Production
+
+The REST application should be configured with a production-grade database outside of development.
+
+For example, the following configures the application for MySQL:
+
+1. Install MySQL if not already available (this example uses version 5.7, installation instructions can be found [here](https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html))
-1. Untar the archive in the target directory. The directory structure will look like:
- ```
- bin
- start_metron_rest.sh
- lib
- metron-rest-$METRON_VERSION.jar
- ```
+1. Create a metron user and REST database and permission the user for that database:
+```
+CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd';
+CREATE DATABASE IF NOT EXISTS metronrest;
+GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
+```
-1. Create an `application.yml` file with the contents of [application-vagrant.yml](src/main/resources/application-vagrant.yml). Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing `node1` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details).
+1. Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:
+```
+cd $METRON_HOME/lib
+wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
+tar xf mysql-connector-java-5.1.41.tar.gz
+```
-1. Start the application with this command:
- ```
- ./bin/start_metron_rest.sh /path/to/application.yml
- ```
+1. Edit these variables in `/etc/sysconfig/metron` to configure the REST application for MySQL:
+```
+METRON_JDBC_DRIVER="com.mysql.jdbc.Driver"
+METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest"
+METRON_JDBC_USERNAME="metron"
+METRON_JDBC_PLATFORM="mysql"
+METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
+```
## Usage
-The exposed REST endpoints can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8080 but can be changed in application.yml by setting "server.port" to the desired port.
+After configuration is complete, the REST application can be managed as a service:
+```
+service metron-rest start
+```
+
+If a production database is configured, the JDBC password should be passed in as the first argument on startup:
+```
+service metron-rest start Myp@ssw0rd
+```
+
+The REST application can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8082.
## Security
-The metron-rest module uses [Spring Security](http://projects.spring.io/spring-security/) for authentication and stores user credentials in a relational database. The H2 database is configured by default and is intended only for development purposes. The "dev" profile can be used to automatically load test users:
+### Authentication
+
+The metron-rest module uses [Spring Security](http://projects.spring.io/spring-security/) for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:
```
-./bin/start_metron_rest.sh /path/to/application.yml --spring.profiles.active=dev
+use metronrest;
+insert into users (username, password, enabled) values ('your_username','your_password',1);
+insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
```
-For [production use](http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#boot-features-connect-to-production-database), a relational database should be configured. For example, configuring MySQL would be done as follows:
-
-1. Create a MySQL user for the Metron REST application (http://dev.mysql.com/doc/refman/5.7/en/adding-users.html).
+### Kerberos
-1. Connect to MySQL and create a Metron REST database:
- ```
- CREATE DATABASE IF NOT EXISTS metronrest
- ```
+Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the `/etc/sysconfig/metron` file:
+```
+SECURITY_ENABLED=true
+METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
+METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM"
+METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
+```
-1. Add users:
- ```
- use metronrest;
- insert into users (username, password, enabled) values ('your_username','your_password',1);
- insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
- ```
+## Spring Profiles
-1. Replace the H2 connection information in the application.yml file with MySQL connection information:
- ```
- spring:
- datasource:
- driverClassName: com.mysql.jdbc.Driver
- url: jdbc:mysql://mysql_host:3306/metronrest
- username: metron_rest_user
- password: metron_rest_password
- platform: mysql
- ```
+The REST application comes with a few [Spring Profiles](http://docs.spring.io/autorepo/docs/spring-boot/current/reference/html/boot-features-profiles.html) to aid in testing and development.
-1. Add a dependency for the MySQL JDBC connector in the metron-rest pom.xml:
- ```
- <dependency>
- <groupId>mysql</groupId>
- <artifactId>mysql-connector-java</artifactId>
- <version>${mysql.client.version}</version>
- </dependency>
- ```
+| Profile | Description |
+| ------------------------ | --------------------------------------------- |
+| test | sets variables to in-memory services, only used for integration testing |
+| dev | adds a test user to the database with credentials `user/password` |
+| vagrant | sets configuration variables to match the Metron vagrant environment |
+| docker | sets configuration variables to match the Metron dcoker environment |
-1. Follow the steps in the [Installation](#installation) section
+Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in `/etc/sysconfig/metron` to configure the REST application for the Vagrant environment and add a test user:
+```
+METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+```
## API
@@ -100,7 +189,8 @@ Request and Response objects are JSON formatted. The JSON schemas are available
| [ `DELETE /api/v1/kafka/topic/{name}`](#delete-apiv1kafkatopicname)|
| [ `GET /api/v1/kafka/topic/{name}/sample`](#get-apiv1kafkatopicnamesample)|
| [ `GET /api/v1/sensor/enrichment/config`](#get-apiv1sensorenrichmentconfig)|
-| [ `GET /api/v1/sensor/enrichment/config/list/available`](#get-apiv1sensorenrichmentconfiglistavailable)|
+| [ `GET /api/v1/sensor/enrichment/config/list/available/enrichments`](#get-apiv1sensorenrichmentconfiglistavailableenrichments)|
+| [ `GET /api/v1/sensor/enrichment/config/list/available/threat/triage/aggregators`](#get-apiv1sensorenrichmentconfiglistavailablethreattriageaggregators)|
| [ `DELETE /api/v1/sensor/enrichment/config/{name}`](#delete-apiv1sensorenrichmentconfigname)|
| [ `POST /api/v1/sensor/enrichment/config/{name}`](#post-apiv1sensorenrichmentconfigname)|
| [ `GET /api/v1/sensor/enrichment/config/{name}`](#get-apiv1sensorenrichmentconfigname)|
@@ -203,11 +293,11 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* 404 - File was not found in HDFS
### `GET /api/v1/hdfs/list`
- * Description: Reads a file from HDFS and returns the contents
+ * Description: Lists an HDFS directory
* Input:
* path - Path to HDFS directory
* Returns:
- * 200 - Returns file contents
+ * 200 - HDFS directory list
### `GET /api/v1/kafka/topic`
* Description: Retrieves all Kafka topics
@@ -250,11 +340,16 @@ Request and Response objects are JSON formatted. The JSON schemas are available
* Returns:
* 200 - Returns all SensorEnrichmentConfigs
-### `GET /api/v1/sensor/enrichment/config/list/available`
+### `GET /api/v1/sensor/enrichment/config/list/available/enrichments`
* Description: Lists the available enrichments
* Returns:
* 200 - Returns a list of available enrichments
+### `GET /api/v1/sensor/enrichment/config/list/available/threat/triage/aggregators`
+ * Description: Lists the available threat triage aggregators
+ * Returns:
+ * 200 - Returns a list of available threat triage aggregators
+
### `DELETE /api/v1/sensor/enrichment/config/{name}`
* Description: Deletes a SensorEnrichmentConfig from Zookeeper
* Input:
@@ -515,38 +610,20 @@ mvn spring-boot:run -Drun.profiles=vagrant,dev
```
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
-To run the application locally on the Quick Dev host, package the application and scp the archive to node1:
-```
-mvn clean package
-scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME
-```
-Login to node1 and unarchive the metron-rest application:
-```
-ssh root@node1
-cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz
-```
-Start the application on a different port to avoid conflicting with Ambari:
+To run the application locally on the Quick Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/sysconfig/metron`:
```
-java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
-In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls:
+and start the application:
```
-sudo su -
-export ZOOKEEPER=node1
-export BROKERLIST=node1
-export HDP_HOME="/usr/hdp/current"
-export METRON_VERSION="0.4.0"
-export METRON_HOME="/usr/metron/${METRON_VERSION}"
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest
+service metron-rest start
```
-Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support:
+In a cluster with Kerberos enabled, update the security settings in `/etc/sysconfig/metron`. Security is disabled by default in the `vagrant` Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:
```
-su metron
-cd ~
-java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true
+METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
+METRON_SPRING_OPTIONS="--kerberos.enabled=true"
```
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/main/config/rest_application.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/config/rest_application.yml b/metron-interface/metron-rest/src/main/config/rest_application.yml
new file mode 100644
index 0000000..dbb0b43
--- /dev/null
+++ b/metron-interface/metron-rest/src/main/config/rest_application.yml
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+spring:
+ datasource:
+ driverClassName: ${METRON_JDBC_DRIVER}
+ url: ${METRON_JDBC_URL}
+ username: ${METRON_JDBC_USERNAME}
+ password: ${METRON_JDBC_PASSWORD}
+ platform: ${METRON_JDBC_PLATFORM}
+ continue-on-error: true
+
+zookeeper:
+ url: ${ZOOKEEPER}
+
+kafka:
+ broker:
+ url: ${BROKERLIST}
+
+hdfs:
+ namenode:
+ url: ${HDFS_URL}
+
+grok:
+ path:
+ temp: ${METRON_TEMP_GROK_PATH}
+ default: ${METRON_DEFAULT_GROK_PATH}
+
+storm:
+ ui:
+ url: ${STORM_REST_URL}
+ parser:
+ script.path: ${METRON_HOME}/bin/start_parser_topology.sh
+ enrichment:
+ script.path: ${METRON_HOME}/bin/start_enrichment_topology.sh
+ indexing:
+ script.path: ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+
+kerberos:
+ enabled: ${SECURITY_ENABLED}
+ principal: ${METRON_PRINCIPAL_NAME}
+ keytab: ${METRON_SERVICE_KEYTAB}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/HdfsController.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/HdfsController.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/HdfsController.java
index c8ff0b6..b8957a9 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/HdfsController.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/HdfsController.java
@@ -44,8 +44,8 @@ public class HdfsController {
@Autowired
private HdfsService hdfsService;
- @ApiOperation(value = "Reads a file from HDFS and returns the contents")
- @ApiResponse(message = "Returns file contents", code = 200)
+ @ApiOperation(value = "Lists an HDFS directory")
+ @ApiResponse(message = "HDFS directory list", code = 200)
@RequestMapping(value = "/list", method = RequestMethod.GET)
ResponseEntity<List<String>> list(@ApiParam(name = "path", value = "Path to HDFS directory", required = true) @RequestParam String path) throws RestException {
return new ResponseEntity<>(hdfsService.list(new Path(path)), HttpStatus.OK);
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
index 158886f..c4964c0 100644
--- a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
+++ b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
@@ -16,7 +16,7 @@
spring:
datasource:
driverClassName: org.h2.Driver
- url: jdbc:h2:file:./metrondb
+ url: jdbc:h2:file:~/metrondb
username: root
password: root
platform: h2
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/main/resources/schema-mysql.sql
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/resources/schema-mysql.sql b/metron-interface/metron-rest/src/main/resources/schema-mysql.sql
index 986b652..638b7c5 100644
--- a/metron-interface/metron-rest/src/main/resources/schema-mysql.sql
+++ b/metron-interface/metron-rest/src/main/resources/schema-mysql.sql
@@ -14,9 +14,6 @@
See the License for the specific language governing permissions and
limitations under the License.
*/
-drop table if exists authorities;
-drop table if exists users;
-
create table if not exists users(
username varchar(50) not null primary key,
password varchar(50) not null,
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/main/scripts/metron-rest
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/scripts/metron-rest b/metron-interface/metron-rest/src/main/scripts/metron-rest
new file mode 100644
index 0000000..f9227a9
--- /dev/null
+++ b/metron-interface/metron-rest/src/main/scripts/metron-rest
@@ -0,0 +1,160 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# metron rest application service
+# chkconfig: - 20 80
+# description: rest application
+# processname: metron-rest
+#
+if [ -f /etc/rc.d/init.d/functions ]; then
+ . /etc/rc.d/init.d/functions
+fi
+
+NAME=metron-rest
+DESC="Metron REST Application"
+METRON_VERSION=${project.version}
+METRON_HOME=/usr/metron/$METRON_VERSION
+METRON_LOG_DIR="/var/log/metron"
+METRON_PID_DIR="/var/run/metron"
+METRON_USER="metron"
+METRON_GROUP="metron"
+METRON_REST_PORT=8082
+METRON_JDBC_PASSWORD="$2"
+METRON_SYSCONFIG="/etc/sysconfig/metron"
+if [ -f "$METRON_SYSCONFIG" ]; then
+ set -a
+ . "$METRON_SYSCONFIG"
+fi
+
+PIDFILE="$METRON_PID_DIR/$NAME.pid"
+LOCKFILE=/var/lock/subsys/$NAME
+
+# the vagrant Spring profile provides configuration values, otherwise configuration is provided by rest_application.yml
+if [[ !($METRON_SPRING_PROFILES_ACTIVE == *"vagrant"*) ]]; then
+ METRON_SPRING_OPTIONS+=" --spring.config.location=$METRON_HOME/config/rest_application.yml"
+fi
+METRON_SPRING_OPTIONS+=" --server.port=$METRON_REST_PORT"
+if [ $METRON_SPRING_PROFILES_ACTIVE ]; then
+ METRON_SPRING_OPTIONS+=" --spring.profiles.active=$METRON_SPRING_PROFILES_ACTIVE"
+fi
+if [ $METRON_JDBC_PASSWORD ]; then
+ METRON_SPRING_OPTIONS+=" --spring.datasource.password=$METRON_JDBC_PASSWORD"
+fi
+if [ $METRON_JDBC_CLIENT_PATH ]; then
+ METRON_JVMFLAGS+=" -Dloader.path=$METRON_JDBC_CLIENT_PATH"
+fi
+DAEMON="java $METRON_JVMFLAGS -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar $METRON_SPRING_OPTIONS"
+
+#
+# start the rest application
+#
+start() {
+
+ # if pidfile exists, do not start another
+ if [ -f $PIDFILE ]; then
+ PID=`cat $PIDFILE`
+ printf "OK [$PID]\n"
+ return
+ fi
+
+ if [ ! -d "$METRON_LOG_DIR" ]; then
+ mkdir -p "$METRON_LOG_DIR" && chown "$METRON_USER":"$METRON_GROUP" "$METRON_LOG_DIR"
+ fi
+
+ if [ ! -d "$METRON_PID_DIR" ]; then
+ mkdir -p "$METRON_PID_DIR" && chown "$METRON_USER":"$METRON_GROUP" "$METRON_PID_DIR"
+ fi
+
+ # kick-off the daemon
+ PID=`runuser -s /bin/bash $METRON_USER -c "$DAEMON" >> $METRON_LOG_DIR/$NAME.log 2>&1 & echo $!`
+
+ if [ -z $PID ]; then
+ printf "Fail\n"
+ else
+ echo $PID > $PIDFILE && touch $LOCKFILE
+ printf "Ok [$PID]\n"
+ fi
+}
+
+#
+# stop the rest application
+#
+stop() {
+ if [ -f $PIDFILE ]; then
+ PID=`cat $PIDFILE`
+ killproc -p $PIDFILE metron-rest
+ RETVAL=$?
+ echo
+ [ $RETVAL -eq 0 ] && rm -f $LOCKFILE
+ printf "Stopped [$PID]\n"
+ else
+ printf "Not running\n"
+ fi
+}
+
+#
+# status check of the rest application
+#
+rest_status() {
+ status -p $PIDFILE metron-rest
+}
+
+case "$1" in
+
+ ##############################################################################
+ # start
+ #
+ start)
+ printf "%-50s \n" "Starting $NAME..."
+ start
+ ;;
+
+ ##############################################################################
+ # status
+ #
+ status)
+ printf "%-50s \n" "Checking $NAME..."
+ rest_status
+ ;;
+
+ ##############################################################################
+ # stop
+ #
+ stop)
+ printf "%-50s \n" "Stopping $NAME..."
+ stop
+ ;;
+
+ ##############################################################################
+ # restart
+ #
+ restart)
+ $0 stop
+ $0 start
+ ;;
+
+ ##############################################################################
+ # reload
+ #
+ reload)
+ ;;
+
+ *)
+ echo "Usage: $0 {status|start|stop|restart}"
+ exit 1
+esac
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/main/scripts/start_metron_rest.sh
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/scripts/start_metron_rest.sh b/metron-interface/metron-rest/src/main/scripts/start_metron_rest.sh
deleted file mode 100755
index ea5d6c9..0000000
--- a/metron-interface/metron-rest/src/main/scripts/start_metron_rest.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-METRON_VERSION=${project.version}
-SCRIPTS_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-if [ "$#" -lt 1 ]; then
- echo "Usage: start.sh <path to application.yml file> <spring options>"
- echo "Path can be absolute or relative to $SCRIPTS_ROOT"
-else
- java -jar $SCRIPTS_ROOT/../lib/metron-rest-$METRON_VERSION.jar --spring.config.location=$@
-fi
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/716bda32/metron-interface/metron-rest/src/test/resources/README.vm
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/resources/README.vm b/metron-interface/metron-rest/src/test/resources/README.vm
index cff7ca7..2d8fd75 100644
--- a/metron-interface/metron-rest/src/test/resources/README.vm
+++ b/metron-interface/metron-rest/src/test/resources/README.vm
@@ -7,76 +7,165 @@ This module provides a RESTful API for interacting with Metron.
* A running Metron cluster
* Java 8 installed
* Storm CLI and Metron topology scripts (start_parser_topology.sh, start_enrichment_topology.sh, start_elasticsearch_topology.sh) installed
+* A relational database
#[[##]]# Installation
+
+#[[###]]# From Source
+
1. Package the application with Maven:
- ```
- mvn clean package
- ```
+```
+mvn clean package
+```
+
+1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
+```
+config
+ rest_application.yml
+bin
+ metron-rest
+lib
+ metron-rest-$METRON_VERSION.jar
+```
+
+1. Copy the `$METRON_HOME/bin/metron-rest` script to `/etc/init.d/metron-rest`
+
+#[[###]]# From Package Manager
+
+1. Deploy the RPM at `/incubator-metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-rest-$METRON_VERSION-*.noarch.rpm`
+
+1. Install the RPM with:
+ ```
+ rpm -ih metron-rest-$METRON_VERSION-*.noarch.rpm
+ ```
+
+#[[##]]# Configuration
+
+The REST application depends on several configuration parameters:
+
+| Environment Variable | Description | Required | Default |
+| ------------------------------------- | ----------------------------------------------------------------- | -------- | ---------------------- |
+| METRON_USER | Run the application as this user | Optional | metron |
+| METRON_LOG_DIR | Directory where the log file is written | Optional | /var/log/metron/ |
+| METRON_PID_DIR | Directory where the pid file is written | Optional | /var/run/metron/ |
+| METRON_REST_PORT | REST application port | Optional | 8082 |
+| METRON_JVMFLAGS | JVM flags added to the start command | Optional | |
+| METRON_SPRING_PROFILES_ACTIVE | Active Spring profiles (see [below](#spring-profiles)) | Optional | |
+| METRON_JDBC_DRIVER | JDBC driver class | Required | |
+| METRON_JDBC_URL | JDBC url | Required | |
+| METRON_JDBC_USERNAME | JDBC username | Required | |
+| METRON_JDBC_PLATFORM | JDBC platform (one of h2, mysql, postgres, oracle | Required | |
+| METRON_JDBC_CLIENT_PATH | Path to JDBC client jar | Optional | H2 is bundled |
+| METRON_TEMP_GROK_PATH | Temporary directory used to test grok statements | Optional | ./patterns/temp |
+| METRON_DEFAULT_GROK_PATH | Defaults HDFS directory used to store grok statements | Optional | /apps/metron/patterns |
+| METRON_SPRING_OPTIONS | Additional Spring input parameters | Optional | |
+| ZOOKEEPER | Zookeeper quorum (ex. node1:2181,node2:2181) | Required | |
+| BROKERLIST | Kafka Broker list (ex. node1:6667,node2:6667) | Required | |
+| HDFS_URL | HDFS url or `fs.defaultFS` Hadoop setting (ex. hdfs://node1:8020) | Required | |
+| SECURITY_ENABLED | Enables Kerberos support | Optional | false |
+| METRON_PRINCIPAL_NAME | Kerberos principal for the metron user | Optional | |
+| METRON_SERVICE_KEYTAB | Path to the Kerberos keytab for the metron user | Optional | |
+
+These are set in the `/etc/sysconfig/metron` file.
+
+#[[##]]# Database setup
+
+The REST application persists data in a relational database and requires a dedicated database user and database (see https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html for more detail).
+
+#[[###]]# Development
+
+The REST application comes with embedded database support for development purposes (https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-sql.html#boot-features-embedded-database-support).
+
+For example, edit these variables in `/etc/sysconfig/metron` before starting the application to configure H2:
+```
+METRON_JDBC_DRIVER="org.h2.Driver"
+METRON_JDBC_URL="jdbc:h2:file:~/metrondb"
+METRON_JDBC_USERNAME="root"
+METRON_JDBC_PASSWORD='root"
+METRON_JDBC_PLATFORM="h2"
+```
+
+#[[###]]# Production
+
+The REST application should be configured with a production-grade database outside of development.
-1. Untar the archive in the target directory. The directory structure will look like:
- ```
- bin
- start_metron_rest.sh
- lib
- metron-rest-$METRON_VERSION.jar
- ```
+For example, the following configures the application for MySQL:
-1. Create an `application.yml` file with the contents of [application-vagrant.yml](src/main/resources/application-vagrant.yml). Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing `node1` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details).
+1. Install MySQL if not already available (this example uses version 5.7, installation instructions can be found [here](https://dev.mysql.com/doc/refman/5.7/en/linux-installation-yum-repo.html))
-1. Start the application with this command:
- ```
- ./bin/start_metron_rest.sh /path/to/application.yml
- ```
+1. Create a metron user and REST database and permission the user for that database:
+```
+CREATE USER 'metron'@'node1' IDENTIFIED BY 'Myp@ssw0rd';
+CREATE DATABASE IF NOT EXISTS metronrest;
+GRANT ALL PRIVILEGES ON metronrest.* TO 'metron'@'node1';
+```
+
+1. Install the MySQL JDBC client onto the REST application host and configurate the METRON_JDBC_CLIENT_PATH variable:
+```
+cd $METRON_HOME/lib
+wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.41.tar.gz
+tar xf mysql-connector-java-5.1.41.tar.gz
+```
+
+1. Edit these variables in `/etc/sysconfig/metron` to configure the REST application for MySQL:
+```
+METRON_JDBC_DRIVER="com.mysql.jdbc.Driver"
+METRON_JDBC_URL="jdbc:mysql://mysql_host:3306/metronrest"
+METRON_JDBC_USERNAME="metron"
+METRON_JDBC_PLATFORM="mysql"
+METRON_JDBC_CLIENT_PATH=$METRON_HOME/lib/mysql-connector-java-5.1.41/mysql-connector-java-5.1.41-bin.jar
+```
#[[##]]# Usage
-The exposed REST endpoints can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8080 but can be changed in application.yml by setting "server.port" to the desired port.
+After configuration is complete, the REST application can be managed as a service:
+```
+service metron-rest start
+```
+
+If a production database is configured, the JDBC password should be passed in as the first argument on startup:
+```
+service metron-rest start Myp@ssw0rd
+```
+
+The REST application can be accessed with the Swagger UI at http://host:port/swagger-ui.html#/. The default port is 8082.
#[[##]]# Security
-The metron-rest module uses [Spring Security](http://projects.spring.io/spring-security/) for authentication and stores user credentials in a relational database. The H2 database is configured by default and is intended only for development purposes. The "dev" profile can be used to automatically load test users:
+#[[###]]# Authentication
+
+The metron-rest module uses [Spring Security](http://projects.spring.io/spring-security/) for authentication and stores user credentials in the relational database configured above. The required tables are created automatically the first time the application is started so that should be done first. For example (continuing the MySQL example above), users can be added by connecting to MySQL and running:
```
-./bin/start_metron_rest.sh /path/to/application.yml --spring.profiles.active=dev
+use metronrest;
+insert into users (username, password, enabled) values ('your_username','your_password',1);
+insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
```
-For [production use](http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/reference/htmlsingle/#boot-features-connect-to-production-database), a relational database should be configured. For example, configuring MySQL would be done as follows:
-
-1. Create a MySQL user for the Metron REST application (http://dev.mysql.com/doc/refman/5.7/en/adding-users.html).
+#[[###]]# Kerberos
-1. Connect to MySQL and create a Metron REST database:
- ```
- CREATE DATABASE IF NOT EXISTS metronrest
- ```
+Metron REST can be configured for a cluster with Kerberos enabled. A client JAAS file is required for Kafka and Zookeeper and a Kerberos keytab for the metron user principal is required for all other services. Configure these settings in the `/etc/sysconfig/metron` file:
+```
+SECURITY_ENABLED=true
+METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
+METRON_PRINCIPAL_NAME="metron@EXAMPLE.COM"
+METRON_SERVICE_KEYTAB="/etc/security/keytabs/metron.keytab"
+```
-1. Add users:
- ```
- use metronrest;
- insert into users (username, password, enabled) values ('your_username','your_password',1);
- insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
- ```
+#[[##]]# Spring Profiles
-1. Replace the H2 connection information in the application.yml file with MySQL connection information:
- ```
- spring:
- datasource:
- driverClassName: com.mysql.jdbc.Driver
- url: jdbc:mysql://mysql_host:3306/metronrest
- username: metron_rest_user
- password: metron_rest_password
- platform: mysql
- ```
+The REST application comes with a few [Spring Profiles](http://docs.spring.io/autorepo/docs/spring-boot/current/reference/html/boot-features-profiles.html) to aid in testing and development.
-1. Add a dependency for the MySQL JDBC connector in the metron-rest pom.xml:
- ```
- <dependency>
- <groupId>mysql</groupId>
- <artifactId>mysql-connector-java</artifactId>
- <version>${mysql.client.version}</version>
- </dependency>
- ```
+| Profile | Description |
+| ------------------------ | --------------------------------------------- |
+| test | sets variables to in-memory services, only used for integration testing |
+| dev | adds a test user to the database with credentials `user/password` |
+| vagrant | sets configuration variables to match the Metron vagrant environment |
+| docker | sets configuration variables to match the Metron dcoker environment |
-1. Follow the steps in the [Installation](#installation) section
+Setting active profiles is done with the METRON_SPRING_PROFILES_ACTIVE variable. For example, set this variable in `/etc/sysconfig/metron` to configure the REST application for the Vagrant environment and add a test user:
+```
+METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+```
#[[##]]# API
@@ -125,38 +214,20 @@ mvn spring-boot:run -Drun.profiles=vagrant,dev
```
The metron-rest application will be available at http://localhost:8080/swagger-ui.html#/.
-To run the application locally on the Quick Dev host, package the application and scp the archive to node1:
-```
-mvn clean package
-scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME
-```
-Login to node1 and unarchive the metron-rest application:
-```
-ssh root@node1
-cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz
-```
-Start the application on a different port to avoid conflicting with Ambari:
+To run the application locally on the Quick Dev host (node1), follow the [Installation](#installation) instructions above. Then set the METRON_SPRING_PROFILES_ACTIVE variable in `/etc/sysconfig/metron`:
```
-java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082
+METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
```
-In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls:
+and start the application:
```
-sudo su -
-export ZOOKEEPER=node1
-export BROKERLIST=node1
-export HDP_HOME="/usr/hdp/current"
-export METRON_VERSION="0.4.0"
-export METRON_HOME="/usr/metron/${METRON_VERSION}"
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets
-${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest
+service metron-rest start
```
-Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support:
+In a cluster with Kerberos enabled, update the security settings in `/etc/sysconfig/metron`. Security is disabled by default in the `vagrant` Spring profile so that setting must be overriden with the METRON_SPRING_OPTIONS variable:
```
-su metron
-cd ~
-java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true
+METRON_SPRING_PROFILES_ACTIVE="vagrant,dev"
+METRON_JVMFLAGS="-Djava.security.auth.login.config=$METRON_HOME/client_jaas.conf"
+METRON_SPRING_OPTIONS="--kerberos.enabled=true"
```
The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
[23/44] metron git commit: METRON-934: Component and task id are
missing in the indexing topology Hdfs file names. closes
apache/incubator-metron#574
Posted by ma...@apache.org.
METRON-934: Component and task id are missing in the indexing topology Hdfs file names. closes apache/incubator-metron#574
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/1277b6c3
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/1277b6c3
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/1277b6c3
Branch: refs/heads/Metron_0.4.0
Commit: 1277b6c3229ee4b7cb066902ac56e03145c18157
Parents: 4b3ccac
Author: dlyle65535 <dl...@gmail.com>
Authored: Tue May 16 10:25:09 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Tue May 16 10:25:09 2017 -0400
----------------------------------------------------------------------
.../metron/common/writer/BulkMessageWriter.java | 3 +-
.../writer/ElasticsearchWriter.java | 3 +-
.../writer/SimpleHbaseEnrichmentWriter.java | 3 +-
.../bolt/BulkMessageWriterBoltTest.java | 5 ++-
.../apache/metron/parsers/bolt/ParserBolt.java | 2 +-
.../apache/metron/parsers/bolt/WriterBolt.java | 2 +-
.../metron/parsers/bolt/WriterHandler.java | 5 ++-
.../metron/parsers/bolt/ParserBoltTest.java | 15 ++++---
.../metron/parsers/bolt/WriterBoltTest.java | 8 ++--
.../apache/metron/solr/writer/SolrWriter.java | 3 +-
.../metron/solr/writer/SolrWriterTest.java | 6 +--
.../org/apache/metron/writer/NoopWriter.java | 3 +-
.../metron/writer/WriterToBulkWriter.java | 3 +-
.../writer/bolt/BulkMessageWriterBolt.java | 6 ++-
.../apache/metron/writer/hdfs/HdfsWriter.java | 4 +-
.../metron/writer/hdfs/HdfsWriterTest.java | 47 +++++++++++++-------
16 files changed, 74 insertions(+), 44 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-common/src/main/java/org/apache/metron/common/writer/BulkMessageWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/writer/BulkMessageWriter.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/writer/BulkMessageWriter.java
index 5f427f4..64ba0cc 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/writer/BulkMessageWriter.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/writer/BulkMessageWriter.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.common.writer;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.metron.common.configuration.Configurations;
import org.apache.metron.common.configuration.EnrichmentConfigurations;
@@ -28,7 +29,7 @@ import java.util.Map;
public interface BulkMessageWriter<MESSAGE_T> extends AutoCloseable, Serializable {
- void init(Map stormConf, WriterConfiguration config) throws Exception;
+ void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration config) throws Exception;
/**
* Writes the messages to a particular output (e.g. Elasticsearch). Exceptions trigger failure of the entire batch.
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
index c90bc8c..a643b44 100644
--- a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
+++ b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.elasticsearch.writer;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableList;
@@ -57,7 +58,7 @@ public class ElasticsearchWriter implements BulkMessageWriter<JSONObject>, Seria
}
@Override
- public void init(Map stormConf, WriterConfiguration configurations) {
+ public void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration configurations) {
Map<String, Object> globalConfiguration = configurations.getGlobalConfig();
Settings.Builder settingsBuilder = Settings.settingsBuilder();
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/writer/SimpleHbaseEnrichmentWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/writer/SimpleHbaseEnrichmentWriter.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/writer/SimpleHbaseEnrichmentWriter.java
index 343eecd..83c7151 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/writer/SimpleHbaseEnrichmentWriter.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/writer/SimpleHbaseEnrichmentWriter.java
@@ -18,6 +18,7 @@
package org.apache.metron.enrichment.writer;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
@@ -122,7 +123,7 @@ public class SimpleHbaseEnrichmentWriter extends AbstractWriter implements BulkM
}
@Override
- public void init(Map stormConf, WriterConfiguration configuration) throws Exception {
+ public void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration configuration) throws Exception {
if(converter == null) {
converter = new EnrichmentConverter();
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBoltTest.java
index 1f33060..79d8285 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/BulkMessageWriterBoltTest.java
@@ -23,6 +23,7 @@ import org.apache.metron.common.message.MessageGetters;
import org.apache.metron.common.writer.BulkWriterResponse;
import org.apache.metron.test.utils.UnitTestHelper;
import org.apache.metron.writer.BulkWriterComponent;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
import org.adrianwalker.multilinestring.Multiline;
@@ -127,7 +128,7 @@ public class BulkMessageWriterBoltTest extends BaseEnrichmentBoltTest {
bulkMessageWriterBolt.declareOutputFields(declarer);
verify(declarer, times(1)).declareStream(eq("error"), argThat(new FieldsMatcher("message")));
Map stormConf = new HashMap();
- doThrow(new Exception()).when(bulkMessageWriter).init(eq(stormConf), any(WriterConfiguration.class));
+ doThrow(new Exception()).when(bulkMessageWriter).init(eq(stormConf),any(TopologyContext.class), any(WriterConfiguration.class));
try {
bulkMessageWriterBolt.prepare(stormConf, topologyContext, outputCollector);
fail("A runtime exception should be thrown when bulkMessageWriter.init throws an exception");
@@ -135,7 +136,7 @@ public class BulkMessageWriterBoltTest extends BaseEnrichmentBoltTest {
reset(bulkMessageWriter);
when(bulkMessageWriter.getName()).thenReturn("hdfs");
bulkMessageWriterBolt.prepare(stormConf, topologyContext, outputCollector);
- verify(bulkMessageWriter, times(1)).init(eq(stormConf), any(WriterConfiguration.class));
+ verify(bulkMessageWriter, times(1)).init(eq(stormConf),any(TopologyContext.class), any(WriterConfiguration.class));
tupleList = new ArrayList<>();
for(int i = 0; i < 4; i++) {
when(tuple.getValueByField("message")).thenReturn(messageList.get(i));
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
index e73b836..2c43c23 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/ParserBolt.java
@@ -97,7 +97,7 @@ public class ParserBolt extends ConfiguredParserBolt implements Serializable {
parser.init();
- writer.init(stormConf, collector, getConfigurations());
+ writer.init(stormConf, context, collector, getConfigurations());
SensorParserConfig config = getSensorParserConfig();
if(config != null) {
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterBolt.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterBolt.java
index ef7288b..ef93ba2 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterBolt.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterBolt.java
@@ -55,7 +55,7 @@ public class WriterBolt extends BaseRichBolt {
public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
this.collector = collector;
messageGetStrategy = MessageGetters.DEFAULT_JSON_FROM_FIELD.get();
- handler.init(stormConf, collector, configuration);
+ handler.init(stormConf, context, collector, configuration);
}
private JSONObject getMessage(Tuple tuple) {
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterHandler.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterHandler.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterHandler.java
index 3273ca7..2192942 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterHandler.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/bolt/WriterHandler.java
@@ -20,6 +20,7 @@ package org.apache.metron.parsers.bolt;
import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.metron.common.configuration.ParserConfigurations;
import org.apache.metron.common.configuration.writer.ParserWriterConfiguration;
@@ -57,7 +58,7 @@ public class WriterHandler implements Serializable {
return isBulk;
}
- public void init(Map stormConf, OutputCollector collector, ParserConfigurations configurations) {
+ public void init(Map stormConf, TopologyContext topologyContext, OutputCollector collector, ParserConfigurations configurations) {
if(isBulk) {
writerTransformer = config -> new ParserWriterConfiguration(config);
}
@@ -65,7 +66,7 @@ public class WriterHandler implements Serializable {
writerTransformer = config -> new SingleBatchConfigurationFacade(new ParserWriterConfiguration(config));
}
try {
- messageWriter.init(stormConf, writerTransformer.apply(configurations));
+ messageWriter.init(stormConf, topologyContext, writerTransformer.apply(configurations));
} catch (Exception e) {
throw new IllegalStateException("Unable to initialize message writer", e);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java
index e924d50..835f17e 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/ParserBoltTest.java
@@ -24,6 +24,7 @@ import org.apache.metron.common.error.MetronError;
import org.apache.metron.test.error.MetronErrorJSONMatcher;
import org.apache.metron.test.utils.UnitTestHelper;
import org.apache.storm.task.OutputCollector;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import com.google.common.collect.ImmutableList;
import org.apache.metron.common.configuration.writer.ParserWriterConfiguration;
@@ -91,7 +92,7 @@ public class ParserBoltTest extends BaseBoltTest {
List<JSONObject> records = new ArrayList<>();
@Override
- public void init(Map stormConf, WriterConfiguration config) throws Exception {
+ public void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration config) throws Exception {
}
@@ -297,7 +298,7 @@ public void testImplicitBatchOfOne() throws Exception {
parserBolt.setTreeCache(cache);
parserBolt.prepare(new HashMap(), topologyContext, outputCollector);
verify(parser, times(1)).init();
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
when(parser.validate(any())).thenReturn(true);
when(parser.parseOptional(any())).thenReturn(Optional.of(ImmutableList.of(new JSONObject())));
when(filter.emitTuple(any(), any(Context.class))).thenReturn(true);
@@ -343,7 +344,7 @@ public void testImplicitBatchOfOne() throws Exception {
parserBolt.setTreeCache(cache);
parserBolt.prepare(new HashMap(), topologyContext, outputCollector);
verify(parser, times(1)).init();
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
BulkWriterResponse successResponse = mock(BulkWriterResponse.class);
when(successResponse.getSuccesses()).thenReturn(ImmutableList.of(t1));
when(batchWriter.write(any(), any(), any(), any())).thenReturn(successResponse);
@@ -380,7 +381,7 @@ public void testImplicitBatchOfOne() throws Exception {
parserBolt.setTreeCache(cache);
parserBolt.prepare(new HashMap(), topologyContext, outputCollector);
verify(parser, times(1)).init();
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
when(parser.validate(any())).thenReturn(true);
when(parser.parseOptional(any())).thenReturn(Optional.of(ImmutableList.of(new JSONObject(new HashMap<String, Object>() {{
put("field2", "blah");
@@ -483,7 +484,7 @@ public void testImplicitBatchOfOne() throws Exception {
parserBolt.setTreeCache(cache);
parserBolt.prepare(new HashMap(), topologyContext, outputCollector);
verify(parser, times(1)).init();
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
when(parser.validate(any())).thenReturn(true);
when(parser.parseOptional(any())).thenReturn(Optional.of(ImmutableList.of(new JSONObject())));
when(filter.emitTuple(any(), any(Context.class))).thenReturn(true);
@@ -522,7 +523,7 @@ public void testImplicitBatchOfOne() throws Exception {
parserBolt.setTreeCache(cache);
parserBolt.prepare(new HashMap(), topologyContext, outputCollector);
verify(parser, times(1)).init();
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
when(parser.validate(any())).thenReturn(true);
when(parser.parseOptional(any())).thenReturn(Optional.of(ImmutableList.of(new JSONObject())));
when(filter.emitTuple(any(), any(Context.class))).thenReturn(true);
@@ -571,7 +572,7 @@ public void testImplicitBatchOfOne() throws Exception {
parserBolt.setTreeCache(cache);
parserBolt.prepare(new HashMap(), topologyContext, outputCollector);
verify(parser, times(1)).init();
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
doThrow(new Exception()).when(batchWriter).write(any(), any(), any(), any());
when(parser.validate(any())).thenReturn(true);
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/WriterBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/WriterBoltTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/WriterBoltTest.java
index 4511b55..a23c368 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/WriterBoltTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bolt/WriterBoltTest.java
@@ -92,7 +92,7 @@ public class WriterBoltTest extends BaseBoltTest{
}
WriterBolt bolt = new WriterBolt(new WriterHandler(batchWriter), configurations, sensorType);
bolt.prepare(new HashMap(), topologyContext, outputCollector);
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
for(int i = 0;i < 4;++i) {
Tuple t = tuples.get(i);
bolt.execute(t);
@@ -186,7 +186,7 @@ public class WriterBoltTest extends BaseBoltTest{
WriterBolt bolt = new WriterBolt(new WriterHandler(batchWriter), configurations, sensorType);
bolt.prepare(new HashMap(), topologyContext, outputCollector);
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
for(int i = 0;i < 4;++i) {
Tuple t = tuples.get(i);
@@ -232,7 +232,7 @@ public class WriterBoltTest extends BaseBoltTest{
WriterBolt bolt = new WriterBolt(new WriterHandler(batchWriter), configurations, sensorType);
bolt.prepare(new HashMap(), topologyContext, outputCollector);
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
for(int i = 0;i < 4;++i) {
Tuple t = tuples.get(i);
@@ -279,7 +279,7 @@ public class WriterBoltTest extends BaseBoltTest{
WriterBolt bolt = new WriterBolt(new WriterHandler(batchWriter), configurations, sensorType);
bolt.prepare(new HashMap(), topologyContext, outputCollector);
doThrow(new Exception()).when(batchWriter).write(any(), any(), any(), any());
- verify(batchWriter, times(1)).init(any(), any());
+ verify(batchWriter, times(1)).init(any(), any(), any());
for(int i = 0;i < 4;++i) {
Tuple t = tuples.get(i);
bolt.execute(t);
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java b/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
index 7180ca9..4e3246b 100644
--- a/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
+++ b/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/writer/SolrWriter.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.solr.writer;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.metron.common.configuration.Configurations;
import org.apache.metron.common.configuration.EnrichmentConfigurations;
@@ -56,7 +57,7 @@ public class SolrWriter implements BulkMessageWriter<JSONObject>, Serializable {
}
@Override
- public void init(Map stormConf, WriterConfiguration configurations) throws IOException, SolrServerException {
+ public void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration configurations) throws IOException, SolrServerException {
Map<String, Object> globalConfiguration = configurations.getGlobalConfig();
if(solr == null) solr = new MetronSolrClient((String) globalConfiguration.get("solr.zookeeper"));
String collection = getCollection(configurations);
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/writer/SolrWriterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/writer/SolrWriterTest.java b/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/writer/SolrWriterTest.java
index 14e5dcb..a56916f 100644
--- a/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/writer/SolrWriterTest.java
+++ b/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/writer/SolrWriterTest.java
@@ -107,7 +107,7 @@ public class SolrWriterTest {
String collection = "metron";
MetronSolrClient solr = Mockito.mock(MetronSolrClient.class);
SolrWriter writer = new SolrWriter().withMetronSolrClient(solr);
- writer.init(null, new IndexingWriterConfiguration("solr", configurations));
+ writer.init(null, null,new IndexingWriterConfiguration("solr", configurations));
verify(solr, times(1)).createCollection(collection, 1, 1);
verify(solr, times(1)).setDefaultCollection(collection);
@@ -120,7 +120,7 @@ public class SolrWriterTest {
globalConfig.put("solr.replicationFactor", replicationFactor);
configurations.updateGlobalConfig(globalConfig);
writer = new SolrWriter().withMetronSolrClient(solr);
- writer.init(null, new IndexingWriterConfiguration("solr", configurations));
+ writer.init(null, null, new IndexingWriterConfiguration("solr", configurations));
verify(solr, times(1)).createCollection(collection, numShards, replicationFactor);
verify(solr, times(1)).setDefaultCollection(collection);
@@ -130,7 +130,7 @@ public class SolrWriterTest {
verify(solr, times(0)).commit(collection);
writer = new SolrWriter().withMetronSolrClient(solr).withShouldCommit(true);
- writer.init(null, new IndexingWriterConfiguration("solr", configurations));
+ writer.init(null, null, new IndexingWriterConfiguration("solr", configurations));
writer.write("test", new IndexingWriterConfiguration("solr", configurations), new ArrayList<>(), messages);
verify(solr, times(2)).add(argThat(new SolrInputDocumentMatcher(message1.toJSONString().hashCode(), "test", 100, 100.0)));
verify(solr, times(2)).add(argThat(new SolrInputDocumentMatcher(message2.toJSONString().hashCode(), "test", 200, 200.0)));
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/NoopWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/NoopWriter.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/NoopWriter.java
index a31f48e..dbdd93d 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/NoopWriter.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/NoopWriter.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.writer;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import com.google.common.base.Splitter;
import com.google.common.collect.Iterables;
@@ -126,7 +127,7 @@ public class NoopWriter extends AbstractWriter implements BulkMessageWriter<JSON
}
@Override
- public void init(Map stormConf, WriterConfiguration config) throws Exception {
+ public void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration config) throws Exception {
}
@Override
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/WriterToBulkWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/WriterToBulkWriter.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/WriterToBulkWriter.java
index 47d9b02..7d7eae5 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/WriterToBulkWriter.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/WriterToBulkWriter.java
@@ -18,6 +18,7 @@
package org.apache.metron.writer;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import com.google.common.collect.Iterables;
import org.apache.metron.common.configuration.writer.SingleBatchConfigurationFacade;
@@ -40,7 +41,7 @@ public class WriterToBulkWriter<MESSAGE_T> implements BulkMessageWriter<MESSAGE_
this.messageWriter = messageWriter;
}
@Override
- public void init(Map stormConf, WriterConfiguration config) throws Exception {
+ public void init(Map stormConf, TopologyContext topologyContext, WriterConfiguration config) throws Exception {
messageWriter.init();
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
index 085ca5c..6e0c371 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
@@ -92,8 +92,10 @@ public class BulkMessageWriterBolt extends ConfiguredIndexingBolt {
configurationTransformation = x -> x;
}
try {
- bulkMessageWriter.init(stormConf
- , configurationTransformation.apply(new IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+ bulkMessageWriter.init(stormConf,
+ context,
+ configurationTransformation.apply(new IndexingWriterConfiguration(bulkMessageWriter.getName(),
+ getConfigurations()))
);
} catch (Exception e) {
throw new RuntimeException(e);
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
index 4800787..a86dfbc 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
@@ -24,6 +24,7 @@ import org.apache.metron.common.dsl.StellarFunctions;
import org.apache.metron.common.dsl.VariableResolver;
import org.apache.metron.common.stellar.StellarCompiler;
import org.apache.metron.common.stellar.StellarProcessor;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.metron.common.configuration.writer.WriterConfiguration;
import org.apache.metron.common.writer.BulkMessageWriter;
@@ -76,9 +77,10 @@ public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
}
@Override
- public void init(Map stormConfig, WriterConfiguration configurations) {
+ public void init(Map stormConfig, TopologyContext topologyContext, WriterConfiguration configurations) {
this.stormConfig = stormConfig;
this.stellarProcessor = new StellarProcessor();
+ this.fileNameFormat.prepare(stormConfig,topologyContext);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1277b6c3/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
index 0a4bdcb..6153ed2 100644
--- a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
+++ b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
@@ -23,6 +23,7 @@ import org.apache.metron.common.configuration.writer.IndexingWriterConfiguration
import org.apache.metron.common.configuration.writer.WriterConfiguration;
import org.apache.storm.hdfs.bolt.format.DefaultFileNameFormat;
import org.apache.storm.hdfs.bolt.format.FileNameFormat;
+import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.junit.Assert;
@@ -63,7 +64,7 @@ public class HdfsWriterTest {
public void testGetHdfsPathNull() {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(),createTopologyContext(), config);
JSONObject message = new JSONObject();
Object result = writer.getHdfsPathExtension(SENSOR_NAME,null, message);
@@ -76,7 +77,7 @@ public class HdfsWriterTest {
public void testGetHdfsPathEmptyString() {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
Object result = writer.getHdfsPathExtension(SENSOR_NAME, "", message);
@@ -89,7 +90,7 @@ public class HdfsWriterTest {
public void testGetHdfsPathConstant() {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
Object result = writer.getHdfsPathExtension(SENSOR_NAME, "'new'", message);
@@ -102,7 +103,7 @@ public class HdfsWriterTest {
public void testGetHdfsPathDirectVariable() {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
message.put("test.key", "test.value");
@@ -116,7 +117,7 @@ public class HdfsWriterTest {
public void testGetHdfsPathFormatConstant() {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
Object result = writer.getHdfsPathExtension(SENSOR_NAME, "FORMAT('/test/folder/')", message);
@@ -130,7 +131,7 @@ public class HdfsWriterTest {
IndexingConfigurations indexingConfig = new IndexingConfigurations();
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
message.put("test.key", "test.value");
@@ -142,12 +143,22 @@ public class HdfsWriterTest {
}
@Test
+ public void testSetsCorrectHdfsFilename() {
+ IndexingConfigurations indexingConfig = new IndexingConfigurations();
+ WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
+ HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
+ String filename = writer.fileNameFormat.getName(1,1);
+ Assert.assertEquals("prefix-Xcom-7-1-1.json", filename);
+ }
+
+ @Test
@SuppressWarnings("unchecked")
public void testGetHdfsPathMultipleFunctions() {
IndexingConfigurations indexingConfig = new IndexingConfigurations();
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
message.put("test.key", "test.value");
@@ -169,7 +180,7 @@ public class HdfsWriterTest {
IndexingConfigurations indexingConfig = new IndexingConfigurations();
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
message.put("test.key", "test.value");
@@ -182,7 +193,7 @@ public class HdfsWriterTest {
public void testGetHdfsPathNonString() {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, new IndexingConfigurations());
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
writer.getHdfsPathExtension(SENSOR_NAME, "{'key':'value'}", message);
@@ -195,7 +206,7 @@ public class HdfsWriterTest {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat)
.withMaxOpenFiles(maxFiles);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
for(int i = 0; i < maxFiles; i++) {
writer.getSourceHandler(SENSOR_NAME, Integer.toString(i));
@@ -209,7 +220,7 @@ public class HdfsWriterTest {
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
HdfsWriter writer = new HdfsWriter().withFileNameFormat(testFormat)
.withMaxOpenFiles(maxFiles);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
for(int i = 0; i < maxFiles+1; i++) {
writer.getSourceHandler(SENSOR_NAME, Integer.toString(i));
@@ -226,7 +237,7 @@ public class HdfsWriterTest {
HdfsWriter writer = new HdfsWriter().withFileNameFormat(format);
IndexingConfigurations indexingConfig = new IndexingConfigurations();
WriterConfiguration config = new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
JSONObject message = new JSONObject();
message.put("test.key", "test.value");
@@ -270,7 +281,7 @@ public class HdfsWriterTest {
.withExtension(".json")
.withPrefix("prefix-");
HdfsWriter writer = new HdfsWriter().withFileNameFormat(format);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
// These two messages will be routed to the same folder, because test.key is the same
JSONObject message = new JSONObject();
@@ -314,7 +325,7 @@ public class HdfsWriterTest {
.withExtension(".json")
.withPrefix("prefix-");
HdfsWriter writer = new HdfsWriter().withFileNameFormat(format);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
// These two messages will be routed to the same folder, because test.key is the same
JSONObject message = new JSONObject();
@@ -371,7 +382,7 @@ public class HdfsWriterTest {
.withExtension(".json")
.withPrefix("prefix-");
HdfsWriter writer = new HdfsWriter().withFileNameFormat(format);
- writer.init(new HashMap<String, String>(), config);
+ writer.init(new HashMap<String, String>(), createTopologyContext(), config);
// These two messages will be routed to the same folder, because test.key is the same
JSONObject message = new JSONObject();
@@ -408,4 +419,10 @@ public class HdfsWriterTest {
indexingConfig.updateSensorIndexingConfig(SENSOR_NAME, sensorIndexingConfig);
return new IndexingWriterConfiguration(WRITER_NAME, indexingConfig);
}
+
+ private TopologyContext createTopologyContext(){
+ Map<Integer, String> taskToComponent = new HashMap<Integer, String>();
+ taskToComponent.put(7, "Xcom");
+ return new TopologyContext(null, null, taskToComponent, null, null, null, null, null, 7, 6703, null, null, null, null, null, null);
+ }
}
[31/44] metron git commit: METRON-961: Spooky intermittent exceptions
in pcap during sync should be handled and explained closes
apache/incubator-metron#592
Posted by ma...@apache.org.
METRON-961: Spooky intermittent exceptions in pcap during sync should be handled and explained closes apache/incubator-metron#592
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/fee758b3
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/fee758b3
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/fee758b3
Branch: refs/heads/Metron_0.4.0
Commit: fee758b3fb2277067421a95054c3e328730decda
Parents: ce381f3
Author: cstella <ce...@gmail.com>
Authored: Sat May 20 09:55:42 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Sat May 20 09:55:42 2017 -0400
----------------------------------------------------------------------
.../metron/spout/pcap/PartitionHDFSWriter.java | 22 ++++++++++++++------
1 file changed, 16 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/fee758b3/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
index 86697db..ebd60dc 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
@@ -65,11 +65,10 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
})
,HDFS(new SyncHandler() {
@Override
- public void sync(FSDataOutputStream outputStream) throws IOException{
-
+ public void sync(FSDataOutputStream outputStream) throws IOException {
outputStream.hflush();
outputStream.hsync();
- ((HdfsDataOutputStream)outputStream).hsync(EnumSet.of(HdfsDataOutputStream.SyncFlag.UPDATE_LENGTH));
+ ((HdfsDataOutputStream) outputStream).hsync(EnumSet.of(HdfsDataOutputStream.SyncFlag.UPDATE_LENGTH));
}
})
,LOCAL(new SyncHandler() {
@@ -91,8 +90,13 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
}
@Override
- public void sync(FSDataOutputStream input) throws IOException {
- func.sync(input);
+ public void sync(FSDataOutputStream input) {
+ try {
+ func.sync(input);
+ }
+ catch(IOException ioe) {
+ LOG.warn("Problems during sync, but this shouldn't be too concerning as long as it's intermittent: " + ioe.getMessage(), ioe);
+ }
}
}
@@ -156,7 +160,13 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
public void handle(long ts, byte[] value) throws IOException {
turnoverIfNecessary(ts);
BytesWritable bw = new BytesWritable(value);
- writer.append(new LongWritable(ts), bw);
+ try {
+ writer.append(new LongWritable(ts), bw);
+ }
+ catch(ArrayIndexOutOfBoundsException aioobe) {
+ LOG.warn("This appears to be HDFS-7765 (https://issues.apache.org/jira/browse/HDFS-7765), " +
+ "which is an issue with syncing and not problematic: " + aioobe.getMessage(), aioobe);
+ }
numWritten++;
if(numWritten % config.getSyncEvery() == 0) {
syncHandler.sync(outputStream);
[24/44] metron git commit: METRON-950: Migrate storm-kafka-client to
1.1 closes apache/incubator-metron#584
Posted by ma...@apache.org.
METRON-950: Migrate storm-kafka-client to 1.1 closes apache/incubator-metron#584
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/6c836d13
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/6c836d13
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/6c836d13
Branch: refs/heads/Metron_0.4.0
Commit: 6c836d13635fcf1639ea7b8adbaf1ba4051c4f1c
Parents: 1277b6c
Author: cstella <ce...@gmail.com>
Authored: Tue May 16 11:02:43 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Tue May 16 11:02:43 2017 -0400
----------------------------------------------------------------------
.../parsers/topology/ParserTopologyBuilder.java | 3 +-
.../metron-storm-kafka-override/pom.xml | 101 +++++++++++++++++++
.../storm/kafka/spout/internal/Timer.java | 58 +++++++++++
.../storm/kafka/spout/internal/TimerTest.java | 36 +++++++
metron-platform/metron-storm-kafka/pom.xml | 5 +
.../kafka/flux/SimpleStormKafkaBuilder.java | 90 +++++++++--------
.../storm/kafka/flux/SpoutConfiguration.java | 20 ++--
.../kafka/flux/SpoutConfigurationTest.java | 13 +--
metron-platform/pom.xml | 1 +
pom.xml | 42 ++++----
10 files changed, 289 insertions(+), 80 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
index e9acbaa..196c19d 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.parsers.topology;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder;
import org.apache.metron.storm.kafka.flux.SpoutConfiguration;
import org.apache.metron.storm.kafka.flux.StormKafkaSpout;
@@ -124,7 +125,7 @@ public class ParserTopologyBuilder {
kafkaSpoutConfigOptions.putIfAbsent( SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key
, KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST.toString()
);
- kafkaSpoutConfigOptions.putIfAbsent( KafkaSpoutConfig.Consumer.GROUP_ID
+ kafkaSpoutConfigOptions.putIfAbsent( ConsumerConfig.GROUP_ID_CONFIG
, inputTopic + "_parser"
);
if(securityProtocol.isPresent()) {
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka-override/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka-override/pom.xml b/metron-platform/metron-storm-kafka-override/pom.xml
new file mode 100644
index 0000000..8683176
--- /dev/null
+++ b/metron-platform/metron-storm-kafka-override/pom.xml
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ Licensed to the Apache Software
+ Foundation (ASF) under one or more contributor license agreements. See the
+ NOTICE file distributed with this work for additional information regarding
+ copyright ownership. The ASF licenses this file to You under the Apache License,
+ Version 2.0 (the "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+ Unless required by applicable law or agreed to in writing, software distributed
+ under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+ the specific language governing permissions and limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.metron</groupId>
+ <artifactId>metron-platform</artifactId>
+ <version>0.4.0</version>
+ </parent>
+ <artifactId>metron-storm-kafka-override</artifactId>
+ <name>metron-storm-kafka-override</name>
+ <description>Components that extend the Storm/Kafka spout</description>
+ <url>https://metron.apache.org/</url>
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
+ <commons.config.version>1.10</commons.config.version>
+ </properties>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-kafka-client</artifactId>
+ <version>${global_storm_kafka_version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka-clients</artifactId>
+ <version>${global_kafka_version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.storm</groupId>
+ <artifactId>storm-core</artifactId>
+ <version>${global_storm_version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>servlet-api</artifactId>
+ <groupId>javax.servlet</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j-over-slf4j</artifactId>
+ <groupId>org.slf4j</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>log4j-slf4j-impl</artifactId>
+ <groupId>org.apache.logging.log4j</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>kafka_2.10</artifactId>
+ <version>${global_kafka_version}</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jmx</groupId>
+ <artifactId>jmxri</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.sun.jdmk</groupId>
+ <artifactId>jmxtools</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.jms</groupId>
+ <artifactId>jms</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.metron</groupId>
+ <artifactId>metron-common</artifactId>
+ <version>${project.parent.version}</version>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <resources>
+ <resource>
+ <directory>src/main/resources</directory>
+ </resource>
+ </resources>
+ </build>
+</project>
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java b/metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
new file mode 100644
index 0000000..f9782ab
--- /dev/null
+++ b/metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout.internal;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.storm.utils.Time;
+
+public class Timer {
+ private final long delay;
+ private final long period;
+ private final TimeUnit timeUnit;
+ private final long periodNanos;
+ private long start;
+
+ public Timer(long delay, long period, TimeUnit timeUnit) {
+ this.delay = delay;
+ this.period = period;
+ this.timeUnit = timeUnit;
+ this.periodNanos = timeUnit.toNanos(period);
+ this.start = System.nanoTime() + timeUnit.toNanos(delay);
+ }
+
+ public long period() {
+ return this.period;
+ }
+
+ public long delay() {
+ return this.delay;
+ }
+
+ public TimeUnit getTimeUnit() {
+ return this.timeUnit;
+ }
+
+ public boolean isExpiredResetOnTrue() {
+ boolean expired = System.nanoTime() - this.start >= this.periodNanos;
+ if(expired) {
+ this.start = System.nanoTime();
+ }
+
+ return expired;
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka-override/src/test/java/org/apache/storm/kafka/spout/internal/TimerTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka-override/src/test/java/org/apache/storm/kafka/spout/internal/TimerTest.java b/metron-platform/metron-storm-kafka-override/src/test/java/org/apache/storm/kafka/spout/internal/TimerTest.java
new file mode 100644
index 0000000..0d49ae1
--- /dev/null
+++ b/metron-platform/metron-storm-kafka-override/src/test/java/org/apache/storm/kafka/spout/internal/TimerTest.java
@@ -0,0 +1,36 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout.internal;
+
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.concurrent.TimeUnit;
+
+public class TimerTest {
+
+ @Test
+ public void testReset() throws InterruptedException {
+ Timer t = new Timer(0, 2, TimeUnit.SECONDS);
+ Thread.sleep(1000);
+ Assert.assertFalse(t.isExpiredResetOnTrue());
+ Thread.sleep(1000);
+ Assert.assertTrue(t.isExpiredResetOnTrue());
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka/pom.xml b/metron-platform/metron-storm-kafka/pom.xml
index b8e3f8d..5c28b34 100644
--- a/metron-platform/metron-storm-kafka/pom.xml
+++ b/metron-platform/metron-storm-kafka/pom.xml
@@ -31,6 +31,11 @@
</properties>
<dependencies>
<dependency>
+ <groupId>org.apache.metron</groupId>
+ <artifactId>metron-storm-kafka-override</artifactId>
+ <version>${project.parent.version}</version>
+ </dependency>
+ <dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-kafka-client</artifactId>
<version>${global_storm_kafka_version}</version>
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java b/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
index bf5250b..592859e 100644
--- a/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
+++ b/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
@@ -20,8 +20,10 @@ package org.apache.metron.storm.kafka.flux;
import com.google.common.base.Joiner;
import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.Deserializer;
import org.apache.metron.common.utils.KafkaUtils;
import org.apache.storm.kafka.spout.*;
import org.apache.storm.spout.SpoutOutputCollector;
@@ -30,10 +32,7 @@ import org.apache.storm.topology.OutputFieldsGetter;
import org.apache.storm.tuple.Fields;
import org.apache.storm.tuple.Values;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
import java.util.function.Function;
/**
@@ -47,7 +46,6 @@ import java.util.function.Function;
* @param <V> The kafka value type
*/
public class SimpleStormKafkaBuilder<K, V> extends KafkaSpoutConfig.Builder<K, V> {
- final static String STREAM = "default";
/**
* The fields exposed by the kafka consumer. These will show up in the Storm tuple.
@@ -113,11 +111,12 @@ public class SimpleStormKafkaBuilder<K, V> extends KafkaSpoutConfig.Builder<K, V
* @param <K> The key type in kafka
* @param <V> The value type in kafka
*/
- public static class TupleBuilder<K, V> extends KafkaSpoutTupleBuilder<K,V> {
+ public static class SpoutRecordTranslator<K, V> implements RecordTranslator<K,V> {
private List<FieldsConfiguration> configurations;
- private TupleBuilder(String topic, List<FieldsConfiguration> configurations) {
- super(topic);
+ private Fields fields;
+ private SpoutRecordTranslator(List<FieldsConfiguration> configurations) {
this.configurations = configurations;
+ this.fields = FieldsConfiguration.getFields(configurations);
}
/**
@@ -127,15 +126,27 @@ public class SimpleStormKafkaBuilder<K, V> extends KafkaSpoutConfig.Builder<K, V
* @return list of tuples
*/
@Override
- public List<Object> buildTuple(ConsumerRecord<K, V> consumerRecord) {
+ public List<Object> apply(ConsumerRecord<K, V> consumerRecord) {
Values ret = new Values();
for(FieldsConfiguration config : configurations) {
ret.add(config.recordExtractor.apply(consumerRecord));
}
return ret;
}
+
+ @Override
+ public Fields getFieldsFor(String s) {
+ return fields;
+ }
+
+ @Override
+ public List<String> streams() {
+ return DEFAULT_STREAM;
+ }
}
+ public static String DEFAULT_DESERIALIZER = ByteArrayDeserializer.class.getName();
+
private String topic;
/**
@@ -165,13 +176,39 @@ public class SimpleStormKafkaBuilder<K, V> extends KafkaSpoutConfig.Builder<K, V
, List<String> fieldsConfiguration
)
{
- super( modifyKafkaProps(kafkaProps, zkQuorum)
- , createStreams(fieldsConfiguration, topic)
- , createTuplesBuilder(fieldsConfiguration, topic)
- );
+ super( getBootstrapServers(zkQuorum, kafkaProps)
+ , createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)), DEFAULT_DESERIALIZER)
+ , createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)), DEFAULT_DESERIALIZER)
+ , topic
+ );
+ setProp(kafkaProps);
+ setRecordTranslator(new SpoutRecordTranslator<>(FieldsConfiguration.toList(fieldsConfiguration)));
this.topic = topic;
}
+ private static <T> Class<Deserializer<T>> createDeserializer( Optional<String> deserializerClass
+ , String defaultDeserializerClass
+ )
+ {
+ try {
+ return (Class<Deserializer<T>>) Class.forName(deserializerClass.orElse(defaultDeserializerClass));
+ } catch (Exception e) {
+ throw new IllegalStateException("Unable to create a deserializer: " + deserializerClass.orElse(defaultDeserializerClass) + ": " + e.getMessage(), e);
+ }
+ }
+
+ private static String getBootstrapServers(String zkQuorum, Map<String, Object> kafkaProps) {
+ String brokers = (String)kafkaProps.get(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
+ if(brokers == null) {
+ try {
+ return Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum));
+ } catch (Exception e) {
+ throw new IllegalStateException("Unable to find the bootstrap servers: " + e.getMessage(), e);
+ }
+ }
+ return brokers;
+ }
+
/**
* Get the kafka topic. TODO: In the future, support multiple topics and regex patterns.
* @return
@@ -202,31 +239,4 @@ public class SimpleStormKafkaBuilder<K, V> extends KafkaSpoutConfig.Builder<K, V
return new StormKafkaSpout<>(builder);
}
- private static Map<String, Object> modifyKafkaProps(Map<String, Object> props, String zkQuorum) {
- try {
- if(!props.containsKey(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS)) {
- //this isn't a putIfAbsent because I only want to pull the brokers from zk if it's absent.
- List<String> brokers = KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum);
- props.put(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS, Joiner.on(",").join(brokers));
- }
- props.putIfAbsent(KafkaSpoutConfig.Consumer.KEY_DESERIALIZER, ByteArrayDeserializer.class.getName());
- props.putIfAbsent(KafkaSpoutConfig.Consumer.VALUE_DESERIALIZER, ByteArrayDeserializer.class.getName());
-
- } catch (Exception e) {
- throw new IllegalStateException("Unable to retrieve brokers from zookeeper: " + e.getMessage(), e);
- }
- return props;
- }
-
- private static <K,V> KafkaSpoutTuplesBuilder<K, V> createTuplesBuilder(List<String> config, String topic) {
- TupleBuilder<K, V> tb = new TupleBuilder<K, V>(topic, FieldsConfiguration.toList(config));
- return new KafkaSpoutTuplesBuilderNamedTopics.Builder<>(tb).build();
- }
-
-
- private static KafkaSpoutStreams createStreams(List<String> config, String topic) {
- final Fields fields = FieldsConfiguration.getFields(FieldsConfiguration.toList(config));
- return new KafkaSpoutStreamsNamedTopics.Builder(fields, STREAM, new String[] { topic} ).build();
- }
-
}
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SpoutConfiguration.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SpoutConfiguration.java b/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SpoutConfiguration.java
index 6c0f148..2a4586d 100644
--- a/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SpoutConfiguration.java
+++ b/metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SpoutConfiguration.java
@@ -17,6 +17,8 @@
*/
package org.apache.metron.storm.kafka.flux;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.metron.common.utils.ConversionUtils;
import org.apache.storm.kafka.spout.KafkaSpoutConfig;
@@ -49,12 +51,6 @@ public enum SpoutConfiguration {
, container -> container.builder.setFirstPollOffsetStrategy(KafkaSpoutConfig.FirstPollOffsetStrategy.valueOf(container.value.toString()))
)
/**
- * The maximum number of retries
- */
- ,MAX_RETRIES("spout.maxRetries"
- , container -> container.builder.setMaxRetries(ConversionUtils.convert(container.value, Integer.class))
- )
- /**
* The maximum amount of uncommitted offsets
*/
,MAX_UNCOMMITTED_OFFSETS("spout.maxUncommittedOffsets"
@@ -66,6 +62,12 @@ public enum SpoutConfiguration {
,OFFSET_COMMIT_PERIOD_MS("spout.offsetCommitPeriodMs"
, container -> container.builder.setOffsetCommitPeriodMs(ConversionUtils.convert(container.value, Long.class))
)
+ /**
+ * The partition refresh period in milliseconds
+ */
+ ,PARTITION_REFRESH_PERIOD_MS("spout.partitionRefreshPeriodMs"
+ , container -> container.builder.setPartitionRefreshPeriodMs(ConversionUtils.convert(container.value, Long.class))
+ )
;
private static class Container {
Map<String, Object> config;
@@ -131,9 +133,9 @@ public enum SpoutConfiguration {
for(SpoutConfiguration spoutConfig : SpoutConfiguration.values()) {
ret.add(spoutConfig.key);
}
- ret.add(KafkaSpoutConfig.Consumer.GROUP_ID);
- ret.add(KafkaSpoutConfig.Consumer.AUTO_COMMIT_INTERVAL_MS);
- ret.add(KafkaSpoutConfig.Consumer.ENABLE_AUTO_COMMIT);
+ ret.add(ConsumerConfig.GROUP_ID_CONFIG);
+ ret.add(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG);
+ ret.add(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG);
return ret;
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/metron-storm-kafka/src/test/java/org/apache/metron/storm/kafka/flux/SpoutConfigurationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-storm-kafka/src/test/java/org/apache/metron/storm/kafka/flux/SpoutConfigurationTest.java b/metron-platform/metron-storm-kafka/src/test/java/org/apache/metron/storm/kafka/flux/SpoutConfigurationTest.java
index fdef69d..c6dbd8f 100644
--- a/metron-platform/metron-storm-kafka/src/test/java/org/apache/metron/storm/kafka/flux/SpoutConfigurationTest.java
+++ b/metron-platform/metron-storm-kafka/src/test/java/org/apache/metron/storm/kafka/flux/SpoutConfigurationTest.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.storm.kafka.flux;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.metron.common.utils.KafkaUtils;
import org.apache.storm.kafka.spout.KafkaSpoutConfig;
import org.junit.Assert;
@@ -33,14 +34,14 @@ public class SpoutConfigurationTest {
public void testSeparation() {
Map<String, Object> config = new HashMap<String, Object>() {{
put(SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key, "UNCOMMITTED_EARLIEST");
- put(SpoutConfiguration.MAX_RETRIES.key, "1000");
+ put(SpoutConfiguration.OFFSET_COMMIT_PERIOD_MS.key, "1000");
put("group.id", "foobar");
}};
Map<String, Object> spoutConfig = SpoutConfiguration.separate(config);
Assert.assertTrue(spoutConfig.containsKey(SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key));
Assert.assertEquals(spoutConfig.get(SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key), "UNCOMMITTED_EARLIEST");
- Assert.assertTrue(spoutConfig.containsKey(SpoutConfiguration.MAX_RETRIES.key));
- Assert.assertEquals(spoutConfig.get(SpoutConfiguration.MAX_RETRIES.key), "1000");
+ Assert.assertTrue(spoutConfig.containsKey(SpoutConfiguration.OFFSET_COMMIT_PERIOD_MS.key));
+ Assert.assertEquals(spoutConfig.get(SpoutConfiguration.OFFSET_COMMIT_PERIOD_MS.key), "1000");
Assert.assertEquals(2, spoutConfig.size());
Assert.assertEquals(1, config.size());
Assert.assertEquals(config.get("group.id"), "foobar");
@@ -49,15 +50,15 @@ public class SpoutConfigurationTest {
@Test
public void testBuilderCreation() {
Map<String, Object> config = new HashMap<String, Object>() {{
- put(SpoutConfiguration.MAX_RETRIES.key, "1000");
- put(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS, "foo:1234");
+ put(SpoutConfiguration.OFFSET_COMMIT_PERIOD_MS.key, "1000");
+ put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "foo:1234");
put("group.id", "foobar");
}};
Map<String, Object> spoutConfig = SpoutConfiguration.separate(config);
KafkaSpoutConfig.Builder<Object, Object> builder = new SimpleStormKafkaBuilder(config, "topic", null);
SpoutConfiguration.configure(builder, spoutConfig);
KafkaSpoutConfig c = builder.build();
- Assert.assertEquals(1000, c.getMaxTupleRetries() );
+ Assert.assertEquals(1000, c.getOffsetsCommitPeriodMs() );
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/metron-platform/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/pom.xml b/metron-platform/pom.xml
index 1376e5c..cc48851 100644
--- a/metron-platform/pom.xml
+++ b/metron-platform/pom.xml
@@ -58,6 +58,7 @@
<module>elasticsearch-shaded</module>
<module>metron-elasticsearch</module>
<module>metron-storm-kafka</module>
+ <module>metron-storm-kafka-override</module>
</modules>
<dependencies>
<dependency>
http://git-wip-us.apache.org/repos/asf/metron/blob/6c836d13/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index bb32c64..9f62249 100644
--- a/pom.xml
+++ b/pom.xml
@@ -39,6 +39,22 @@
<repositories>
<repository>
+ <releases>
+ <enabled>true</enabled>
+ <updatePolicy>always</updatePolicy>
+ <checksumPolicy>warn</checksumPolicy>
+ </releases>
+ <snapshots>
+ <enabled>true</enabled>
+ <updatePolicy>never</updatePolicy>
+ <checksumPolicy>warn</checksumPolicy>
+ </snapshots>
+ <id>HDPPrivateReleases</id>
+ <name>HDP Private Releases</name>
+ <url>http://nexus-private.hortonworks.com/nexus/content/groups/public</url>
+ <layout>default</layout>
+ </repository>
+ <repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
@@ -83,30 +99,7 @@
<global_curator_version>2.7.1</global_curator_version>
<global_classindex_version>3.3</global_classindex_version>
<global_storm_version>1.0.3</global_storm_version>
- <!--
- This bears some explanation. storm-kafka-client is our kafka spout.
- If we ever hope to support kerberos, this provides the capability to do so
- in apache. Unfortunately, it also does not support, as of Storm 1.0.x
- Kafka 0.10.x (see https://issues.apache.org/jira/browse/STORM-2091).
- The consumer libraries (not to be confused with the protocol) on the JVM
- are binary incompatible. Note the discussion on https://issues.apache.org/jira/browse/KAFKA-3006,
- the main issue is the move to Collection over List. While this would be polymorphically
- a non-issue, it would require a recompile of storm-kafka-client against Kafka 0.10.x.
-
- Since a targeted platform is HDP 2.5.x, which ships only kafka 0.10.x, we need
- to support kafka 0.10.x. Therefore, if we are to use this, then we would need
- to support both Kafka 0.9.x and 0.10.x. Unfortunately, this would require us
- to fork some of the internal projects because the 0.9.x API has shifted
- (e.g. the Admin functions have different parameters) and behaves
- differently than 0.10.x in subtle ways (e.g. KAFKA_GET doesn't work as implemented).
-
- Rather than do this, we chose to depend on the HDP version of storm-kafka because
- it is compiled against 0.10.x and therefore would allow us to not fork our support
- for kafka. I do not like this bleeding of the HDP profile dependency into the default,
- but I justify it by noting that this should be able to be removed when we migrate to
- Storm 1.1.x, which properly supports Kafka 0.10.x.
- -->
- <global_storm_kafka_version>1.0.1.2.5.0.0-1245</global_storm_kafka_version>
+ <global_storm_kafka_version>1.1.0</global_storm_kafka_version>
<global_flux_version>${base_flux_version}</global_flux_version>
<global_pcap_version>1.7.1</global_pcap_version>
<global_kafka_version>0.10.0.1</global_kafka_version>
@@ -138,6 +131,7 @@
<properties>
<hdp_version>2.5.0.0</hdp_version>
<build_number>1245</build_number>
+ <global_storm_kafka_version>1.1.0.2.6.1.0-SNAPSHOT</global_storm_kafka_version>
<global_storm_version>${base_storm_version}.${hdp_version}-${build_number}</global_storm_version>
<global_kafka_version>${base_kafka_version}.${hdp_version}-${build_number}</global_kafka_version>
</properties>
[27/44] metron git commit: METRON-936: Fixes to pcap for performance
and testing (mmiklavc via cestella) closes apache/incubator-metron#585
Posted by ma...@apache.org.
METRON-936: Fixes to pcap for performance and testing (mmiklavc via cestella) closes apache/incubator-metron#585
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/c0b08252
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/c0b08252
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/c0b08252
Branch: refs/heads/Metron_0.4.0
Commit: c0b0825235ce5c26649282c43f3c57f092864f94
Parents: be03076
Author: mmiklavc <mi...@gmail.com>
Authored: Wed May 17 10:32:46 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Wed May 17 10:32:46 2017 -0400
----------------------------------------------------------------------
metron-platform/metron-pcap-backend/README.md | 198 +++++++++++++++++++
.../src/main/config/pcap.properties | 10 +-
.../src/main/flux/pcap/remote.yaml | 14 +-
.../metron/spout/pcap/HDFSWriterCallback.java | 76 ++++---
.../metron/spout/pcap/HDFSWriterConfig.java | 47 +++++
.../metron/spout/pcap/PartitionHDFSWriter.java | 61 ++++--
.../pcap/deserializer/FromKeyDeserializer.java | 22 +--
.../deserializer/FromPacketDeserializer.java | 16 +-
.../pcap/deserializer/KeyValueDeserializer.java | 15 +-
.../org/apache/metron/utils/PcapInspector.java | 29 +--
.../PcapTopologyIntegrationTest.java | 9 +-
.../deserializer/FromKeyDeserializerTest.java | 39 ++++
.../java/org/apache/metron/pcap/PcapHelper.java | 53 ++++-
.../java/org/apache/metron/pcap/mr/PcapJob.java | 19 +-
14 files changed, 505 insertions(+), 103 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/README.md b/metron-platform/metron-pcap-backend/README.md
index 5b554b5..da19611 100644
--- a/metron-platform/metron-pcap-backend/README.md
+++ b/metron-platform/metron-pcap-backend/README.md
@@ -4,6 +4,16 @@ The purpose of the Metron PCAP backend is to create a storm topology
capable of ingesting rapidly raw packet capture data directly into HDFS
from Kafka.
+* [Sensors](#the-sensors-feeding-kafka)
+* [PCAP Topology](#the-pcap-topology)
+* [HDFS Files](#the-files-on-hdfs)
+* [Configuration](#configuration)
+* [Starting the Topology](#starting-the-topology)
+* [Utilities](#utilities)
+ * [Inspector Utility](#inspector-utility)
+ * [Query Filter Utility](#query-filter-utility)
+* [Performance Tuning](#performance-tuning)
+
## The Sensors Feeding Kafka
This component must be fed by fast packet capture components upstream
@@ -150,3 +160,191 @@ The packet data will be exposed via the`packet` variable in Stellar.
The format of this regular expression is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).
+## Performance Tuning
+The PCAP topology is extremely lightweight and functions as a Spout-only topology. In order to tune the topology, users currently must specify a combination of
+properties in pcap.properties as well as configuration in the pcap remote.yaml flux file itself. Tuning the number of partitions in your Kafka topic
+will have a dramatic impact on performance as well. We ran data into Kafka at 1.1 Gbps and our tests resulted in configuring 128 partitions for our kakfa topic
+along with the following settings in pcap.properties and remote.yaml (unrelated properties for performance have been removed):
+
+### pcap.properties file
+```
+spout.kafka.topic.pcap=pcap
+storm.topology.workers=16
+kafka.spout.parallelism=128
+kafka.pcap.numPackets=1000000000
+kafka.pcap.maxTimeMS=0
+hdfs.replication=1
+hdfs.sync.every=10000
+```
+You'll notice that the number of kakfa partitions equals the spout parallelism, and this is no coincidence. The ordering guarantees for a partition in Kafka enforces that you may have no more
+consumers than 1 per topic. Any additional parallelism will leave you with dormant threads consuming resources but performing no additional work. For our cluster with 4 Storm Supervisors, we found 16 workers to
+provide optimal throughput as well. We were largely IO bound rather than CPU bound with the incoming PCAP data.
+
+### remote.yaml
+In the flux file, we introduced the following configuration:
+
+```
+name: "pcap"
+config:
+ topology.workers: ${storm.topology.workers}
+ topology.worker.childopts: ${topology.worker.childopts}
+ topology.auto-credentials: ${storm.auto.credentials}
+ topology.ackers.executors: 0
+components:
+
+ # Any kafka props for the producer go here.
+ - id: "kafkaProps"
+ className: "java.util.HashMap"
+ configMethods:
+ - name: "put"
+ args:
+ - "value.deserializer"
+ - "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+ - name: "put"
+ args:
+ - "key.deserializer"
+ - "org.apache.kafka.common.serialization.ByteArrayDeserializer"
+ - name: "put"
+ args:
+ - "group.id"
+ - "pcap"
+ - name: "put"
+ args:
+ - "security.protocol"
+ - "${kafka.security.protocol}"
+ - name: "put"
+ args:
+ - "poll.timeout.ms"
+ - 100
+ - name: "put"
+ args:
+ - "offset.commit.period.ms"
+ - 30000
+ - name: "put"
+ args:
+ - "session.timeout.ms"
+ - 30000
+ - name: "put"
+ args:
+ - "max.uncommitted.offsets"
+ - 200000000
+ - name: "put"
+ args:
+ - "max.poll.interval.ms"
+ - 10
+ - name: "put"
+ args:
+ - "max.poll.records"
+ - 200000
+ - name: "put"
+ args:
+ - "receive.buffer.bytes"
+ - 431072
+ - name: "put"
+ args:
+ - "max.partition.fetch.bytes"
+ - 8097152
+
+ - id: "hdfsProps"
+ className: "java.util.HashMap"
+ configMethods:
+ - name: "put"
+ args:
+ - "io.file.buffer.size"
+ - 1000000
+ - name: "put"
+ args:
+ - "dfs.blocksize"
+ - 1073741824
+
+ - id: "kafkaConfig"
+ className: "org.apache.metron.storm.kafka.flux.SimpleStormKafkaBuilder"
+ constructorArgs:
+ - ref: "kafkaProps"
+ # topic name
+ - "${spout.kafka.topic.pcap}"
+ - "${kafka.zk}"
+ configMethods:
+ - name: "setFirstPollOffsetStrategy"
+ args:
+ # One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
+ - ${kafka.pcap.start}
+
+ - id: "writerConfig"
+ className: "org.apache.metron.spout.pcap.HDFSWriterConfig"
+ configMethods:
+ - name: "withOutputPath"
+ args:
+ - "${kafka.pcap.out}"
+ - name: "withNumPackets"
+ args:
+ - ${kafka.pcap.numPackets}
+ - name: "withMaxTimeMS"
+ args:
+ - ${kafka.pcap.maxTimeMS}
+ - name: "withZookeeperQuorum"
+ args:
+ - "${kafka.zk}"
+ - name: "withSyncEvery"
+ args:
+ - ${hdfs.sync.every}
+ - name: "withReplicationFactor"
+ args:
+ - ${hdfs.replication}
+ - name: "withHDFSConfig"
+ args:
+ - ref: "hdfsProps"
+ - name: "withDeserializer"
+ args:
+ - "${kafka.pcap.ts_scheme}"
+ - "${kafka.pcap.ts_granularity}"
+spouts:
+ - id: "kafkaSpout"
+ className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout"
+ parallelism: ${kafka.spout.parallelism}
+ constructorArgs:
+ - ref: "kafkaConfig"
+ - ref: "writerConfig"
+
+```
+
+#### Flux Changes Introduced
+
+##### Topology Configuration
+
+The only change here is `topology.ackers.executors: 0`, which disables Storm tuple acking for maximum throughput.
+
+##### Kafka configuration
+
+```
+poll.timeout.ms
+offset.commit.period.ms
+session.timeout.ms
+max.uncommitted.offsets
+max.poll.interval.ms
+max.poll.records
+receive.buffer.bytes
+max.partition.fetch.bytes
+```
+
+##### Writer Configuration
+
+This is a combination of settings for the HDFSWriter (see pcap.properties values above) as well as HDFS.
+
+__HDFS config__
+
+Component config HashMap with the following properties:
+```
+io.file.buffer.size
+dfs.blocksize
+```
+
+__Writer config__
+
+References the HDFS props component specified above.
+```
+ - name: "withHDFSConfig"
+ args:
+ - ref: "hdfsProps"
+```
+
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/config/pcap.properties
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/config/pcap.properties b/metron-platform/metron-pcap-backend/src/main/config/pcap.properties
index 6e51dc5..7160178 100644
--- a/metron-platform/metron-pcap-backend/src/main/config/pcap.properties
+++ b/metron-platform/metron-pcap-backend/src/main/config/pcap.properties
@@ -15,12 +15,18 @@
# limitations under the License.
spout.kafka.topic.pcap=pcap
-storm.auto.credentials=[]
+topology.worker.childopts=
+topology.auto-credentials=[]
+topology.workers=1
kafka.zk=node1:2181
+hdfs.sync.every=1
+hdfs.replication.factor=-1
kafka.security.protocol=PLAINTEXT
-kafka.pcap.start=END
+# One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
+kafka.pcap.start=UNCOMMITTED_EARLIEST
kafka.pcap.numPackets=1000
kafka.pcap.maxTimeMS=300000
kafka.pcap.ts_scheme=FROM_KEY
kafka.pcap.out=/apps/metron/pcap
kafka.pcap.ts_granularity=MICROSECONDS
+kafka.spout.parallelism=1
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/flux/pcap/remote.yaml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/flux/pcap/remote.yaml b/metron-platform/metron-pcap-backend/src/main/flux/pcap/remote.yaml
index 2b7e0fd..d7f6f2f 100644
--- a/metron-platform/metron-pcap-backend/src/main/flux/pcap/remote.yaml
+++ b/metron-platform/metron-pcap-backend/src/main/flux/pcap/remote.yaml
@@ -16,8 +16,9 @@
name: "pcap"
config:
- topology.workers: 1
- topology.auto-credentials: ${storm.auto.credentials}
+ topology.workers: ${topology.workers}
+ topology.worker.childopts: ${topology.worker.childopts}
+ topology.auto-credentials: ${topology.auto-credentials}
components:
@@ -53,7 +54,7 @@ components:
- name: "setFirstPollOffsetStrategy"
args:
# One of EARLIEST, LATEST, UNCOMMITTED_EARLIEST, UNCOMMITTED_LATEST
- - "UNCOMMITTED_EARLIEST"
+ - ${kafka.pcap.start}
- id: "writerConfig"
className: "org.apache.metron.spout.pcap.HDFSWriterConfig"
@@ -70,6 +71,12 @@ components:
- name: "withZookeeperQuorum"
args:
- "${kafka.zk}"
+ - name: "withSyncEvery"
+ args:
+ - ${hdfs.sync.every}
+ - name: "withReplicationFactor"
+ args:
+ - ${hdfs.replication.factor}
- name: "withDeserializer"
args:
- "${kafka.pcap.ts_scheme}"
@@ -77,6 +84,7 @@ components:
spouts:
- id: "kafkaSpout"
className: "org.apache.metron.spout.pcap.KafkaToHDFSSpout"
+ parallelism: ${kafka.spout.parallelism}
constructorArgs:
- ref: "kafkaConfig"
- ref: "writerConfig"
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
index 43cd7e0..a6823e6 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
@@ -19,14 +19,13 @@
package org.apache.metron.spout.pcap;
import com.google.common.base.Joiner;
-import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.io.LongWritable;
-import org.apache.log4j.Logger;
+import org.apache.metron.spout.pcap.deserializer.KeyValueDeserializer;
import org.apache.storm.kafka.Callback;
import org.apache.storm.kafka.EmitContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import javax.xml.bind.DatatypeConverter;
-import java.io.Closeable;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
@@ -38,7 +37,7 @@ import java.util.Map;
*/
public class HDFSWriterCallback implements Callback {
static final long serialVersionUID = 0xDEADBEEFL;
- private static final Logger LOG = Logger.getLogger(HDFSWriterCallback.class);
+ private static final Logger LOG = LoggerFactory.getLogger(HDFSWriterCallback.class);
/**
* A topic+partition. We split the files up by topic+partition so the writers don't clobber each other
@@ -80,29 +79,12 @@ public class HDFSWriterCallback implements Callback {
}
}
- /**
- * This is a static container of threadlocal LongWritables and BytesWritables. This keeps us from having to create so
- * many objects on the heap. The Deserializers update these for every packet.
- */
- private static class KeyValue {
- static ThreadLocal<LongWritable> key = new ThreadLocal<LongWritable> () {
- @Override
- protected LongWritable initialValue() {
- return new LongWritable();
- }
- };
- static ThreadLocal<BytesWritable> value = new ThreadLocal<BytesWritable> () {
- @Override
- protected BytesWritable initialValue() {
- return new BytesWritable();
- }
- };
- }
private HDFSWriterConfig config;
private EmitContext context;
private Map<Partition, PartitionHDFSWriter> writers = new HashMap<>();
private PartitionHDFSWriter lastWriter = null;
private String topic;
+ private boolean inited = false;
public HDFSWriterCallback() {
}
@@ -116,35 +98,43 @@ public class HDFSWriterCallback implements Callback {
public List<Object> apply(List<Object> tuple, EmitContext context) {
byte[] key = (byte[]) tuple.get(0);
byte[] value = (byte[]) tuple.get(1);
- if(!config.getDeserializer().deserializeKeyValue(key, value, KeyValue.key.get(), KeyValue.value.get())) {
- if(LOG.isDebugEnabled()) {
- List<String> debugStatements = new ArrayList<>();
- if(key != null) {
- debugStatements.add("Key length: " + key.length);
- debugStatements.add("Key: " + DatatypeConverter.printHexBinary(key));
- }
- else {
- debugStatements.add("Key is null!");
- }
-
- if(value != null) {
- debugStatements.add("Value length: " + value.length);
- debugStatements.add("Value: " + DatatypeConverter.printHexBinary(value));
- }
- else {
- debugStatements.add("Value is null!");
- }
- LOG.debug("Dropping malformed packet: " + Joiner.on(" / ").join(debugStatements));
+ long tsDeserializeStart = System.nanoTime();
+ KeyValueDeserializer.Result result = config.getDeserializer().deserializeKeyValue(key, value);
+ long tsDeserializeEnd = System.nanoTime();
+
+ if (LOG.isDebugEnabled() && !result.foundTimestamp) {
+ List<String> debugStatements = new ArrayList<>();
+ if (key != null) {
+ debugStatements.add("Key length: " + key.length);
+ debugStatements.add("Key: " + DatatypeConverter.printHexBinary(key));
+ } else {
+ debugStatements.add("Key is null!");
}
+
+ if (value != null) {
+ debugStatements.add("Value length: " + value.length);
+ debugStatements.add("Value: " + DatatypeConverter.printHexBinary(value));
+ } else {
+ debugStatements.add("Value is null!");
+ }
+ LOG.debug("Dropping malformed packet: " + Joiner.on(" / ").join(debugStatements));
}
+
+ long tsWriteStart = System.nanoTime();
try {
getWriter(new Partition( topic
, context.get(EmitContext.Type.PARTITION))
- ).handle(KeyValue.key.get(), KeyValue.value.get());
+ ).handle(result.key, result.value);
} catch (IOException e) {
LOG.error(e.getMessage(), e);
//drop? not sure..
}
+ long tsWriteEnd = System.nanoTime();
+ if(LOG.isDebugEnabled() && (Math.random() < 0.001 || !inited)) {
+ LOG.debug("Deserialize time (ns): " + (tsDeserializeEnd - tsDeserializeStart));
+ LOG.debug("Write time (ns): " + (tsWriteEnd - tsWriteStart));
+ }
+ inited = true;
return tuple;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterConfig.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterConfig.java
index 66bb359..b6a2809 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterConfig.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterConfig.java
@@ -25,7 +25,9 @@ import org.apache.metron.spout.pcap.deserializer.KeyValueDeserializer;
import java.io.Serializable;
import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
/**
* Configure the HDFS Writer for PCap
@@ -34,9 +36,12 @@ public class HDFSWriterConfig implements Serializable {
static final long serialVersionUID = 0xDEADBEEFL;
private long numPackets;
private long maxTimeNS;
+ private int syncEvery = 1;
+ private int replicationFactor = -1;
private String outputPath;
private String zookeeperQuorum;
private KeyValueDeserializer deserializer;
+ private Map<String, Object> hdfsConfig = new HashMap<>();
/**
* Set the deserializer, the bit of logic that defines how the timestamp and packet are read.
@@ -70,6 +75,36 @@ public class HDFSWriterConfig implements Serializable {
}
/**
+ * The number of packets to write before a file is rolled.
+ * @param n
+ * @return
+ */
+ public HDFSWriterConfig withSyncEvery(int n) {
+ syncEvery = n;
+ return this;
+ }
+
+ /**
+ * The map config for HDFS
+ * @param config
+ * @return
+ */
+ public HDFSWriterConfig withHDFSConfig(Map<String, Object> config) {
+ hdfsConfig = config;
+ return this;
+ }
+
+ /**
+ * The HDFS replication factor to use. A value of -1 will not set replication factor.
+ * @param n
+ * @return
+ */
+ public HDFSWriterConfig withReplicationFactor(int n) {
+ replicationFactor = n;
+ return this;
+ }
+
+ /**
* The total amount of time (in ms) to write before a file is rolled.
* @param t
* @return
@@ -103,6 +138,10 @@ public class HDFSWriterConfig implements Serializable {
return out;
}
+ public Map<String, Object> getHDFSConfig() {
+ return hdfsConfig;
+ }
+
public Integer getZookeeperPort() {
if(zookeeperQuorum != null) {
String hostPort = Iterables.getFirst(Splitter.on(',').split(zookeeperQuorum), null);
@@ -112,6 +151,14 @@ public class HDFSWriterConfig implements Serializable {
return null;
}
+ public int getSyncEvery() {
+ return syncEvery;
+ }
+
+ public int getReplicationFactor() {
+ return replicationFactor;
+ }
+
public KeyValueDeserializer getDeserializer() {
return deserializer;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
index f0ea1eb..86697db 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/PartitionHDFSWriter.java
@@ -27,18 +27,23 @@ import org.apache.hadoop.hdfs.client.HdfsDataOutputStream;
import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.SequenceFile;
-import org.apache.log4j.Logger;
import org.apache.metron.pcap.PcapHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
-import java.io.*;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.Serializable;
import java.util.EnumSet;
+import java.util.Map;
/**
* This class is intended to handle the writing of an individual file.
*/
public class PartitionHDFSWriter implements AutoCloseable, Serializable {
static final long serialVersionUID = 0xDEADBEEFL;
- private static final Logger LOG = Logger.getLogger(PartitionHDFSWriter.class);
+ private static final Logger LOG = LoggerFactory.getLogger(PartitionHDFSWriter.class);
public static interface SyncHandler {
@@ -102,14 +107,43 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
private SyncHandler syncHandler;
private long batchStartTime;
private long numWritten;
+ private Configuration fsConfig = new Configuration();
public PartitionHDFSWriter(String topic, int partition, String uuid, HDFSWriterConfig config) {
this.topic = topic;
this.partition = partition;
this.uuid = uuid;
this.config = config;
+
try {
- this.fs = FileSystem.get(new Configuration());
+ int replicationFactor = config.getReplicationFactor();
+ if (replicationFactor > 0) {
+ fsConfig.set("dfs.replication", String.valueOf(replicationFactor));
+ }
+ if(config.getHDFSConfig() != null && !config.getHDFSConfig().isEmpty()) {
+ for(Map.Entry<String, Object> entry : config.getHDFSConfig().entrySet()) {
+ if(entry.getValue() instanceof Integer) {
+ fsConfig.setInt(entry.getKey(), (int)entry.getValue());
+ }
+ else if(entry.getValue() instanceof Boolean)
+ {
+ fsConfig.setBoolean(entry.getKey(), (Boolean) entry.getValue());
+ }
+ else if(entry.getValue() instanceof Long)
+ {
+ fsConfig.setLong(entry.getKey(), (Long) entry.getValue());
+ }
+ else if(entry.getValue() instanceof Float)
+ {
+ fsConfig.setFloat(entry.getKey(), (Float) entry.getValue());
+ }
+ else
+ {
+ fsConfig.set(entry.getKey(), String.valueOf(entry.getValue()));
+ }
+ }
+ }
+ this.fs = FileSystem.get(fsConfig);
} catch (IOException e) {
throw new RuntimeException("Unable to get FileSystem", e);
}
@@ -119,11 +153,14 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
return Long.toUnsignedString(ts);
}
- public void handle(LongWritable ts, BytesWritable value) throws IOException {
- turnoverIfNecessary(ts.get());
- writer.append(ts, value);
- syncHandler.sync(outputStream);
+ public void handle(long ts, byte[] value) throws IOException {
+ turnoverIfNecessary(ts);
+ BytesWritable bw = new BytesWritable(value);
+ writer.append(new LongWritable(ts), bw);
numWritten++;
+ if(numWritten % config.getSyncEvery() == 0) {
+ syncHandler.sync(outputStream);
+ }
}
public String getTopic() {
@@ -144,8 +181,8 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
outputStream.close();
}
}
- private Path getPath(long ts) {
+ private Path getPath(long ts) {
String fileName = PcapHelper.toFilename(topic, ts, partition + "", uuid);
return new Path(config.getOutputPath(), fileName);
}
@@ -157,7 +194,7 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
private void turnoverIfNecessary(long ts, boolean force) throws IOException {
long duration = ts - batchStartTime;
boolean initial = outputStream == null;
- boolean overDuration = duration >= config.getMaxTimeNS();
+ boolean overDuration = config.getMaxTimeNS() <= 0 ? false : Long.compareUnsigned(duration, config.getMaxTimeNS()) >= 0;
boolean tooManyPackets = numWritten >= config.getNumPackets();
if(force || initial || overDuration || tooManyPackets ) {
//turnover
@@ -183,14 +220,14 @@ public class PartitionHDFSWriter implements AutoCloseable, Serializable {
}
}
- writer = SequenceFile.createWriter(new Configuration()
+ writer = SequenceFile.createWriter(this.fsConfig
, SequenceFile.Writer.keyClass(LongWritable.class)
, SequenceFile.Writer.valueClass(BytesWritable.class)
, SequenceFile.Writer.stream(outputStream)
, SequenceFile.Writer.compression(SequenceFile.CompressionType.NONE)
);
//reset state
- LOG.info("Turning over and writing to " + path);
+ LOG.info("Turning over and writing to {}: [duration={} NS, force={}, initial={}, overDuration={}, tooManyPackets={}]", path, duration, force, initial, overDuration, tooManyPackets);
batchStartTime = ts;
numWritten = 0;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializer.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializer.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializer.java
index de1e24b..749d74c 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializer.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializer.java
@@ -18,19 +18,18 @@
package org.apache.metron.spout.pcap.deserializer;
-import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.io.LongWritable;
-import org.apache.log4j.Logger;
import org.apache.metron.common.utils.timestamp.TimestampConverter;
import org.apache.metron.pcap.PcapHelper;
import org.apache.metron.spout.pcap.Endianness;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/**
* Extract the timestamp from the key and raw data from the packet.
*/
public class FromKeyDeserializer extends KeyValueDeserializer {
- private static final Logger LOG = Logger.getLogger(FromKeyDeserializer.class);
+ private static final Logger LOG = LoggerFactory.getLogger(FromKeyDeserializer.class);
private static Endianness endianness = Endianness.getNativeEndianness();
@@ -39,13 +38,12 @@ public class FromKeyDeserializer extends KeyValueDeserializer {
}
@Override
- public boolean deserializeKeyValue(byte[] key, byte[] value, LongWritable outKey, BytesWritable outValue) {
- Long ts = converter.toNanoseconds(fromBytes(key));
- outKey.set(ts);
- byte[] packetHeaderized = PcapHelper.addPacketHeader(ts, value, endianness);
- byte[] globalHeaderized= PcapHelper.addGlobalHeader(packetHeaderized, endianness);
- outValue.set(globalHeaderized, 0, globalHeaderized.length);
- return true;
+ public Result deserializeKeyValue(byte[] key, byte[] value) {
+ if (key == null) {
+ throw new IllegalArgumentException("Expected a key but none provided");
+ }
+ long ts = converter.toNanoseconds(fromBytes(key));
+ return new Result(ts, PcapHelper.addHeaders(ts, value, endianness), true);
}
/**
@@ -65,6 +63,6 @@ public class FromKeyDeserializer extends KeyValueDeserializer {
value |= (long)(b & 255);
}
- return Long.valueOf(value);
+ return value;
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromPacketDeserializer.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromPacketDeserializer.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromPacketDeserializer.java
index 6098904..0ba92f8 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromPacketDeserializer.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/FromPacketDeserializer.java
@@ -18,26 +18,24 @@
package org.apache.metron.spout.pcap.deserializer;
-import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.io.LongWritable;
-import org.apache.log4j.Logger;
import org.apache.metron.pcap.PcapHelper;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
/**
* Extract the timestamp and raw data from the packet.
*/
public class FromPacketDeserializer extends KeyValueDeserializer {
- private static final Logger LOG = Logger.getLogger(FromPacketDeserializer.class);
+ private static final Logger LOG = LoggerFactory.getLogger(FromPacketDeserializer.class);
@Override
- public boolean deserializeKeyValue(byte[] key, byte[] value, LongWritable outKey, BytesWritable outValue) {
+ public Result deserializeKeyValue(byte[] key, byte[] value) {
Long ts = PcapHelper.getTimestamp(value);
if(ts != null) {
- outKey.set(ts);
- outValue.set(value, 0, value.length);
- return true;
+ return new Result(ts, value, true);
}
else {
- return false;
+ return new Result(ts, value, false);
}
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/KeyValueDeserializer.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/KeyValueDeserializer.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/KeyValueDeserializer.java
index 48bea87..311be04 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/KeyValueDeserializer.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/KeyValueDeserializer.java
@@ -18,8 +18,6 @@
package org.apache.metron.spout.pcap.deserializer;
-import org.apache.hadoop.io.BytesWritable;
-import org.apache.hadoop.io.LongWritable;
import org.apache.metron.common.utils.timestamp.TimestampConverter;
import org.apache.metron.common.utils.timestamp.TimestampConverters;
@@ -28,6 +26,17 @@ import java.io.Serializable;
public abstract class KeyValueDeserializer implements Serializable {
protected TimestampConverter converter;
+ public static class Result {
+ public byte[] value;
+ public Long key;
+ public boolean foundTimestamp;
+ public Result(Long key, byte[] value, boolean foundTimestamp) {
+ this.key = key;
+ this.value = value;
+ this.foundTimestamp = foundTimestamp;
+ }
+ }
+
public KeyValueDeserializer() {
this(TimestampConverters.MICROSECONDS);
}
@@ -36,6 +45,6 @@ public abstract class KeyValueDeserializer implements Serializable {
this.converter = converter;
}
- public abstract boolean deserializeKeyValue(byte[] key, byte[] value, LongWritable outKey, BytesWritable outValue);
+ public abstract Result deserializeKeyValue(byte[] key, byte[] value);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
index f460db3..c887606 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/utils/PcapInspector.java
@@ -35,7 +35,10 @@ import javax.annotation.Nullable;
import java.io.IOException;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
-import java.util.*;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
public class PcapInspector {
private static abstract class OptionHandler implements Function<String, Option> {}
@@ -136,20 +139,24 @@ public class PcapInspector {
LongWritable key = new LongWritable();
BytesWritable value = new BytesWritable();
- for(int i = 0;(n < 0 || i < n) && reader.next(key, value);++i) {
+ for (int i = 0; (n < 0 || i < n) && reader.next(key, value); ++i) {
long millis = Long.divideUnsigned(key.get(), 1000000);
String ts = DATE_FORMAT.format(new Date(millis));
- for(PacketInfo pi : PcapHelper.toPacketInfo(value.copyBytes())) {
- Map<String, Object> result = PcapHelper.packetToFields(pi);
- List<String> fieldResults = new ArrayList<String>() {{
- add("TS: " + ts);
- }};
- for(Constants.Fields field : Constants.Fields.values()) {
- if(result.containsKey(field.getName())) {
- fieldResults.add(field.getName() + ": " + result.get(field.getName()));
+ try {
+ for (PacketInfo pi : PcapHelper.toPacketInfo(value.copyBytes())) {
+ Map<String, Object> result = PcapHelper.packetToFields(pi);
+ List<String> fieldResults = new ArrayList<String>() {{
+ add("TS: " + ts);
+ }};
+ for (Constants.Fields field : Constants.Fields.values()) {
+ if (result.containsKey(field.getName())) {
+ fieldResults.add(field.getName() + ": " + result.get(field.getName()));
+ }
}
+ System.out.println(Joiner.on(",").join(fieldResults));
}
- System.out.println(Joiner.on(",").join(fieldResults));
+ } catch (Exception e) {
+ System.out.println(String.format("Error: malformed packet #=%s, ts=%s, error msg=%s", i + 1, ts, e.getMessage()));
}
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
index a869723..d6d54dc 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
@@ -213,14 +213,19 @@ public class PcapTopologyIntegrationTest {
final List<Map.Entry<byte[], byte[]>> pcapEntries = Lists.newArrayList(readPcaps(pcapFile, withHeaders));
Assert.assertTrue(Iterables.size(pcapEntries) > 0);
final Properties topologyProperties = new Properties() {{
+ setProperty("topology.workers", "1");
+ setProperty("topology.worker.childopts", "");
setProperty("spout.kafka.topic.pcap", KAFKA_TOPIC);
- setProperty("kafka.pcap.start", "BEGINNING");
+ setProperty("kafka.pcap.start", "EARLIEST");
setProperty("kafka.pcap.out", outDir.getAbsolutePath());
setProperty("kafka.pcap.numPackets", "2");
setProperty("kafka.pcap.maxTimeMS", "200000000");
setProperty("kafka.pcap.ts_granularity", "NANOSECONDS");
- setProperty("storm.auto.credentials", "[]");
+ setProperty("kafka.spout.parallelism", "1");
+ setProperty("topology.auto-credentials", "[]");
setProperty("kafka.security.protocol", "PLAINTEXT");
+ setProperty("hdfs.sync.every", "1");
+ setProperty("hdfs.replication.factor", "-1");
}};
updatePropertiesCallback.apply(topologyProperties);
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializerTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializerTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializerTest.java
new file mode 100644
index 0000000..1d49103
--- /dev/null
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/spout/pcap/deserializer/FromKeyDeserializerTest.java
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.spout.pcap.deserializer;
+
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+
+public class FromKeyDeserializerTest {
+
+ @Rule
+ public final ExpectedException exception = ExpectedException.none();
+
+ @Test
+ public void empty_or_null_key_throws_illegal_argument_exception() {
+ exception.expect(IllegalArgumentException.class);
+ exception.expectMessage("Expected a key but none provided");
+
+ FromKeyDeserializer deserializer = new FromKeyDeserializer(TimestampConverters.NANOSECONDS);
+ deserializer.deserializeKeyValue(null, null);
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
index ebd7ac7..e1ad3ca 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/PcapHelper.java
@@ -39,9 +39,10 @@ import org.krakenapps.pcap.util.ByteOrderConverter;
import java.io.EOFException;
import java.io.IOException;
-import java.util.*;
-
-import static org.apache.metron.pcap.Constants.*;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
public class PcapHelper {
@@ -81,6 +82,14 @@ public class PcapHelper {
}
}
+ /**
+ *
+ * @param topic
+ * @param timestamp
+ * @param partition kafka partition
+ * @param uuid
+ * @return filename in this format: pcap_topic_timestamp_partition_uuid, e.g. pcap_pcap_1494886105667571000_0_pcap-8-1494965816
+ */
public static String toFilename(String topic, long timestamp, String partition, String uuid)
{
return Joiner.on("_").join("pcap"
@@ -163,6 +172,44 @@ public class PcapHelper {
}
return null;
}
+
+ public static byte[] addHeaders(long tsNano, byte[] packet, Endianness endianness) {
+ byte[] ret = new byte[GLOBAL_HEADER_SIZE + PACKET_HEADER_SIZE + packet.length];
+ byte[] globalHeader = getPcapGlobalHeader(endianness);
+ int offset = 0;
+ System.arraycopy(globalHeader, 0, ret, offset, GLOBAL_HEADER_SIZE);
+ offset += globalHeader.length;
+ {
+ boolean swapBytes = swapBytes(endianness);
+ long micros = Long.divideUnsigned(tsNano, 1000);
+ int secs = (int)(micros / 1000000);
+ int usec = (int)(micros % 1000000);
+ int capLen = packet.length;
+ {
+ byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(secs):secs);
+ System.arraycopy(b, 0, ret, offset, Integer.BYTES);
+ offset += Integer.BYTES;
+ }
+ {
+ byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(usec):usec);
+ System.arraycopy(b, 0, ret, offset, Integer.BYTES);
+ offset += Integer.BYTES;
+ }
+ {
+ byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(capLen):capLen);
+ System.arraycopy(b, 0, ret, offset, Integer.BYTES);
+ offset += Integer.BYTES;
+ }
+ {
+ byte[] b = Bytes.toBytes(swapBytes?ByteOrderConverter.swap(capLen):capLen);
+ System.arraycopy(b, 0, ret, offset, Integer.BYTES);
+ offset += Integer.BYTES;
+ }
+ }
+ System.arraycopy(packet, 0, ret, offset, packet.length);
+ return ret;
+ }
+
public static byte[] addGlobalHeader(byte[] packet, Endianness endianness) {
byte[] globalHeader = getPcapGlobalHeader(endianness);
byte[] ret = new byte[packet.length + GLOBAL_HEADER_SIZE];
http://git-wip-us.apache.org/repos/asf/metron/blob/c0b08252/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
index 3d4a4b3..8d40e5f 100644
--- a/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
+++ b/metron-platform/metron-pcap/src/main/java/org/apache/metron/pcap/mr/PcapJob.java
@@ -52,6 +52,11 @@ public class PcapJob {
public static final String START_TS_CONF = "start_ts";
public static final String END_TS_CONF = "end_ts";
public static final String WIDTH_CONF = "width";
+
+ public static enum PCAP_COUNTER {
+ MALFORMED_PACKET_COUNT
+ }
+
public static class PcapPartitioner extends Partitioner<LongWritable, BytesWritable> implements Configurable {
private Configuration configuration;
Long start = null;
@@ -110,15 +115,23 @@ public class PcapJob {
// object will result in the whole set being passed through if any pass the filter. We cannot serialize PacketInfo
// objects back to byte arrays, otherwise we could support more than one packet.
// Note: short-circuit findAny() func on stream
- boolean send = filteredPacketInfo(value).findAny().isPresent();
+ List<PacketInfo> packetInfos;
+ try {
+ packetInfos = PcapHelper.toPacketInfo(value.copyBytes());
+ } catch(Exception e) {
+ // toPacketInfo is throwing RuntimeExceptions. Attempt to catch and count errors with malformed packets
+ context.getCounter(PCAP_COUNTER.MALFORMED_PACKET_COUNT).increment(1);
+ return;
+ }
+ boolean send = filteredPacketInfo(packetInfos).findAny().isPresent();
if (send) {
context.write(key, value);
}
}
}
- private Stream<PacketInfo> filteredPacketInfo(BytesWritable value) throws IOException {
- return PcapHelper.toPacketInfo(value.copyBytes()).stream().filter(filter);
+ private Stream<PacketInfo> filteredPacketInfo(List<PacketInfo> packetInfos) throws IOException {
+ return packetInfos.stream().filter(filter);
}
}
[04/44] metron git commit: METRON-915 platform_info script should
have node and npm version (ottobackwards) closes apache/incubator-metron#562
Posted by ma...@apache.org.
METRON-915 platform_info script should have node and npm version (ottobackwards) closes apache/incubator-metron#562
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/494643c3
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/494643c3
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/494643c3
Branch: refs/heads/Metron_0.4.0
Commit: 494643c38a64834a05ab3c6c2615aab6e9450a4e
Parents: 29e7c70
Author: ottobackwards <ot...@gmail.com>
Authored: Wed May 3 08:47:09 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Wed May 3 08:47:09 2017 -0400
----------------------------------------------------------------------
metron-deployment/scripts/platform-info.sh | 11 +++++++++++
1 file changed, 11 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/494643c3/metron-deployment/scripts/platform-info.sh
----------------------------------------------------------------------
diff --git a/metron-deployment/scripts/platform-info.sh b/metron-deployment/scripts/platform-info.sh
index f84d822..a1dce34 100755
--- a/metron-deployment/scripts/platform-info.sh
+++ b/metron-deployment/scripts/platform-info.sh
@@ -63,6 +63,17 @@ mvn --version
echo "--"
docker --version
+# node
+echo "--"
+echo "node"
+node --version
+
+# npm
+echo "--"
+echo "npm"
+npm --version
+
+
# operating system
echo "--"
uname -a
[26/44] metron git commit: METRON-955: Make the default sync policy
for HDFS Writer be based on the batch size closes apache/incubator-metron#589
Posted by ma...@apache.org.
METRON-955: Make the default sync policy for HDFS Writer be based on the batch size closes apache/incubator-metron#589
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/be030765
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/be030765
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/be030765
Branch: refs/heads/Metron_0.4.0
Commit: be03076599544f2baedb1b3010ad50625b2f32ae
Parents: c1c2121
Author: cstella <ce...@gmail.com>
Authored: Tue May 16 16:50:04 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Tue May 16 16:50:04 2017 -0400
----------------------------------------------------------------------
.../apache/metron/writer/hdfs/HdfsWriter.java | 22 ++++++++++++-----
.../metron/writer/hdfs/SourceHandler.java | 14 +++++------
.../metron/writer/hdfs/SyncPolicyCreator.java | 25 ++++++++++++++++++++
.../metron/writer/hdfs/HdfsWriterTest.java | 4 ++--
4 files changed, 50 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/be030765/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
index a86dfbc..e0ab502 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
@@ -40,16 +40,18 @@ import org.json.simple.JSONObject;
import java.io.IOException;
import java.io.Serializable;
import java.util.*;
+import java.util.function.Function;
public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
List<RotationAction> rotationActions = new ArrayList<>();
FileRotationPolicy rotationPolicy = new NoRotationPolicy();
- SyncPolicy syncPolicy = new CountSyncPolicy(1); //sync every time, duh.
+ SyncPolicy syncPolicy;
FileNameFormat fileNameFormat;
Map<SourceHandlerKey, SourceHandler> sourceHandlerMap = new HashMap<>();
int maxOpenFiles = 500;
transient StellarProcessor stellarProcessor;
transient Map stormConfig;
+ transient SyncPolicyCreator syncPolicyCreator;
public HdfsWriter withFileNameFormat(FileNameFormat fileNameFormat){
@@ -81,6 +83,14 @@ public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
this.stormConfig = stormConfig;
this.stellarProcessor = new StellarProcessor();
this.fileNameFormat.prepare(stormConfig,topologyContext);
+ if(syncPolicy != null) {
+ //if the user has specified the sync policy, we don't want to override their wishes.
+ syncPolicyCreator = (source,config) -> syncPolicy;
+ }
+ else {
+ //if the user has not, then we want to have the sync policy depend on the batch size.
+ syncPolicyCreator = (source, config) -> new CountSyncPolicy(config == null?1:config.getBatchSize(source));
+ }
}
@@ -92,18 +102,18 @@ public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
) throws Exception
{
BulkWriterResponse response = new BulkWriterResponse();
+
// Currently treating all the messages in a group for pass/failure.
try {
// Messages can all result in different HDFS paths, because of Stellar Expressions, so we'll need to iterate through
for(JSONObject message : messages) {
- Map<String, Object> val = configurations.getSensorConfig(sourceType);
String path = getHdfsPathExtension(
sourceType,
(String)configurations.getSensorConfig(sourceType).getOrDefault(IndexingConfigurations.OUTPUT_PATH_FUNCTION_CONF, ""),
message
);
- SourceHandler handler = getSourceHandler(sourceType, path);
- handler.handle(message);
+ SourceHandler handler = getSourceHandler(sourceType, path, configurations);
+ handler.handle(message, sourceType, configurations, syncPolicyCreator);
}
} catch (Exception e) {
response.addAllErrors(e, tuples);
@@ -142,7 +152,7 @@ public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
sourceHandlerMap.clear();
}
- synchronized SourceHandler getSourceHandler(String sourceType, String stellarResult) throws IOException {
+ synchronized SourceHandler getSourceHandler(String sourceType, String stellarResult, WriterConfiguration config) throws IOException {
SourceHandlerKey key = new SourceHandlerKey(sourceType, stellarResult);
SourceHandler ret = sourceHandlerMap.get(key);
if(ret == null) {
@@ -151,7 +161,7 @@ public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
}
ret = new SourceHandler(rotationActions,
rotationPolicy,
- syncPolicy,
+ syncPolicyCreator.create(sourceType, config),
new PathExtensionFileNameFormat(key.getStellarResult(), fileNameFormat),
new SourceHandlerCallback(sourceHandlerMap, key));
sourceHandlerMap.put(key, ret);
http://git-wip-us.apache.org/repos/asf/metron/blob/be030765/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
index fa6d8da..d895465 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SourceHandler.java
@@ -24,9 +24,11 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.client.HdfsDataOutputStream;
import org.apache.log4j.Logger;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
import org.apache.storm.hdfs.bolt.format.FileNameFormat;
import org.apache.storm.hdfs.bolt.rotation.FileRotationPolicy;
import org.apache.storm.hdfs.bolt.rotation.TimedRotationPolicy;
+import org.apache.storm.hdfs.bolt.sync.CountSyncPolicy;
import org.apache.storm.hdfs.bolt.sync.SyncPolicy;
import org.apache.storm.hdfs.common.rotation.RotationAction;
import org.json.simple.JSONObject;
@@ -34,6 +36,7 @@ import org.json.simple.JSONObject;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.*;
+import java.util.function.Function;
public class SourceHandler {
private static final Logger LOG = Logger.getLogger(SourceHandler.class);
@@ -61,13 +64,8 @@ public class SourceHandler {
initialize();
}
- public void handle(List<JSONObject> messages) throws Exception{
- for(JSONObject message : messages) {
- handle(message);
- }
- }
- protected void handle(JSONObject message) throws IOException {
+ protected void handle(JSONObject message, String sensor, WriterConfiguration config, SyncPolicyCreator syncPolicyCreator) throws IOException {
byte[] bytes = (message.toJSONString() + "\n").getBytes();
synchronized (this.writeLock) {
out.write(bytes);
@@ -79,7 +77,9 @@ public class SourceHandler {
} else {
this.out.hsync();
}
- this.syncPolicy.reset();
+ //recreate the sync policy for the next batch just in case something changed in the config
+ //and the sync policy depends on the config.
+ this.syncPolicy = syncPolicyCreator.create(sensor, config);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/be030765/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SyncPolicyCreator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SyncPolicyCreator.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SyncPolicyCreator.java
new file mode 100644
index 0000000..9b7d205
--- /dev/null
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/SyncPolicyCreator.java
@@ -0,0 +1,25 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.writer.hdfs;
+
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.storm.hdfs.bolt.sync.SyncPolicy;
+
+public interface SyncPolicyCreator {
+ SyncPolicy create(String sensor, WriterConfiguration config);
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/be030765/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
index 6153ed2..0118a15 100644
--- a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
+++ b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
@@ -209,7 +209,7 @@ public class HdfsWriterTest {
writer.init(new HashMap<String, String>(), createTopologyContext(), config);
for(int i = 0; i < maxFiles; i++) {
- writer.getSourceHandler(SENSOR_NAME, Integer.toString(i));
+ writer.getSourceHandler(SENSOR_NAME, Integer.toString(i), null);
}
}
@@ -223,7 +223,7 @@ public class HdfsWriterTest {
writer.init(new HashMap<String, String>(), createTopologyContext(), config);
for(int i = 0; i < maxFiles+1; i++) {
- writer.getSourceHandler(SENSOR_NAME, Integer.toString(i));
+ writer.getSourceHandler(SENSOR_NAME, Integer.toString(i), null);
}
}
[25/44] metron git commit: METRON-954: Create ability to change
output topic of parsers from the CLI closes apache/incubator-metron#588
Posted by ma...@apache.org.
METRON-954: Create ability to change output topic of parsers from the CLI closes apache/incubator-metron#588
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/c1c21211
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/c1c21211
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/c1c21211
Branch: refs/heads/Metron_0.4.0
Commit: c1c212117d6123f2897a606fedbb2583fb3bb9c3
Parents: 6c836d1
Author: cstella <ce...@gmail.com>
Authored: Tue May 16 15:24:17 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Tue May 16 15:24:17 2017 -0400
----------------------------------------------------------------------
.../parsers/topology/ParserTopologyBuilder.java | 8 +++++---
.../metron/parsers/topology/ParserTopologyCLI.java | 17 ++++++++++++++++-
.../parsers/integration/ParserIntegrationTest.java | 1 +
.../components/ParserTopologyComponent.java | 13 +++++++++++--
.../parsers/topology/ParserTopologyCLITest.java | 17 +++++++++++++++++
5 files changed, 50 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/c1c21211/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
index 196c19d..0c88573 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
@@ -74,7 +74,8 @@ public class ParserTopologyBuilder {
int errorWriterParallelism,
int errorWriterNumTasks,
Map<String, Object> kafkaSpoutConfig,
- Optional<String> securityProtocol
+ Optional<String> securityProtocol,
+ Optional<String> outputTopic
) throws Exception {
// fetch configuration from zookeeper
@@ -88,7 +89,7 @@ public class ParserTopologyBuilder {
.setNumTasks(spoutNumTasks);
// create the parser bolt
- ParserBolt parserBolt = createParserBolt(zookeeperUrl, brokerUrl, sensorType, securityProtocol, configs, parserConfig);
+ ParserBolt parserBolt = createParserBolt(zookeeperUrl, brokerUrl, sensorType, securityProtocol, configs, parserConfig, outputTopic);
builder.setBolt("parserBolt", parserBolt, parserParallelism)
.setNumTasks(parserNumTasks)
.shuffleGrouping("kafkaSpout");
@@ -170,6 +171,7 @@ public class ParserTopologyBuilder {
, Optional<String> securityProtocol
, ParserConfigurations configs
, SensorParserConfig parserConfig
+ , Optional<String> outputTopic
)
{
@@ -182,7 +184,7 @@ public class ParserTopologyBuilder {
createKafkaWriter( brokerUrl
, zookeeperUrl
, securityProtocol
- ).withTopic(Constants.ENRICHMENT_TOPIC) :
+ ).withTopic(outputTopic.orElse(Constants.ENRICHMENT_TOPIC)) :
ReflectionUtils.createInstance(parserConfig.getWriterClassName());
writer.configure(sensorType, new ParserWriterConfiguration(configs));
http://git-wip-us.apache.org/repos/asf/metron/blob/c1c21211/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyCLI.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyCLI.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyCLI.java
index 7523333..c68e101 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyCLI.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyCLI.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.parsers.topology;
+import org.apache.metron.common.Constants;
import org.apache.metron.storm.kafka.flux.SpoutConfiguration;
import org.apache.storm.Config;
import org.apache.storm.LocalCluster;
@@ -188,6 +189,18 @@ public class ParserTopologyCLI {
return o;
}
)
+ ,OUTPUT_TOPIC("ot", code -> {
+ Option o = new Option(code
+ , "output_topic"
+ , true
+ , "The output kafka topic for the parser. If unset, the default is " + Constants.ENRICHMENT_TOPIC
+ );
+ o.setArgName("KAFKA_TOPIC");
+ o.setRequired(false);
+ o.setType(String.class);
+ return o;
+ }
+ )
,TEST("t", code ->
{
Option o = new Option("t", "test", true, "Run in Test Mode");
@@ -296,6 +309,7 @@ public class ParserTopologyCLI {
if(ParserOptions.SPOUT_CONFIG.has(cmd)) {
spoutConfig = readSpoutConfig(new File(ParserOptions.SPOUT_CONFIG.get(cmd)));
}
+ Optional<String> outputTopic = ParserOptions.OUTPUT_TOPIC.has(cmd)?Optional.of(ParserOptions.OUTPUT_TOPIC.get(cmd)):Optional.empty();
Optional<String> securityProtocol = ParserOptions.SECURITY_PROTOCOL.has(cmd)?Optional.of(ParserOptions.SECURITY_PROTOCOL.get(cmd)):Optional.empty();
securityProtocol = getSecurityProtocol(securityProtocol, spoutConfig);
TopologyBuilder builder = ParserTopologyBuilder.build(zookeeperUrl,
@@ -308,7 +322,8 @@ public class ParserTopologyCLI {
errorParallelism,
errorNumTasks,
spoutConfig,
- securityProtocol
+ securityProtocol,
+ outputTopic
);
Config stormConf = ParserOptions.getConfig(cmd);
if (ParserOptions.TEST.has(cmd)) {
http://git-wip-us.apache.org/repos/asf/metron/blob/c1c21211/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
index defd815..fe6475d 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
@@ -62,6 +62,7 @@ public abstract class ParserIntegrationTest extends BaseIntegrationTest {
ParserTopologyComponent parserTopologyComponent = new ParserTopologyComponent.Builder()
.withSensorType(sensorType)
.withTopologyProperties(topologyProperties)
+ .withOutputTopic(Constants.ENRICHMENT_TOPIC)
.withBrokerUrl(kafkaComponent.getBrokerList()).build();
//UnitTestHelper.verboseLogging();
http://git-wip-us.apache.org/repos/asf/metron/blob/c1c21211/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
index b6a76d0..6ad7427 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
@@ -45,11 +45,13 @@ public class ParserTopologyComponent implements InMemoryComponent {
private String brokerUrl;
private String sensorType;
private LocalCluster stormCluster;
+ private String outputTopic;
public static class Builder {
Properties topologyProperties;
String brokerUrl;
String sensorType;
+ String outputTopic;
public Builder withTopologyProperties(Properties topologyProperties) {
this.topologyProperties = topologyProperties;
return this;
@@ -63,15 +65,21 @@ public class ParserTopologyComponent implements InMemoryComponent {
return this;
}
+ public Builder withOutputTopic(String topic) {
+ this.outputTopic = topic;
+ return this;
+ }
+
public ParserTopologyComponent build() {
- return new ParserTopologyComponent(topologyProperties, brokerUrl, sensorType);
+ return new ParserTopologyComponent(topologyProperties, brokerUrl, sensorType, outputTopic);
}
}
- public ParserTopologyComponent(Properties topologyProperties, String brokerUrl, String sensorType) {
+ public ParserTopologyComponent(Properties topologyProperties, String brokerUrl, String sensorType, String outputTopic) {
this.topologyProperties = topologyProperties;
this.brokerUrl = brokerUrl;
this.sensorType = sensorType;
+ this.outputTopic = outputTopic;
}
@@ -89,6 +97,7 @@ public class ParserTopologyComponent implements InMemoryComponent {
, 1
, null
, Optional.empty()
+ , Optional.ofNullable(outputTopic)
);
Map<String, Object> stormConf = new HashMap<>();
stormConf.put(Config.TOPOLOGY_DEBUG, true);
http://git-wip-us.apache.org/repos/asf/metron/blob/c1c21211/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/topology/ParserTopologyCLITest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/topology/ParserTopologyCLITest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/topology/ParserTopologyCLITest.java
index ac73a2b..5f536a5 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/topology/ParserTopologyCLITest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/topology/ParserTopologyCLITest.java
@@ -116,6 +116,7 @@ public class ParserTopologyCLITest {
.build(true);
UnitTestHelper.setLog4jLevel(Parser.class, Level.ERROR);
}
+
public void happyPath(boolean longOpt) throws ParseException {
CommandLine cli = new CLIBuilder().with(ParserTopologyCLI.ParserOptions.BROKER_URL, "mybroker")
.with(ParserTopologyCLI.ParserOptions.ZK_QUORUM, "myzk")
@@ -147,6 +148,22 @@ public class ParserTopologyCLITest {
Assert.assertEquals(3, config.get(Config.TOPOLOGY_MAX_TASK_PARALLELISM));
Assert.assertEquals(4, config.get(Config.TOPOLOGY_MESSAGE_TIMEOUT_SECS));
}
+
+ @Test
+ public void testOutputTopic() throws Exception {
+ testOutputTopic(true);
+ testOutputTopic(false);
+ }
+
+ public void testOutputTopic(boolean longOpt) throws ParseException {
+ CommandLine cli = new CLIBuilder().with(ParserTopologyCLI.ParserOptions.BROKER_URL, "mybroker")
+ .with(ParserTopologyCLI.ParserOptions.ZK_QUORUM, "myzk")
+ .with(ParserTopologyCLI.ParserOptions.SENSOR_TYPE, "mysensor")
+ .with(ParserTopologyCLI.ParserOptions.OUTPUT_TOPIC, "my_topic")
+ .build(longOpt);
+ Assert.assertEquals("my_topic", ParserTopologyCLI.ParserOptions.OUTPUT_TOPIC.get(cli));
+ }
+
/**
{
"string" : "foo"
[21/44] metron git commit: METRON-932: Change HLLP Stellar functions
to accept empty lists (mmiklavc) closes apache/incubator-metron#566
Posted by ma...@apache.org.
METRON-932: Change HLLP Stellar functions to accept empty lists (mmiklavc) closes apache/incubator-metron#566
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/345eba8d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/345eba8d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/345eba8d
Branch: refs/heads/Metron_0.4.0
Commit: 345eba8d5940f2789e5a860eb51e8fa9f003c676
Parents: 9a5e1ba
Author: mmiklavc <mi...@gmail.com>
Authored: Fri May 12 11:59:28 2017 -0600
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Fri May 12 11:59:28 2017 -0600
----------------------------------------------------------------------
metron-analytics/metron-statistics/README.md | 4 ++--
.../approximation/HyperLogLogPlusFunctions.java | 20 +++++++++++++-------
...HyperLogLogPlusFunctionsIntegrationTest.java | 13 +++++++++++++
.../HyperLogLogPlusFunctionsTest.java | 17 +++++++++++++----
4 files changed, 41 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/345eba8d/metron-analytics/metron-statistics/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/README.md b/metron-analytics/metron-statistics/README.md
index cfd44f2..74e78b0 100644
--- a/metron-analytics/metron-statistics/README.md
+++ b/metron-analytics/metron-statistics/README.md
@@ -23,7 +23,7 @@ functions can be used from everywhere where Stellar is used.
* Description: Returns HyperLogLogPlus-estimated cardinality for this set. See [HLLP README](HLLP.md)
* Input:
* hyperLogLogPlus - the hllp set
- * Returns: Long value representing the cardinality for this set
+ * Returns: Long value representing the cardinality for this set. Cardinality of a null set is 0.
#### `HLLP_INIT`
* Description: Initializes the HyperLogLogPlus estimator set. p must be a value between 4 and sp and sp must be less than 32 and greater than 4. See [HLLP README](HLLP.md)
@@ -36,7 +36,7 @@ functions can be used from everywhere where Stellar is used.
* Description: Merge hllp sets together. The resulting estimator is initialized with p and sp precision values from the first provided hllp estimator set. See [HLLP README](HLLP.md)
* Input:
* hllp - List of hllp estimators to merge. Takes a single hllp set or a list.
- * Returns: A new merged HyperLogLogPlus estimator set
+ * Returns: A new merged HyperLogLogPlus estimator set. Passing an empty list returns null.
### Mathematical Functions
http://git-wip-us.apache.org/repos/asf/metron/blob/345eba8d/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctions.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctions.java b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctions.java
index 7578403..f123494 100644
--- a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctions.java
+++ b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctions.java
@@ -61,16 +61,22 @@ public class HyperLogLogPlusFunctions {
, name = "CARDINALITY"
, description = "Returns HyperLogLogPlus-estimated cardinality for this set. See [HLLP README](HLLP.md)"
, params = {"hyperLogLogPlus - the hllp set"}
- , returns = "Long value representing the cardinality for this set"
+ , returns = "Long value representing the cardinality for this set. Cardinality of a null set is 0."
)
public static class HLLPCardinality extends BaseStellarFunction {
@Override
public Object apply(List<Object> args) {
- if (args.size() < 1) {
- throw new IllegalArgumentException("Must pass an hllp set to get the cardinality for");
+ if (args.size() == 1) {
+ if (args.get(0) instanceof HyperLogLogPlus) {
+ HyperLogLogPlus hllpSet = (HyperLogLogPlus) args.get(0);
+ return hllpSet.cardinality();
+ } else {
+ return 0L;
+ }
+ } else {
+ return 0L;
}
- return ((HyperLogLogPlus) args.get(0)).cardinality();
}
}
@@ -113,7 +119,7 @@ public class HyperLogLogPlusFunctions {
, name = "MERGE"
, description = "Merge hllp sets together. The resulting estimator is initialized with p and sp precision values from the first provided hllp estimator set. See [HLLP README](HLLP.md)"
, params = {"hllp - List of hllp estimators to merge. Takes a single hllp set or a list."}
- , returns = "A new merged HyperLogLogPlus estimator set"
+ , returns = "A new merged HyperLogLogPlus estimator set. Passing an empty list returns null."
)
public static class HLLPMerge extends BaseStellarFunction {
@@ -128,8 +134,8 @@ public class HyperLogLogPlusFunctions {
} else {
estimators.add(args.get(0));
}
- if (estimators.size() < 1) {
- throw new IllegalArgumentException("Must pass 1..n hllp sets to merge");
+ if (estimators.size() == 0) {
+ return null;
}
HyperLogLogPlus hllp = ConversionUtils.convert(estimators.get(0), HyperLogLogPlus.class);
if (estimators.size() > 1) {
http://git-wip-us.apache.org/repos/asf/metron/blob/345eba8d/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsIntegrationTest.java b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsIntegrationTest.java
index 85e938a..cd9cae4 100644
--- a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsIntegrationTest.java
+++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsIntegrationTest.java
@@ -34,6 +34,7 @@ public class HyperLogLogPlusFunctionsIntegrationTest {
put("val2", "miklavcic");
put("val3", "metron");
put("val4", "batman");
+ put("nullArg", null);
}};
/**
@@ -117,4 +118,16 @@ public class HyperLogLogPlusFunctionsIntegrationTest {
Assert.assertThat("Incorrect cardinality returned", estimate, equalTo(4L));
}
+ /**
+ *HLLP_CARDINALITY(nullArg)
+ */
+ @Multiline
+ private static String zeroCardinalityRule;
+
+ @Test
+ public void cardinality_of_null_value_is_0() {
+ Long estimate = (Long) StellarProcessorUtils.run(zeroCardinalityRule, values);
+ Assert.assertThat("Incorrect cardinality returned", estimate, equalTo(0L));
+ }
+
}
http://git-wip-us.apache.org/repos/asf/metron/blob/345eba8d/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsTest.java b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsTest.java
index 20be8e4..a43eeb7 100644
--- a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsTest.java
+++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/approximation/HyperLogLogPlusFunctionsTest.java
@@ -23,7 +23,9 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
+import java.util.ArrayList;
import java.util.Arrays;
+import java.util.List;
import static org.hamcrest.CoreMatchers.*;
@@ -84,10 +86,11 @@ public class HyperLogLogPlusFunctionsTest {
}
@Test
- public void hllp_cardinality_with_invalid_arguments_throws_exception() {
- thrown.expect(IllegalArgumentException.class);
- thrown.expectMessage("Must pass an hllp set to get the cardinality for");
- new HyperLogLogPlusFunctions.HLLPCardinality().apply(ImmutableList.of());
+ public void hllp_cardinality_returns_0_for_null_set() {
+ List nullArg = new ArrayList() {{
+ add(null);
+ }};
+ Assert.assertThat("Cardinality should be 0", new HyperLogLogPlusFunctions.HLLPCardinality().apply(nullArg), equalTo(0L));
}
@Test
@@ -139,4 +142,10 @@ public class HyperLogLogPlusFunctionsTest {
new HyperLogLogPlusFunctions.HLLPMerge().apply(ImmutableList.of(hllp1, hllp2));
}
+ @Test
+ public void merge_returns_null_if_passed_an_empty_list_to_merge() {
+ List emptyList = ImmutableList.of();
+ Assert.assertThat("Should be empty list", new HyperLogLogPlusFunctions.HLLPMerge().apply(ImmutableList.of(emptyList)), equalTo(null));
+ }
+
}
[40/44] metron git commit: METRON-964 Add logging in JoinBolt for
unexpected cache evictions (merrimanr) closes apache/metron#595
Posted by ma...@apache.org.
METRON-964 Add logging in JoinBolt for unexpected cache evictions (merrimanr) closes apache/metron#595
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/f127c6dc
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/f127c6dc
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/f127c6dc
Branch: refs/heads/Metron_0.4.0
Commit: f127c6dcfd8c2a2d19314b98eb3840da90d2ab44
Parents: 61105c7
Author: merrimanr <me...@gmail.com>
Authored: Wed May 31 14:54:14 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Wed May 31 14:54:14 2017 -0500
----------------------------------------------------------------------
.../apache/metron/enrichment/bolt/JoinBolt.java | 24 +++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/f127c6dc/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
index 3bbb3f5..a8e793d 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
@@ -21,6 +21,9 @@ import com.google.common.base.Joiner;
import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalCause;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
import com.google.common.collect.Sets;
import org.apache.metron.common.Constants;
import org.apache.metron.common.bolt.ConfiguredEnrichmentBolt;
@@ -90,11 +93,30 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
}
};
cache = CacheBuilder.newBuilder().maximumSize(maxCacheSize)
- .expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES)
+ .expireAfterWrite(maxTimeRetain, TimeUnit.MINUTES).removalListener(new JoinRemoveListener())
.build(loader);
prepare(map, topologyContext);
}
+ class JoinRemoveListener implements RemovalListener<String, Map<String, V>> {
+
+ @Override
+ public void onRemoval(RemovalNotification<String, Map<String, V>> removalNotification) {
+ if (removalNotification.getCause() == RemovalCause.SIZE) {
+ String errorMessage = "Join cache reached max size limit. Increase the maxCacheSize setting or add more tasks to enrichment/threatintel join bolt.";
+ Exception exception = new Exception(errorMessage);
+ LOG.error(errorMessage, exception);
+ collector.reportError(exception);
+ }
+ if (removalNotification.getCause() == RemovalCause.EXPIRED) {
+ String errorMessage = "Message was in the join cache too long which may be caused by slow enrichments/threatintels. Increase the maxTimeRetain setting.";
+ Exception exception = new Exception(errorMessage);
+ LOG.error(errorMessage, exception);
+ collector.reportError(exception);
+ }
+ }
+ }
+
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
[17/44] metron git commit: METRON-886 ansible-docker issues on
centos6 with docker-io (ottobackwards) closes apache/incubator-metron#546
Posted by ma...@apache.org.
METRON-886 ansible-docker issues on centos6 with docker-io (ottobackwards) closes apache/incubator-metron#546
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e375936a
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e375936a
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e375936a
Branch: refs/heads/Metron_0.4.0
Commit: e375936a17ee0ce743d5d9f4446ea21dee474450
Parents: 64b66f2
Author: ottobackwards <ot...@gmail.com>
Authored: Wed May 10 14:36:08 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Wed May 10 14:36:08 2017 -0400
----------------------------------------------------------------------
metron-deployment/packaging/docker/ansible-docker/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/e375936a/metron-deployment/packaging/docker/ansible-docker/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/ansible-docker/Dockerfile b/metron-deployment/packaging/docker/ansible-docker/Dockerfile
index 825e8a7..3fc8de5 100644
--- a/metron-deployment/packaging/docker/ansible-docker/Dockerfile
+++ b/metron-deployment/packaging/docker/ansible-docker/Dockerfile
@@ -35,7 +35,7 @@ RUN python2.7 setup.py install
RUN easy_install-2.7 pip
RUN pip2.7 install ansible==2.0.0.2
RUN pip2.7 install boto
-COPY ansible.cfg /root
+COPY ansible.cfg /root/
ENV ANSIBLE_CONFIG /root/ansible.cfg
RUN yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
RUN yum install -y which
[32/44] metron git commit: METRON-965: In the case where we specify
the syncpolicy in the HDFS Writer,
we do not properly clone and end up syncing for every record closes
apache/incubator-metron#596
Posted by ma...@apache.org.
METRON-965: In the case where we specify the syncpolicy in the HDFS Writer, we do not properly clone and end up syncing for every record closes apache/incubator-metron#596
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/64473d4e
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/64473d4e
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/64473d4e
Branch: refs/heads/Metron_0.4.0
Commit: 64473d4e8399e71d0ac1a81b6e78e72b982e42f8
Parents: fee758b
Author: cstella <ce...@gmail.com>
Authored: Sat May 20 09:57:13 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Sat May 20 09:57:13 2017 -0400
----------------------------------------------------------------------
.../writer/hdfs/ClonedSyncPolicyCreator.java | 47 ++++++++++++++++++++
.../apache/metron/writer/hdfs/HdfsWriter.java | 6 +--
.../hdfs/ClonedSyncPolicyCreatorTest.java | 42 +++++++++++++++++
3 files changed, 92 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/64473d4e/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreator.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreator.java
new file mode 100644
index 0000000..4d32fc9
--- /dev/null
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreator.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.writer.hdfs;
+
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.apache.metron.common.utils.SerDeUtils;
+import org.apache.storm.hdfs.bolt.sync.SyncPolicy;
+
+public class ClonedSyncPolicyCreator implements SyncPolicyCreator {
+ SyncPolicy syncPolicy;
+ public ClonedSyncPolicyCreator(SyncPolicy policy) {
+ syncPolicy = policy;
+ }
+
+ @Override
+ public SyncPolicy create(String sensor, WriterConfiguration config) {
+ try {
+ //we do a deep clone of the SyncPolicy via kryo serialization. This gives us a fresh policy
+ //to work with. The reason we need a fresh policy is that we want to ensure each handler
+ //(one handler per task & sensor type and one handler per file) has its own sync policy.
+ // Reusing a sync policy is a bad idea, so we need to clone it here. Unfortunately the
+ // SyncPolicy object does not implement Cloneable, so we'll need to clone it via serialization
+ //to get a fresh policy object. Note: this would be expensive if it was in the critical path,
+ // but should be called infrequently (once per sync).
+ byte[] serializedForm = SerDeUtils.toBytes(syncPolicy);
+ return SerDeUtils.fromBytes(serializedForm, SyncPolicy.class);
+ }
+ catch (Exception e) {
+ throw new IllegalStateException(e.getMessage(), e);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/64473d4e/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
index e0ab502..c5d1e4f 100644
--- a/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
+++ b/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
@@ -24,6 +24,7 @@ import org.apache.metron.common.dsl.StellarFunctions;
import org.apache.metron.common.dsl.VariableResolver;
import org.apache.metron.common.stellar.StellarCompiler;
import org.apache.metron.common.stellar.StellarProcessor;
+import org.apache.metron.common.utils.SerDeUtils;
import org.apache.storm.task.TopologyContext;
import org.apache.storm.tuple.Tuple;
import org.apache.metron.common.configuration.writer.WriterConfiguration;
@@ -37,8 +38,7 @@ import org.apache.storm.hdfs.bolt.sync.SyncPolicy;
import org.apache.storm.hdfs.common.rotation.RotationAction;
import org.json.simple.JSONObject;
-import java.io.IOException;
-import java.io.Serializable;
+import java.io.*;
import java.util.*;
import java.util.function.Function;
@@ -85,7 +85,7 @@ public class HdfsWriter implements BulkMessageWriter<JSONObject>, Serializable {
this.fileNameFormat.prepare(stormConfig,topologyContext);
if(syncPolicy != null) {
//if the user has specified the sync policy, we don't want to override their wishes.
- syncPolicyCreator = (source,config) -> syncPolicy;
+ syncPolicyCreator = new ClonedSyncPolicyCreator(syncPolicy);
}
else {
//if the user has not, then we want to have the sync policy depend on the batch size.
http://git-wip-us.apache.org/repos/asf/metron/blob/64473d4e/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreatorTest.java b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreatorTest.java
new file mode 100644
index 0000000..092bf0f
--- /dev/null
+++ b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/ClonedSyncPolicyCreatorTest.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.writer.hdfs;
+
+import org.apache.storm.hdfs.bolt.sync.CountSyncPolicy;
+import org.apache.storm.hdfs.bolt.sync.SyncPolicy;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class ClonedSyncPolicyCreatorTest {
+
+ @Test
+ public void testClonedPolicy() {
+ CountSyncPolicy basePolicy = new CountSyncPolicy(5);
+ ClonedSyncPolicyCreator creator = new ClonedSyncPolicyCreator(basePolicy);
+ //ensure cloned policy continues to work and adheres to the contract: mark on 5th call.
+ SyncPolicy clonedPolicy = creator.create("blah", null);
+ for(int i = 0;i < 4;++i) {
+ Assert.assertFalse(clonedPolicy.mark(null, i));
+ }
+ Assert.assertTrue(clonedPolicy.mark(null, 5));
+ //reclone policy and ensure it adheres to the original contract.
+ clonedPolicy = creator.create("blah", null);
+ Assert.assertFalse(clonedPolicy.mark(null, 0));
+ }
+}
[19/44] metron git commit: METRON-912 Metron vagrant setup steps no
longer work (JonZeolla via nickwallen) closes apache/incubator-metron#560
Posted by ma...@apache.org.
METRON-912 Metron vagrant setup steps no longer work (JonZeolla via nickwallen) closes apache/incubator-metron#560
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/53990090
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/53990090
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/53990090
Branch: refs/heads/Metron_0.4.0
Commit: 53990090e554aff0873ef8ddf1a492fcb05604c9
Parents: b3e7222
Author: JonZeolla <ze...@gmail.com>
Authored: Fri May 12 09:09:59 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Fri May 12 09:09:59 2017 -0400
----------------------------------------------------------------------
.../vagrant/full-dev-platform/README.md | 15 ++++++++-------
1 file changed, 8 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/53990090/metron-deployment/vagrant/full-dev-platform/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/vagrant/full-dev-platform/README.md b/metron-deployment/vagrant/full-dev-platform/README.md
index 177b905..8f13b17 100644
--- a/metron-deployment/vagrant/full-dev-platform/README.md
+++ b/metron-deployment/vagrant/full-dev-platform/README.md
@@ -14,10 +14,10 @@ The computer used to deploy Apache Metron will need to have the following compon
- [Ansible](https://github.com/ansible/ansible) (2.0.0.2 or 2.2.2.0)
- [Docker](https://www.docker.com/community-edition)
- - [Vagrant](https://www.vagrantup.com) 1.8.1
- - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) `vagrant plugin install vagrant-hostmanager`
- - [Virtualbox](https://virtualbox.org) 5.0.16
- - Python 2.7.11
+ - [Vagrant](https://www.vagrantup.com) 1.8+
+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager)
+ - [Virtualbox](https://virtualbox.org) 5.0+
+ - Python 2.7
- Maven 3.3.9
#### macOS
@@ -30,11 +30,12 @@ Any platform that supports these tools is suitable, but the following instructio
```
brew cask install vagrant virtualbox java docker
- brew install maven git
+ brew install maven@3.3 git
+ pip install ansible==2.2.2.0
+ vagrant plugin install vagrant-hostmanager
+ open /Applications/Docker.app
```
-1. Install Ansible by following the instructions [here](http://docs.ansible.com/ansible/intro_installation.html#latest-releases-via-pip).
-
### Deploy Metron
1. Ensure that the Docker service is running.
[08/44] metron git commit: METRON-896 Document Having Kerberos Issue
Renewable Tickets (justinleet) closes apache/incubator-metron#553
Posted by ma...@apache.org.
METRON-896 Document Having Kerberos Issue Renewable Tickets (justinleet) closes apache/incubator-metron#553
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e08b7d47
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e08b7d47
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e08b7d47
Branch: refs/heads/Metron_0.4.0
Commit: e08b7d47f3f644e12a9abb59bd3901d93c6d34af
Parents: 716bda3
Author: justinleet <ju...@gmail.com>
Authored: Mon May 8 09:33:09 2017 -0400
Committer: leet <le...@apache.org>
Committed: Mon May 8 09:33:09 2017 -0400
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 22 +++++++++++++++++++++-
1 file changed, 21 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/e08b7d47/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index b444b0e..8a7f07a 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -27,7 +27,6 @@ Setup
export ZOOKEEPER=node1:2181
export ELASTICSEARCH=node1:9200
export BROKERLIST=node1:6667
-
export HDP_HOME="/usr/hdp/current"
export KAFKA_HOME="${HDP_HOME}/kafka-broker"
export METRON_VERSION="0.4.0"
@@ -72,6 +71,16 @@ Setup a KDC
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
```
+1. Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section
+ ```
+ max_renewable_life = 7d
+ ```
+
+ If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:
+ ```
+ Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
+ ```
+
1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
```
@@ -238,6 +247,17 @@ Storm Authorization
cd /home/metron/.storm
```
+1. Ensure the Metron keytab is renewable. Look for the 'R' flag from the following command
+ ```
+ klist -f
+ ```
+
+ If not present, modify the appropriate principals to allow renewable tickets. Adjust the parameters to match desired KDC parameters
+ ```
+ kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
+ kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
+ ```
+
1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
```
[11/44] metron git commit: METRON-935 EC2 Deployment Failure - Could
Not Create Blueprint (merrimanr) closes apache/incubator-metron#568
Posted by ma...@apache.org.
METRON-935 EC2 Deployment Failure - Could Not Create Blueprint (merrimanr) closes apache/incubator-metron#568
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/876a5fc1
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/876a5fc1
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/876a5fc1
Branch: refs/heads/Metron_0.4.0
Commit: 876a5fc1379e0027d82941d956bd85660bc2c2ac
Parents: 8cf109d
Author: merrimanr <me...@gmail.com>
Authored: Tue May 9 13:23:02 2017 -0500
Committer: merrimanr <me...@gmail.com>
Committed: Tue May 9 13:23:02 2017 -0500
----------------------------------------------------------------------
metron-deployment/roles/ambari_config/vars/small_cluster.yml | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/876a5fc1/metron-deployment/roles/ambari_config/vars/small_cluster.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/small_cluster.yml b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
index 76d7761..5f261a0 100644
--- a/metron-deployment/roles/ambari_config/vars/small_cluster.yml
+++ b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
@@ -98,6 +98,11 @@ required_configurations:
storm_rest_addr: "{{ groups.ambari_slave[1] }}:8744"
es_hosts: "{{ groups.web[0] }},{{ groups.search | join(',') }}"
zeppelin_server_url: "{{ groups.zeppelin[0] }}"
+ metron_jdbc_driver: "org.h2.Driver"
+ metron_jdbc_url: "jdbc:h2:file:~/metrondb"
+ metron_jdbc_username: "root"
+ metron_jdbc_password: "root"
+ metron_jdbc_platform: "h2"
- kibana-env:
kibana_pid_dir: /var/run/kibana
kibana_es_url: http://{{ groups.web[0] }}:9200
[18/44] metron git commit: METRON-943 Create traffic connections
report in zeppelin (justinleet) closes apache/incubator-metron#573
Posted by ma...@apache.org.
METRON-943 Create traffic connections report in zeppelin (justinleet) closes apache/incubator-metron#573
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/b3e7222f
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/b3e7222f
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/b3e7222f
Branch: refs/heads/Metron_0.4.0
Commit: b3e7222f1c4fb527ff3b18e8bb47383528d2eb57
Parents: e375936
Author: justinleet <ju...@gmail.com>
Authored: Fri May 12 08:49:28 2017 -0400
Committer: leet <le...@apache.org>
Committed: Fri May 12 08:49:28 2017 -0400
----------------------------------------------------------------------
metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec | 3 +++
.../config/zeppelin/metron/metron-connection-volume-report.json | 1 +
2 files changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/b3e7222f/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index e851c7f..b49e0e6 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -271,6 +271,7 @@ This package installs the Metron Indexing files
%{metron_home}/config/zeppelin/metron/metron-yaf-telemetry.json
%{metron_home}/config/zeppelin/metron/metron-connection-report.json
%{metron_home}/config/zeppelin/metron/metron-ip-report.json
+%{metron_home}/config/zeppelin/metron/metron-connection-volume-report.json
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -400,6 +401,8 @@ This package installs the Metron Management UI %{metron_home}
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%changelog
+* Tue May 9 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
+- Add Zeppelin Connection Volume Report Dashboard
* Thu May 4 2017 Ryan Merriman <me...@gmail.com> - 0.4.0
- Added REST
* Tue May 2 2017 David Lyle <dl...@gmail.com> - 0.4.0
http://git-wip-us.apache.org/repos/asf/metron/blob/b3e7222f/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-volume-report.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-volume-report.json b/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-volume-report.json
new file mode 100644
index 0000000..22765d4
--- /dev/null
+++ b/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-volume-report.json
@@ -0,0 +1 @@
+{"paragraphs":[{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1494339150477_-816854736","id":"20170509-141230_1340330181","dateCreated":"2017-05-09T14:12:30+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:7251","text":"%spark.sql\n\n#\n# load the Yaf telemetry that has been archived by Metron\n#\ncreate temporary table yaf\n using org.apache.spark.sql.json\n options (path \"hdfs:///apps/metron/indexing/indexed/yaf\")","dateUpdated":"2017-05-09T14:20:10+0000","dateFinished":"2017-05-09T14:20:11+0000","dateStarted":"2017-05-09T14:20:10+0000","result":{"code":"SUCCESS","type":"TEXT","msg":""}},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_14943391
84335_-2067041830","id":"20170509-141304_1479312597","dateCreated":"2017-05-09T14:13:04+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:7322","text":"%md\n\n### Connection Volume (Source) - Yaf\n\nThe volume of connections made from source IPs\n\nThis IPs retrieved are given by an IPv4 CIDR block.","dateUpdated":"2017-05-09T14:22:02+0000","dateFinished":"2017-05-09T14:22:02+0000","dateStarted":"2017-05-09T14:22:02+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Connection Volume (Source) - Yaf</h3>\n<p>The volume of connections made from source IPs</p>\n<p>This IPs retrieved are given by an IPv4 CIDR block.</p>\n"}},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"CIDR":"192.0.0.0/8"},"forms":{"CIDR":{"name":"CIDR","displayName":"CIDR","type":"input","defaultValue":"","hidden":false}}},"jobName":"paragraph_14943391
91894_-766575224","id":"20170509-141311_2132481247","dateCreated":"2017-05-09T14:13:11+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:7398","text":"%spark\nimport org.apache.spark.sql.Row\nimport scala.concurrent.duration._\nimport java.util.concurrent.TimeUnit\nimport org.apache.commons.net.util.SubnetUtils\n\nval cidr = z.input(\"CIDR\").toString\n@transient val utils = new SubnetUtils(cidr)\nutils.setInclusiveHostCount(true)\n@transient val info = utils.getInfo\n\nval ipToLong = (ipAddress: String) => {\n val ipAddressInArray = ipAddress.split(\"\\\\.\")\n var result = 0L\n for (i <- 0 to ipAddressInArray.length-1) {\n\t val power = 3 - i\n\t val ip = Integer.parseInt(ipAddressInArray(i))\n\t result += ip * Math.pow(256.0, power.toDouble).toLong\n }\n result\n}\n\nval bcLow = sc.broadcast(ipToLong(info.getLowAddress))\nval bcHigh = sc.broadcast(ipToLong(info.getHighAddress))\nval bcIpToLong = sc.broadcast(ipToLong
)\n\nval results = sqlContext.sql(\ns\"\"\"SELECT\n ip_src_addr,\n COUNT(*) AS count\nFROM\n yaf\nGROUP BY ip_src_addr\nORDER BY ip_src_addr\n\"\"\").flatMap {\n case Row(ip_src_addr: String, count: Long) => {\n val longSrc = bcIpToLong.value(ip_src_addr)\n if(bcLow.value <= longSrc && longSrc <= bcHigh.value) {\n\t\t List(ip_src_addr + \"\\t\" + count)\n } else {\n List.empty[String]\n }\n }\n }.collect()\n\nprint(\"%table ip_src_addr\\tcount\\n\" + results.mkString(\"\\n\"))","dateUpdated":"2017-05-09T14:20:10+0000","dateFinished":"2017-05-09T14:20:21+0000","dateStarted":"2017-05-09T14:20:10+0000","result":{"code":"SUCCESS","type":"TABLE","msg":"ip_src_addr\tcount\n192.168.138.158\t111\n192.168.138.2\t1\n192.168.66.1\t89\n192.168.66.121\t69","comment":"","msgTable":[[{"key":"count","value":"192.168.138.158"},{"key":"count","value":"111"}],[{"value":"192.168.138.2"},{"value":"1"}],[{"value":"192.168.66.1"},{"value":"89"}],[{"val
ue":"192.168.66.121"},{"value":"69"}]],"columnNames":[{"name":"ip_src_addr","index":0,"aggr":"sum"},{"name":"count","index":1,"aggr":"sum"}],"rows":[["192.168.138.158","111"],["192.168.138.2","1"],["192.168.66.1","89"],["192.168.66.121","69"]]}},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1494339578537_-194906756","id":"20170509-141938_486503393","dateCreated":"2017-05-09T14:19:38+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:7794","text":"%md\n\n### Connection Volume (Destination) - Yaf\n\nThe volume of connections made to destination IPs.\n\nThis IPs retrieved are given by an IPv4 CIDR block.","dateUpdated":"2017-05-09T14:22:06+0000","dateFinished":"2017-05-09T14:22:06+0000","dateStarted":"2017-05-09T14:22:06+0000","result":{"code":"SUCCESS","type":"HTM
L","msg":"<h3>Connection Volume (Destination) - Yaf</h3>\n<p>The volume of connections made to destination IPs.</p>\n<p>This IPs retrieved are given by an IPv4 CIDR block.</p>\n"}},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{"CIDR":"192.0.0.0/8"},"forms":{"CIDR":{"name":"CIDR","displayName":"CIDR","type":"input","defaultValue":"","hidden":false}}},"jobName":"paragraph_1494339202329_1284921236","id":"20170509-141322_1098639923","dateCreated":"2017-05-09T14:13:22+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:7474","text":"%spark\nimport org.apache.spark.sql.Row\nimport scala.concurrent.duration._\nimport java.util.concurrent.TimeUnit\nimport org.apache.commons.net.util.SubnetUtils\n\nval cidr = z.input(\"CIDR\").toString\n@transient val utils = new SubnetUtils(cidr)\nutils.setInclusiveHostCo
unt(true)\n@transient val info = utils.getInfo\n\nval ipToLong = (ipAddress: String) => {\n val ipAddressInArray = ipAddress.split(\"\\\\.\")\n var result = 0L\n for (i <- 0 to ipAddressInArray.length-1) {\n\t val power = 3 - i\n\t val ip = Integer.parseInt(ipAddressInArray(i))\n\t result += ip * Math.pow(256.0, power.toDouble).toLong\n }\n result\n}\n\nval bcLow = sc.broadcast(ipToLong(info.getLowAddress))\nval bcHigh = sc.broadcast(ipToLong(info.getHighAddress))\nval bcIpToLong = sc.broadcast(ipToLong)\n\nval results = sqlContext.sql(\ns\"\"\"SELECT\n ip_dst_addr,\n COUNT(*) AS count\nFROM\n yaf\nGROUP BY ip_dst_addr\nORDER BY ip_dst_addr\n\"\"\").flatMap {\n case Row(ip_dst_addr: String, count: Long) => {\n val longDst = bcIpToLong.value(ip_dst_addr)\n if(bcLow.value <= longDst && longDst <= bcHigh.value) {\n\t\t List(ip_dst_addr + \"\\t\" + count)\n } else {\n List.empty[String]\n }\n }\n }.collect()\n
\nprint(\"%table ip_dst_addr\\tcount\\n\" + results.mkString(\"\\n\"))\n","dateUpdated":"2017-05-09T14:20:10+0000","dateFinished":"2017-05-09T14:20:31+0000","dateStarted":"2017-05-09T14:20:11+0000","result":{"code":"SUCCESS","type":"TABLE","msg":"ip_dst_addr\tcount\n192.168.138.158\t151\n192.168.138.2\t2\n192.168.66.1\t69\n192.168.66.121\t86","comment":"","msgTable":[[{"key":"count","value":"192.168.138.158"},{"key":"count","value":"151"}],[{"value":"192.168.138.2"},{"value":"2"}],[{"value":"192.168.66.1"},{"value":"69"}],[{"value":"192.168.66.121"},{"value":"86"}]],"columnNames":[{"name":"ip_dst_addr","index":0,"aggr":"sum"},{"name":"count","index":1,"aggr":"sum"}],"rows":[["192.168.138.158","151"],["192.168.138.2","2"],["192.168.66.1","69"],["192.168.66.121","86"]]}},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1494339303735_1
424887757","id":"20170509-141503_898772342","dateCreated":"2017-05-09T14:15:03+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:7557","dateUpdated":"2017-05-09T14:20:10+0000","dateFinished":"2017-05-09T14:20:31+0000","dateStarted":"2017-05-09T14:20:21+0000","result":{"code":"SUCCESS","type":"TEXT","msg":""}}],"name":"Metron - Connection Volume Report","id":"2CER9F199","angularObjects":{"2CET9UGAN:shared_process":[],"2CHC4B4TT:shared_process":[],"2CF24S5PD:shared_process":[],"2CF6W9QPU:shared_process":[],"2CJJ4RS82:shared_process":[],"2CGYCAYBG:shared_process":[]},"config":{"looknfeel":"simple"},"info":{}}
\ No newline at end of file
[41/44] metron git commit: METRON-819: Document kafka console
producer parameter for sensors with kerberos (mmiklavc) closes
apache/metron#507
Posted by ma...@apache.org.
METRON-819: Document kafka console producer parameter for sensors with kerberos (mmiklavc) closes apache/metron#507
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/9dc8c3a7
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/9dc8c3a7
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/9dc8c3a7
Branch: refs/heads/Metron_0.4.0
Commit: 9dc8c3a707bbdd4ac2070d5e29a558abde63f360
Parents: f127c6d
Author: mmiklavc <mi...@gmail.com>
Authored: Wed May 31 15:31:51 2017 -0600
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Wed May 31 15:31:51 2017 -0600
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 30 ++++++++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/9dc8c3a7/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index 76d3701..1540231 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -17,7 +17,7 @@ This document provides instructions for kerberizing Metron's Vagrant-based devel
Setup
-----
-1. Deploy a Vagrant development environment; either [Full Dev](full-dev-platform) or [Quick Dev](quick-dev-platform).
+1. Deploy a Vagrant development environment; either [Full Dev](vagrant/full-dev-platform/README.md) or [Quick Dev](vagrant/quick-dev-platform/README.md).
1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant.
@@ -424,6 +424,34 @@ KVNO Timestamp Principal
### Kafka with Kerberos enabled
+#### Running Sensors
+
+A couple steps are required to produce data to a Kerberized Kafka topic. On the host you'll be setting up your sensor(s), switch to the metron user and create a client_jaas.conf file in the metron home directory if one doesn't already exist. It should be owned by metron:metron and
+contain at least the following stanza that tells the Kafka client how to interact with Kerberos:
+```
+su - metron
+cat ${METRON_HOME}/client_jaas.conf
+...
+KafkaClient {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ keyTab="/etc/security/keytabs/metron.headless.keytab"
+ storeKey=true
+ useTicketCache=false
+ serviceName="kafka"
+ principal="metron@EXAMPLE.COM";
+};
+```
+
+You'll also need to set KAFKA_OPTS to tell the Kafka client how to interact with Kerberos.
+```
+export KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf"
+```
+
+For sensors that leverage the Kafka console producer to pipe data into Metron, e.g. Snort and Yaf, you will need to modify the corresponding sensor shell scripts or config to append the SASL security protocol property. `--security-protocol SASL_PLAINTEXT`. Be sure to kinit with the metron user's keytab before executing the script that starts the sensor.
+
+More notes can be found in [metron/metron-sensors/README.md](../metron-sensors/README.md)
+
#### Write data to a topic with SASL
```
[44/44] metron git commit: METRON-858 bro-plugin-kafka is throwing
segfaults (JonZeolla) closes apache/metron#547
Posted by ma...@apache.org.
METRON-858 bro-plugin-kafka is throwing segfaults (JonZeolla) closes apache/metron#547
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/85872bd6
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/85872bd6
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/85872bd6
Branch: refs/heads/Metron_0.4.0
Commit: 85872bd68698149692c97a48dfe41a78435dcc99
Parents: 8779eb3
Author: JonZeolla <ze...@gmail.com>
Authored: Thu Jun 1 11:28:42 2017 -0400
Committer: jonzeolla <jo...@apache.org>
Committed: Thu Jun 1 11:28:42 2017 -0400
----------------------------------------------------------------------
metron-sensors/bro-plugin-kafka/README.md | 56 ++++++++++++++++++--
.../bro-plugin-kafka/configure.plugin | 2 +-
.../bro-plugin-kafka/src/KafkaWriter.cc | 21 ++++----
3 files changed, 63 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/85872bd6/metron-sensors/bro-plugin-kafka/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/README.md b/metron-sensors/bro-plugin-kafka/README.md
index 31b1f54..e219360 100644
--- a/metron-sensors/bro-plugin-kafka/README.md
+++ b/metron-sensors/bro-plugin-kafka/README.md
@@ -36,13 +36,14 @@ Installation
Activation
----------
-The following examples highlight different ways that the plugin can be used. Simply add Bro script to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to activate the plugin.
+The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to demonstrate the example.
### Example 1
The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`.
* Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration table.
* By defining `topic_name` all records will be sent to the same Kafka topic.
+ * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent.
```
@load Bro/Kafka/logs-to-kafka.bro
@@ -73,7 +74,6 @@ event bro_init()
$name = "kafka-http",
$writer = Log::WRITER_KAFKAWRITER,
$config = table(
- ["stream_id"] = "HTTP::LOG",
["metadata.broker.list"] = "localhost:9092"
),
$path = "http"
@@ -85,7 +85,6 @@ event bro_init()
$name = "kafka-dns",
$writer = Log::WRITER_KAFKAWRITER,
$config = table(
- ["stream_id"] = "DNS::LOG",
["metadata.broker.list"] = "localhost:9092"
),
$path = "dns"
@@ -94,6 +93,57 @@ event bro_init()
}
```
+### Example 3
+
+You may want to configure bro to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Metron currently doesn't support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages from being sent to kafka (although there are [multiple ways](#notes) to approach this). In this example we will do that that, and are assuming a somewhat standard bro kafka plugin configuration, such that:
+ * All bro logs are sent to the `bro` topic, by configuring `Kafka::topic_name`.
+ * Each JSON message is tagged with the appropriate log type (such as `http`, `dns`, or `conn`), by setting `tag_json` to true.
+ * If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka.
+
+```
+@load Bro/Kafka/logs-to-kafka.bro
+redef Kafka::topic_name = "bro";
+redef Kafka::tag_json = T;
+
+event bro_init() &priority=-5
+{
+ # handles HTTP
+ Log::add_filter(HTTP::LOG, [
+ $name = "kafka-http",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: HTTP::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ )
+ ]);
+
+ # handles DNS
+ Log::add_filter(DNS::LOG, [
+ $name = "kafka-dns",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: DNS::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ )
+ ]);
+
+ # handles Conn
+ Log::add_filter(Conn::LOG, [
+ $name = "kafka-conn",
+ $writer = Log::WRITER_KAFKAWRITER,
+ $pred(rec: Conn::Info) = { return ! (( |rec$id$orig_h| == 128 || |rec$id$resp_h| == 128 )); },
+ $config = table(
+ ["metadata.broker.list"] = "localhost:9092"
+ )
+ ]);
+}
+```
+
+#### Notes
+ * `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`.
+ * You can also filter IPv6 logs from within your Metron cluster [using Stellar](../../metron-platform/metron-common#IS_IP). In that case, you wouldn't apply a predicate in your bro configuration, and instead Stellar would filter the logs out before they were processed by the enrichment layer of Metron.
+ * It is also possible to use the `is_v6_subnet()` bro function in your predicate, as of their [2.5 release](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-5), however the above example should work on [bro 2.4](https://www.bro.org/sphinx-git/install/release-notes.html#bro-2-4) and newer, which has been the focus of the kafka plugin.
+
Settings
--------
http://git-wip-us.apache.org/repos/asf/metron/blob/85872bd6/metron-sensors/bro-plugin-kafka/configure.plugin
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/configure.plugin b/metron-sensors/bro-plugin-kafka/configure.plugin
index 1cb2086..c7e6662 100644
--- a/metron-sensors/bro-plugin-kafka/configure.plugin
+++ b/metron-sensors/bro-plugin-kafka/configure.plugin
@@ -31,7 +31,7 @@ plugin_option()
{
case "$1" in
--with-librdkafka=*)
- append_cache_entry LibRdKafka_ROOT_DIR PATH $optarg
+ append_cache_entry LibRDKafka_ROOT_DIR PATH $optarg
;;
--with-openssl=*)
append_cache_entry OpenSSL_ROOT_DIR PATH $optarg
http://git-wip-us.apache.org/repos/asf/metron/blob/85872bd6/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
----------------------------------------------------------------------
diff --git a/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc b/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
index 951a60c..c9ad44f 100644
--- a/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
+++ b/metron-sensors/bro-plugin-kafka/src/KafkaWriter.cc
@@ -75,13 +75,10 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
debug.assign((const char*)BifConst::Kafka::debug->Bytes(), BifConst::Kafka::debug->Len());
bool is_debug(!debug.empty());
if(is_debug) {
- reporter->Info( "Debug is turned on and set to: %s. Available debug context: %s."
- , debug.c_str()
- , RdKafka::get_debug_contexts().c_str()
- );
+ MsgThread::Info(Fmt("Debug is turned on and set to: %s. Available debug context: %s.", debug.c_str(), RdKafka::get_debug_contexts().c_str()));
}
else {
- reporter->Info( "Debug is turned off.");
+ MsgThread::Info(Fmt("Debug is turned off."));
}
// kafka global configuration
@@ -96,7 +93,7 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
// apply setting to kafka
if (RdKafka::Conf::CONF_OK != conf->set(key, val, err)) {
- reporter->Error("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str());
+ Error(Fmt("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str()));
return false;
}
}
@@ -105,7 +102,7 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
string key("debug");
string val(debug);
if (RdKafka::Conf::CONF_OK != conf->set(key, val, err)) {
- reporter->Error("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str());
+ Error(Fmt("Failed to set '%s'='%s': %s", key.c_str(), val.c_str(), err.c_str()));
return false;
}
}
@@ -113,7 +110,7 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
// create kafka producer
producer = RdKafka::Producer::create(conf, err);
if (!producer) {
- reporter->Error("Failed to create producer: %s", err.c_str());
+ Error(Fmt("Failed to create producer: %s", err.c_str()));
return false;
}
@@ -121,12 +118,12 @@ bool KafkaWriter::DoInit(const WriterInfo& info, int num_fields, const threading
topic_conf = RdKafka::Conf::create(RdKafka::Conf::CONF_TOPIC);
topic = RdKafka::Topic::create(producer, topic_name, topic_conf, err);
if (!topic) {
- reporter->Error("Failed to create topic handle: %s", err.c_str());
+ Error(Fmt("Failed to create topic handle: %s", err.c_str()));
return false;
}
if(is_debug) {
- reporter->Info("Successfully created producer.");
+ MsgThread::Info(Fmt("Successfully created producer."));
}
return true;
@@ -154,7 +151,7 @@ bool KafkaWriter::DoFinish(double network_time)
if (producer->outq_len() == 0) {
success = true;
} else {
- reporter->Error("Unable to deliver %0d message(s)", producer->outq_len());
+ Error(Fmt("Unable to deliver %0d message(s)", producer->outq_len()));
}
delete topic;
@@ -187,7 +184,7 @@ bool KafkaWriter::DoWrite(int num_fields, const threading::Field* const* fields,
}
else {
string err = RdKafka::err2str(resp);
- reporter->Error("Kafka send failed: %s", err.c_str());
+ Error(Fmt("Kafka send failed: %s", err.c_str()));
}
return true;
[38/44] metron git commit: METRON-949 Change from error to warning
message for Metron REST in Ambari service_advisor (anandsubbu via leet)
closes apache/metron#583
Posted by ma...@apache.org.
METRON-949 Change from error to warning message for Metron REST in Ambari service_advisor (anandsubbu via leet) closes apache/metron#583
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/1a0e8f25
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/1a0e8f25
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/1a0e8f25
Branch: refs/heads/Metron_0.4.0
Commit: 1a0e8f252a7a59fc6328b6fa008bc1f45387ac5e
Parents: ae05f38
Author: anandsubbu <an...@gmail.com>
Authored: Tue May 30 10:15:14 2017 -0400
Committer: YOUR NAME as In Apache <YO...@apache.org>
Committed: Tue May 30 10:15:14 2017 -0400
----------------------------------------------------------------------
.../resources/common-services/METRON/CURRENT/service_advisor.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/1a0e8f25/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index 464857a..7da06f5 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -66,7 +66,7 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
if metronRESTHost not in stormSupervisors:
message = "Metron REST must be colocated with an instance of STORM SUPERVISOR"
- items.append({ "type": 'host-component', "level": 'ERROR', "message": message, "component-name": 'METRON_REST', "host": metronRESTHost })
+ items.append({ "type": 'host-component', "level": 'WARN', "message": message, "component-name": 'METRON_REST', "host": metronRESTHost })
if metronParsersHost != metronEnrichmentMaster:
message = "Metron Enrichment Master must be co-located with Metron Parsers on {0}".format(metronParsersHost)
[02/44] metron git commit: METRON-836 Use Pycapa with Kerberos
(nickwallen) closes apache/incubator-metron#524
Posted by ma...@apache.org.
METRON-836 Use Pycapa with Kerberos (nickwallen) closes apache/incubator-metron#524
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/47e5aa70
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/47e5aa70
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/47e5aa70
Branch: refs/heads/Metron_0.4.0
Commit: 47e5aa70adfd2fb000e412c3a6b6d06c91e2809c
Parents: f36db22
Author: nickwallen <ni...@nickallen.org>
Authored: Fri Apr 28 16:23:29 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Fri Apr 28 16:23:29 2017 -0400
----------------------------------------------------------------------
metron-deployment/roles/pycapa/meta/main.yml | 1 +
metron-sensors/README.md | 9 +-
metron-sensors/pycapa/README.md | 286 ++++++++++++++++++----
metron-sensors/pycapa/pycapa/common.py | 4 +
metron-sensors/pycapa/pycapa/consumer.py | 112 ++++++---
metron-sensors/pycapa/pycapa/producer.py | 97 ++++++--
metron-sensors/pycapa/pycapa/pycapa_cli.py | 113 +++++++--
metron-sensors/pycapa/requirements.txt | 2 +-
8 files changed, 497 insertions(+), 127 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-deployment/roles/pycapa/meta/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/pycapa/meta/main.yml b/metron-deployment/roles/pycapa/meta/main.yml
index c3d807b..a5b54b7 100644
--- a/metron-deployment/roles/pycapa/meta/main.yml
+++ b/metron-deployment/roles/pycapa/meta/main.yml
@@ -17,3 +17,4 @@
---
dependencies:
- ambari_gather_facts
+ - librdkafka
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/README.md b/metron-sensors/README.md
index af932e5..e458630 100644
--- a/metron-sensors/README.md
+++ b/metron-sensors/README.md
@@ -1,5 +1,8 @@
-# Metron Sensors
+Metron Sensors
+--------------
-- Fast CAPA
-- Py CAPA
+ * [`bro-plugin-kafka`](bro-plugin-kafka/): Provides integration between [Bro](https://www.bro.org/) and Kafka. A Bro plugin that sends logging output to Kafka. This provides a convenient means for tools in the Hadoop ecosystem, such as Storm, Spark, and others to process the data generated by Bro.
+ * [`fastcapa`](fastcapa/): Performs fast network packet capture by leveraging Linux kernel-bypass and user space networking technology. The probe will bind to a network interface, capture network packets, and send the raw packet data to Kafka. This provides a scalable mechanism for ingesting high-volumes of network packet data.
+
+ * [`pycapa`](pycapa/): Performs lightweight network packet capture, retrieves network packets from Kafka, generates `libpcap`-compliant files, and enables integration with third-party tools like Wireshark.
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/pycapa/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md
index 33e9d0a..1e8c2c5 100644
--- a/metron-sensors/pycapa/README.md
+++ b/metron-sensors/pycapa/README.md
@@ -1,74 +1,272 @@
-# Pycapa
+Pycapa
+------
-## Overview
+* [Overview](#overview)
+* [Installation](#installation)
+* [Usage](#usage)
+ * [Parameters](#parameters)
+ * [Examples](#examples)
+ * [Kerberos](#kerberos)
+* [FAQs](#faqs)
-Pycapa performs network packet capture, both off-the-wire and from Kafka, which is useful for the testing and development of [Apache Metron](https://github.com/apache/incubator-metron). It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic.
+Overview
+========
-## Installation
+Pycapa performs network packet capture, both off-the-wire and from a Kafka topic, which is useful for the testing and development of [Apache Metron](https://github.com/apache/incubator-metron). It is not intended for production use. The tool will capture packets from a specified interface and push them into a Kafka Topic. The tool can also do the reverse. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat) or even to feed directly into a tool like Wireshark to monitor ongoing activity.
-```
-pip install -r requirements.txt
-python setup.py install
-```
+Installation
+============
+
+General notes on the installation of Pycapa.
+* Python 2.7 is required.
+* The following package dependencies are required and can be installed automatically with `pip`.
+ * [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python)
+ * [pcapy](https://github.com/CoreSecurity/pcapy)
+* These instructions can be used directly on CentOS 7+.
+* Other Linux distributions that come with Python 2.7 can use these instructions with some minor modifications.
+* Older distributions, like CentOS 6, that come with Python 2.6 installed, should install Python 2.7 within a virtual environment and then run Pycapa from within the virtual environment.
+
+
+1. Install system dependencies including the core development tools, Python libraries and header files, and Libpcap libraries and header files. On CentOS 7+, you can install these requirements with the following command.
+
+ ```
+ yum -y install "@Development tools" python-devel libpcap-devel
+ ```
+
+1. Install Librdkafka at your chosen $PREFIX.
+
+ ```
+ export PREFIX=/usr
+
+ wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+ cd librdkafka-0.9.4/
+ ./configure --prefix=$PREFIX
+ make
+ make install
+ ```
+
+1. Add Librdkafka to the dynamic library load path.
+
+ ```
+ echo "$PREFIX/lib" >> /etc/ld.so.conf.d/pycapa.conf
+ ldconfig -v
+ ```
+
+1. Install Pycapa. This assumes that you already have the Metron source code on the host.
+
+ ```
+ cd incubator-metron/metron-sensors/pycapa
+ pip install -r requirements.txt
+ python setup.py install
+ ```
+
+Usage
+=====
+
+Pycapa has two primary runtime modes.
-## Usage
+* **Producer Mode**: Pycapa can capture packets from a network interface and forward those packets to a Kafka topic. Pycapa embeds the raw network packet data in the Kafka message body. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch, in network byte order.
+
+* **Consumer Mode**: Pycapa can also perform the reverse operation. It can consume packets from Kafka and reconstruct each network packet. This can then be used to create a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat) or even to feed directly into a tool like Wireshark to monitor activity.
+
+### Parameters
```
-$ pycapa --help
-usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t TOPIC] [-n PACKET_COUNT]
- [-d DEBUG] [-i INTERFACE]
+$ pycapa -h
+usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t KAFKA_TOPIC]
+ [-i NETWORK_IFACE] [-m MAX_PACKETS] [-pp PRETTY_PRINT]
+ [-ll LOG_LEVEL] [-X KAFKA_CONFIGS] [-s SNAPLEN]
optional arguments:
-h, --help show this help message and exit
-p, --producer sniff packets and send to kafka
-c, --consumer read packets from kafka
- -k KAFKA_BROKERS, --kafka KAFKA_BROKERS
+ -k KAFKA_BROKERS, --kafka-broker KAFKA_BROKERS
kafka broker(s)
- -t TOPIC, --topic TOPIC
+ -t KAFKA_TOPIC, --kafka-topic KAFKA_TOPIC
kafka topic
- -n PACKET_COUNT, --number PACKET_COUNT
- number of packets to consume
- -d DEBUG, --debug DEBUG
- debug every X packets
- -i INTERFACE, --interface INTERFACE
- interface to listen on
+ -i NETWORK_IFACE, --interface NETWORK_IFACE
+ network interface to listen on
+ -m MAX_PACKETS, --max-packets MAX_PACKETS
+ stop after this number of packets
+ -pp PRETTY_PRINT, --pretty-print PRETTY_PRINT
+ pretty print every X packets
+ -ll LOG_LEVEL, --log-level LOG_LEVEL
+ set the log level
+ -X KAFKA_CONFIGS define a kafka client parameter; key=value
+ -s SNAPLEN, --snaplen SNAPLEN
+ snapshot length
```
-Pycapa has two primary runtime modes.
+### Examples
-### Producer Mode
+**Example**: Capture 10 packets from the `eth0` network interface and forward those to a Kafka topic called `pcap` running on `localhost:9092`.
+ ```
+ $ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10
+ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'AWBHMIAESAHJ'}
+ INFO:root:Starting packet capture
+ INFO:root:Waiting for '10' message(s) to flush
+ INFO:root:'10' packet(s) in, '10' packet(s) out
+ ```
-Pycapa can be configured to capture packets from a network interface and then forward those packets to a Kafka topic. The following example will capture packets from the `eth0` network interface and forward those to a Kafka topic called `pcap` running on `localhost`.
+**Example**: Capture packets until SIGINT is received. A SIGINT is the interrupt signal sent when entering CTRL-D in the console.
+ ```
+ $ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap
+ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'EULLGDOMZDCT'}
+ INFO:root:Starting packet capture
+ ^C
+ INFO:root:Clean shutdown process started
+ INFO:root:Waiting for '0' message(s) to flush
+ INFO:root:'7' packet(s) in, '7' packet(s) out
+ ```
-```
-pycapa --producer --kafka localhost:9092 --topic pcap -i eth0
-```
+**Example**: While capturing packets, output diagnostic information every 10 packets.
+ ```
+ $ pycapa --producer \
+ --interface en0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --pretty-print 10
+ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'YMDSEEDIHVWD'}
+ INFO:root:Starting packet capture
+ 10 packet(s) received
+ ac bc 32 bf 0d 43 b8 3e 59 8b 8a 8a 08 00 45 00
+ 00 3c 00 00 40 00 40 06 b9 66 c0 a8 00 02 c0 a8
+ 00 03 1f 7c d7 14 5f 8b 82 b4 a8 c5 f6 63 a0 12
+ 38 90 59 cc 00 00 02 04 05 b4 04 02 08 0a 00 51
+ 44 17 39 43 3e 9b 01 03 03 04
+ 20 packet(s) received
+ 01 00 5e 00 00 fb ac bc 32 bf 0d 43 08 00 45 00
+ 00 44 d2 09 00 00 ff 11 47 f8 c0 a8 00 03 e0 00
+ 00 fb 14 e9 14 e9 00 30 69 fc 00 00 00 00 00 01
+ 00 00 00 00 00 00 0b 5f 67 6f 6f 67 6c 65 63 61
+ 73 74 04 5f 74 63 70 05 6c 6f 63 61 6c 00 00 0c
+ 80 01
+ ^C
+ INFO:root:Clean shutdown process started
+ INFO:root:Waiting for '2' message(s) to flush
+ INFO:root:'20' packet(s) in, '20' packet(s) out
+ ```
-To output debug messages every 100 captured packets, run the following.
+**Example**: Consume 10 packets from the Kafka topic `pcap` running on `localhost:9092`, then pipe those into Wireshark for DPI.
+ ```
+ $ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ | tshark -i -
+ Capturing on 'Standard input'
+ 1 0.000000 ArrisGro_0e:65:df → Apple_bf:0d:43 ARP 56 Who has 192.168.0.3? Tell 192.168.0.1
+ 2 0.000044 Apple_bf:0d:43 → ArrisGro_0e:65:df ARP 42 192.168.0.3 is at ac:bc:32:bf:0d:43
+ 3 0.203495 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 4 2.031988 192.168.0.3 → 96.27.183.249 TCP 54 55110 → 443 [ACK] Seq=1 Ack=1 Win=4108 Len=0
+ 5 2.035816 192.30.253.125 → 192.168.0.3 TLSv1.2 97 Application Data
+ 6 2.035892 192.168.0.3 → 192.30.253.125 TCP 66 54671 → 443 [ACK] Seq=1 Ack=32 Win=4095 Len=0 TSval=961120495 TSecr=2658503052
+ 7 2.035994 192.168.0.3 → 192.30.253.125 TLSv1.2 101 Application Data
+ 8 2.053866 96.27.183.249 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 55110 [ACK] Seq=1 Ack=2 Win=243 Len=0 TSval=728145145 TSecr=961030381
+ 9 2.083872 192.30.253.125 → 192.168.0.3 TCP 66 443 → 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495
+ 10 3.173189 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 10 packets captured
+ ```
-```
-pycapa --producer --kafka localhost:9092 --topic pcap -i eth0 --debug 100
-```
+**Example**: Consume 10 packets and create a libpcap-compliant pcap file.
+ ```
+ $ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ > out.pcap
+ $ tshark -r out.pcap
+ 1 0.000000 199.193.204.147 → 192.168.0.3 TLSv1.2 151 Application Data
+ 2 0.000005 199.193.204.147 → 192.168.0.3 TLSv1.2 1191 Application Data
+ 3 0.000088 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=86 Win=4093 Len=0 TSval=961284465 TSecr=943744612
+ 4 0.000089 192.168.0.3 → 199.193.204.147 TCP 66 54788 → 443 [ACK] Seq=1 Ack=1211 Win=4058 Len=0 TSval=961284465 TSecr=943744612
+ 5 0.948788 192.168.0.3 → 192.30.253.125 TCP 54 54671 → 443 [ACK] Seq=1 Ack=1 Win=4096 Len=0
+ 6 1.005175 192.30.253.125 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 54671 [ACK] Seq=1 Ack=2 Win=31 Len=0 TSval=2658544467 TSecr=961240339
+ 7 1.636312 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 8 2.253052 192.175.27.112 → 192.168.0.3 TLSv1.2 928 Application Data
+ 9 2.253140 192.168.0.3 → 192.175.27.112 TCP 66 55078 → 443 [ACK] Seq=1 Ack=863 Win=4069 Len=0 TSval=961286699 TSecr=967172238
+ 10 2.494769 192.168.0.3 → 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, "QM" question
+ ```
-### Consumer Mode
+### Kerberos
-Pycapa can be configured to consume packets from a Kafka topic and then write those packets to a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat). To read 100 packets from a kafka topic and then write those to a [libpcap-compliant file](https://wiki.wireshark.org/Development/LibpcapFileFormat), run the following command. The file `out.pcap` can then be opened with a tool such as Wireshark for further validation.
+The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.
-```
-pycapa --consumer --kafka localhost:9092 --topic pcap --n 100 > out.pcap
-```
+ * The Kafka broker is at `kafka1:6667`
+ * Zookeeper is at `zookeeper1:2181`
+ * The Kafka security protocol is `SASL_PLAINTEXT`
+ * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab`
+ * The service principal is `metron@EXAMPLE.COM`
-To consume packets from Kafka continuously and print debug messages every 10 packets, run the following command.
+1. Build Librdkafka with SASL support (` --enable-sasl`) and install at your chosen $PREFIX.
+ ```
+ wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
+ cd librdkafka-0.9.4/
+ ./configure --prefix=$PREFIX --enable-sasl
+ make
+ make install
+ ```
-```
-pycapa --consumer --kafka localhost:9092 --topic pcap --debug 10
-```
+1. Validate Librdkafka does indeed support SASL. Run the following command and ensure that `sasl` is returned as a built-in feature.
+ ```
+ $ examples/rdkafka_example -X builtin.features
+ builtin.features = gzip,snappy,ssl,sasl,regex
+ ```
+
+ If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command.
+ ```
+ yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi
+ ```
-## Dependencies
+1. Grant access to your Kafka topic. In this example the topic is simply named `pcap`.
+ ```
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add \
+ --allow-principal User:metron \
+ --topic pcap
-* [kafka-python](https://github.com/dpkp/kafka-python)
-* [pcapy](https://github.com/CoreSecurity/pcapy)
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
+ --authorizer kafka.security.auth.SimpleAclAuthorizer \
+ --authorizer-properties zookeeper.connect=zookeeper1:2181 \
+ --add \
+ --allow-principal User:metron \
+ --group pycapa
+ ```
-## Implementation
+1. Use Pycapa as you normally would, but append the following three additional parameters
+ * `security.protocol`
+ * `sasl.kerberos.keytab`
+ * `sasl.kerberos.principal`
+ ```
+ $ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker kafka1:6667 \
+ --kafka-topic pcap --max-packets 10 \
+ -X security.protocol=SASL_PLAINTEXT \
+ -X sasl.kerberos.keytab=/etc/security/keytabs/metron.headless.keytab \
+ -X sasl.kerberos.principal=metron-metron@METRONEXAMPLE.COM
+ INFO:root:Connecting to Kafka; {'sasl.kerberos.principal': 'metron-metron@METRONEXAMPLE.COM', 'group.id': 'ORNLVWJZZUAA', 'security.protocol': 'SASL_PLAINTEXT', 'sasl.kerberos.keytab': '/etc/security/keytabs/metron.headless.keytab', 'bootstrap.servers': 'kafka1:6667'}
+ INFO:root:Starting packet capture
+ INFO:root:Waiting for '1' message(s) to flush
+ INFO:root:'10' packet(s) in, '10' packet(s) out
+ ```
-When run in Producer Mode, Pycapa embeds the raw network packet data in the Kafka message. The message key contains the timestamp indicating when the packet was captured in microseconds from the epoch. This value is in network byte order.
+FAQs
+====
+
+**Question**: How do I get more logs?
+
+Use the following two command-line arguments to get detailed logging.
+```
+-X debug=all --log-level DEBUG
+```
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/pycapa/pycapa/common.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/common.py b/metron-sensors/pycapa/pycapa/common.py
index 3fb78fd..535f756 100644
--- a/metron-sensors/pycapa/pycapa/common.py
+++ b/metron-sensors/pycapa/pycapa/common.py
@@ -19,18 +19,22 @@ import struct
def to_hex(s):
+ """ Transforms a string to hexadecimal notation. """
hex_str = ' '.join("{0:02x}".format(ord(c)) for c in s)
return '\n'.join([hex_str[i:i+48] for i in range(0, len(hex_str), 48)])
def to_date(epoch_micros):
+ """ Transforms a timestamp in epoch microseconds to a more legible format. """
epoch_secs = epoch_micros / 1000000.0
return datetime.fromtimestamp(epoch_secs).strftime('%Y-%m-%d %H:%M:%S.%f')
def pack_ts(ts):
+ """ Packs a timestamp into a binary form. """
return struct.pack(">Q", ts)
def unpack_ts(packed_ts):
+ """ Unpacks a timestamp from a binary form. """
return struct.unpack_from(">Q", bytes(packed_ts), 0)[0]
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/pycapa/pycapa/consumer.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/consumer.py b/metron-sensors/pycapa/pycapa/consumer.py
index 950dc83..7029f25 100644
--- a/metron-sensors/pycapa/pycapa/consumer.py
+++ b/metron-sensors/pycapa/pycapa/consumer.py
@@ -15,50 +15,98 @@
# limitations under the License.
#
import sys
+import threading
+import signal
+import pcapy
+import argparse
+import random
+import logging
import time
-import kafka
import struct
+from confluent_kafka import Consumer, KafkaException, KafkaError
from common import to_date, to_hex, unpack_ts
-def global_header(magic=0xa1b2c3d4L, version_major=2, version_minor=4, zone=0,
- sigfigs=0, snaplen=65535, network=1):
+finished = threading.Event()
+
+
+def signal_handler(signum, frame):
+ """ Initiates a clean shutdown for a SIGINT """
+
+ finished.set()
+ logging.debug("Clean shutdown process started")
+
+
+def global_header(args, magic=0xa1b2c3d4L, version_major=2, version_minor=4, zone=0,
+ sigfigs=0, network=1):
+ """ Returns the global header used in libpcap-compliant file. """
+
return struct.pack("IHHIIII", magic, version_major, version_minor, zone,
- sigfigs, snaplen, network)
+ sigfigs, args.snaplen, network)
+
+def packet_header(msg):
+ """ Returns the packet header used in a libpcap-compliant file. """
-def packet_header(pkt_raw, msg_key):
- epoch_micros = struct.unpack_from(">Q", bytes(msg_key), 0)[0]
+ epoch_micros = struct.unpack_from(">Q", bytes(msg.key()), 0)[0]
secs = epoch_micros / 1000000
usec = epoch_micros % 1000000
- caplen = wirelen = len(pkt_raw)
+ caplen = wirelen = len(msg.value())
hdr = struct.pack('IIII', secs, usec, caplen, wirelen)
return hdr
-def consumer(args):
+def consumer(args, poll_timeout=3.0):
+ """ Consumes packets from a Kafka topic. """
+
+ # setup the signal handler
+ signal.signal(signal.SIGINT, signal_handler)
+
# connect to kafka
- brokers = args.kafka_brokers.split(",")
- kafka_consumer = kafka.KafkaConsumer(args.topic, bootstrap_servers=brokers)
-
- # if debug not set, write libpcap global header
- if args.debug == 0:
- sys.stdout.write(global_header())
-
- # start packet capture
- packet_count = 0
- for msg in kafka_consumer:
-
- # if debug not set, write the packet header and packet
- if args.debug == 0:
- sys.stdout.write(packet_header(msg.value, msg.key))
- sys.stdout.write(msg.value)
-
- elif packet_count % args.debug == 0:
- print 'Packet: count=%s dt=%s topic=%s' % (
- packet_count, to_date(unpack_ts(msg.key)), args.topic)
- print to_hex(msg.value)
-
- packet_count += 1
- if args.packet_count > 0 and packet_count >= args.packet_count:
- break
+ logging.debug("Connecting to Kafka; %s", args.kafka_configs)
+ kafka_consumer = Consumer(args.kafka_configs)
+ kafka_consumer.subscribe([args.kafka_topic])
+
+ # if 'pretty-print' not set, write libpcap global header
+ if args.pretty_print == 0:
+ sys.stdout.write(global_header(args))
+ sys.stdout.flush()
+
+ try:
+ pkts_in = 0
+ while not finished.is_set() and (args.max_packets <= 0 or pkts_in < args.max_packets):
+
+ # consume a message from kafka
+ msg = kafka_consumer.poll(timeout=poll_timeout)
+ if msg is None:
+ # no message received
+ continue;
+
+ elif msg.error():
+
+ if msg.error().code() == KafkaError._PARTITION_EOF:
+ logging.debug("reached end of topar: topic=%s, partition=%d, offset=%s", msg.topic(), msg.partition(), msg.offset())
+ elif msg.error():
+ raise KafkaException(msg.error())
+
+ else:
+ pkts_in += 1
+ logging.debug("Packet received: pkts_in=%d", pkts_in)
+
+ if args.pretty_print == 0:
+
+ # write the packet header and packet
+ sys.stdout.write(packet_header(msg))
+ sys.stdout.write(msg.value())
+ sys.stdout.flush()
+
+ elif pkts_in % args.pretty_print == 0:
+
+ # pretty print
+ print 'Packet: count=%s date=%s topic=%s' % (
+ pkts_in, to_date(unpack_ts(msg.key())), args.kafka_topic)
+ print to_hex(msg.value())
+
+ finally:
+ sys.stdout.close()
+ kafka_consumer.close()
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/pycapa/pycapa/producer.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/producer.py b/metron-sensors/pycapa/pycapa/producer.py
index 8910b9b..7374522 100644
--- a/metron-sensors/pycapa/pycapa/producer.py
+++ b/metron-sensors/pycapa/pycapa/producer.py
@@ -14,45 +14,96 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+import sys
+import threading
+import signal
import pcapy
import argparse
-import kafka
import random
+import logging
from common import to_date, to_hex, pack_ts
+from confluent_kafka import Producer
+
+finished = threading.Event()
+
+
+def signal_handler(signum, frame):
+ """ Initiates a clean shutdown for a SIGINT """
+
+ finished.set()
+ logging.info("Clean shutdown process started")
def partitioner(key_bytes, all_parts, avail_parts):
+ """ Partitions messages randomly across all available partitions. """
+
return random.choice(avail_parts)
def timestamp(pkt_hdr):
+ """ Returns the timestamp of the packet in epoch milliseconds. """
+
(epoch_secs, delta_micros) = pkt_hdr.getts()
epoch_micros = (epoch_secs * 1000000.0) + delta_micros
return epoch_micros
-def producer(args):
+def delivery_callback(err, msg):
+ """ Callback executed when message delivery either succeeds or fails. """
+
+ # initialize counter, if needed
+ if not hasattr(delivery_callback, "pkts_out"):
+ delivery_callback.pkts_out = 0
+
+ if err:
+ logging.error("message delivery failed: error=%s", err)
+ else:
+ logging.debug("message delivery succeeded: pkts_out=%d", delivery_callback.pkts_out)
+ delivery_callback.pkts_out += 1
+
+
+def producer(args, sniff_timeout_ms=500, sniff_promisc=True):
+ """ Captures packets from a network interface and sends them to a Kafka topic. """
+
+ # setup the signal handler
+ signal.signal(signal.SIGINT, signal_handler)
+
# connect to kafka
- producer = kafka.KafkaProducer(
- bootstrap_servers=args.kafka_brokers.split(","),
- partitioner=partitioner)
+ logging.info("Connecting to Kafka; %s", args.kafka_configs)
+ kafka_producer = Producer(args.kafka_configs)
# initialize packet capture
- capture = pcapy.open_live(args.interface, 65535, True, 3000)
- packet_count = 0
-
- # start packet capture
- while True:
- (pkt_hdr, pkt_raw) = capture.next()
- if pkt_hdr is not None:
-
- # send packet to kafka
- pkt_ts = timestamp(pkt_hdr)
- producer.send(args.topic, key=pack_ts(pkt_ts), value=pkt_raw)
-
- # debug messages, if needed
- packet_count += 1
- if args.debug > 0 and packet_count % args.debug == 0:
- print 'Sent Packet: count=%s dt=%s topic=%s' % (
- packet_count, to_date(pkt_ts), args.topic)
- print to_hex(pkt_raw)
+ logging.info("Starting packet capture")
+ capture = pcapy.open_live(args.interface, args.snaplen, sniff_promisc, sniff_timeout_ms)
+ pkts_in = 0
+
+ try:
+ while not finished.is_set() and (args.max_packets <= 0 or pkts_in < args.max_packets):
+
+ # capture a packet
+ (pkt_hdr, pkt_raw) = capture.next()
+ if pkt_hdr is not None:
+ logging.debug("Packet received: pkts_in=%d, pkt_len=%s", pkts_in, pkt_hdr.getlen())
+ pkts_in += 1
+ pkt_ts = timestamp(pkt_hdr)
+ kafka_producer.produce(args.kafka_topic, key=pack_ts(pkt_ts), value=pkt_raw, callback=delivery_callback)
+
+ # debug messages, if needed
+ if args.pretty_print > 0 and pkts_in % args.pretty_print == 0:
+ print '{} packet(s) received'.format(pkts_in)
+ print to_hex(pkt_raw)
+
+ # serve the callback queue
+ kafka_producer.poll(0)
+
+ finally:
+ # flush all messages
+ logging.info("Waiting for '%d' message(s) to flush", len(kafka_producer))
+ kafka_producer.flush()
+
+ # pkts_out may not be initialized if the callback was never executed
+ pkts_out = 0
+ if hasattr(delivery_callback, "pkts_out"):
+ pkts_out = delivery_callback.pkts_out
+
+ logging.info("'%d' packet(s) in, '%d' packet(s) out", pkts_in, pkts_out)
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/pycapa/pycapa/pycapa_cli.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/pycapa_cli.py b/metron-sensors/pycapa/pycapa/pycapa_cli.py
index 33b2df6..f650280 100644
--- a/metron-sensors/pycapa/pycapa/pycapa_cli.py
+++ b/metron-sensors/pycapa/pycapa/pycapa_cli.py
@@ -15,63 +15,127 @@
# limitations under the License.
#
import argparse
+import logging
+import random
+import string
from producer import producer
from consumer import consumer
def make_parser():
+ """ Creates a command-line argument parser. """
+
parser = argparse.ArgumentParser()
- parser.add_argument('-p',
- '--producer',
+ parser.add_argument('-p', '--producer',
help='sniff packets and send to kafka',
dest='producer',
action='store_true',
default=False)
- parser.add_argument('-c',
- '--consumer',
+
+ parser.add_argument('-c', '--consumer',
help='read packets from kafka',
dest='consumer',
action='store_true',
default=False)
- parser.add_argument('-k',
- '--kafka',
+
+ parser.add_argument('-k', '--kafka-broker',
help='kafka broker(s)',
dest='kafka_brokers')
- parser.add_argument('-t',
- '--topic',
+
+ parser.add_argument('-t', '--kafka-topic',
help='kafka topic',
- dest='topic')
- parser.add_argument('-n',
- '--number',
- help='number of packets to consume',
- dest='packet_count',
- type=int)
- parser.add_argument('-d',
- '--debug',
- help='debug every X packets',
- dest='debug',
+ dest='kafka_topic')
+
+ parser.add_argument('-i', '--interface',
+ help='network interface to listen on',
+ dest='interface',
+ metavar='NETWORK_IFACE')
+
+ parser.add_argument('-m', '--max-packets',
+ help='stop after this number of packets',
+ dest='max_packets',
type=int,
default=0)
- parser.add_argument('-i',
- '--interface',
- help='interface to listen on',
- dest='interface')
+
+ parser.add_argument('-pp','--pretty-print',
+ help='pretty print every X packets',
+ dest='pretty_print',
+ type=int,
+ default=0)
+
+ parser.add_argument('-ll', '--log-level',
+ help='set the log level',
+ dest='log_level',
+ default='INFO')
+
+ parser.add_argument('-X',
+ type=keyval,
+ help='define a kafka client parameter; key=value',
+ dest='kafka_configs',
+ action='append')
+
+ parser.add_argument('-s','--snaplen',
+ help="snapshot length",
+ dest='snaplen',
+ type=int,
+ default=65535)
+
return parser
+def keyval(input, delim="="):
+ """ Expects a single key=value. """
+
+ keyval = input.split("=")
+ if(len(keyval) != 2):
+ raise ValueError("expect key=val")
+
+ return keyval
+
+
def valid_args(args):
- if args.producer and args.kafka_brokers and args.topic and args.interface:
+ """ Validates the command-line arguments. """
+
+ if args.producer and args.kafka_brokers and args.kafka_topic and args.interface:
return True
- elif args.consumer and args.kafka_brokers and args.topic:
+ elif args.consumer and args.kafka_brokers and args.kafka_topic:
return True
else:
return False
+def clean_kafka_configs(args):
+ """ Cleans and transforms the Kafka client configs. """
+
+ # transform 'kafka_configs' args from "list of lists" to dict
+ configs = {}
+ if(args.kafka_configs is not None):
+ for keyval in args.kafka_configs:
+ configs[keyval[0]] = keyval[1:][0]
+
+ # boostrap servers can be set as a "-X bootstrap.servers=KAFKA:9092" or "-k KAFKA:9092"
+ bootstrap_key = "bootstrap.servers"
+ if(bootstrap_key not in configs):
+ configs[bootstrap_key] = args.kafka_brokers;
+
+ # if no 'group.id', generate a random one
+ group_key = "group.id"
+ if(group_key not in configs):
+ configs[group_key] = ''.join(random.choice(string.ascii_uppercase) for _ in range(12))
+
+ args.kafka_configs = configs
+
def main():
parser = make_parser()
args = parser.parse_args()
+ # setup logging
+ numeric_log_level = getattr(logging, args.log_level.upper(), None)
+ if not isinstance(numeric_log_level, int):
+ raise ValueError('invalid log level: %s' % args.log_level)
+ logging.basicConfig(level=numeric_log_level)
+
+ clean_kafka_configs(args)
if not valid_args(args):
parser.print_help()
elif args.consumer:
@@ -81,5 +145,6 @@ def main():
else:
parser.print_help()
+
if __name__ == '__main__':
main()
http://git-wip-us.apache.org/repos/asf/metron/blob/47e5aa70/metron-sensors/pycapa/requirements.txt
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/requirements.txt b/metron-sensors/pycapa/requirements.txt
index b2dcb05..0119daa 100644
--- a/metron-sensors/pycapa/requirements.txt
+++ b/metron-sensors/pycapa/requirements.txt
@@ -14,6 +14,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-kafka-python
+confluent_kafka
pcapy
argparse
[20/44] metron git commit: METRON-902 ES improperly indexes Bro logs
(JonZeolla) closes apache/incubator-metron#555
Posted by ma...@apache.org.
METRON-902 ES improperly indexes Bro logs (JonZeolla) closes apache/incubator-metron#555
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/9a5e1ba7
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/9a5e1ba7
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/9a5e1ba7
Branch: refs/heads/Metron_0.4.0
Commit: 9a5e1ba73ddf1d7154fff6f41fa9b9e41fed3328
Parents: 5399009
Author: JonZeolla <ze...@gmail.com>
Authored: Fri May 12 09:27:41 2017 -0400
Committer: jonzeolla <jo...@apache.org>
Committed: Fri May 12 09:27:41 2017 -0400
----------------------------------------------------------------------
.../METRON/CURRENT/package/files/bro_index.template | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/9a5e1ba7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
index 50eae30..7521af7 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
@@ -154,11 +154,12 @@
"index": "not_analyzed"
},
"request_body_len": {
- "type": "integer"
+ "type": "long"
},
"uri": {
"type": "string",
- "index": "not_analyzed"
+ "index": "not_analyzed",
+ "ignore_above": 8191
},
"uid": {
"type": "string",
@@ -180,7 +181,7 @@
"index": "not_analyzed"
},
"response_body_len": {
- "type": "integer"
+ "type": "long"
},
"user_agent": {
"type": "string"
@@ -190,7 +191,7 @@
"index": "not_analyzed"
},
"answers": {
- "type": "ip"
+ "type": "string"
},
"AA": {
"type": "boolean"
[05/44] metron git commit: METRON-907: Zeppelin Dashboard to execute
and download pcap queries closes apache/incubator-metron#559
Posted by ma...@apache.org.
METRON-907: Zeppelin Dashboard to execute and download pcap queries closes apache/incubator-metron#559
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/1d27a32d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/1d27a32d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/1d27a32d
Branch: refs/heads/Metron_0.4.0
Commit: 1d27a32d5fe20ecb57a0cc143c1cb707f79ad20c
Parents: 494643c
Author: cstella <ce...@gmail.com>
Authored: Wed May 3 10:14:41 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Wed May 3 10:14:41 2017 -0400
----------------------------------------------------------------------
.../docker/rpm-docker/SPECS/metron.spec | 2 +
.../config/zeppelin/metron/metron-pcap.json | 1 +
.../org/apache/metron/pcap/query/CliConfig.java | 19 ++++-
.../org/apache/metron/pcap/query/CliParser.java | 4 +-
.../metron/pcap/query/FixedCliConfig.java | 3 +-
.../metron/pcap/query/FixedCliParser.java | 9 +-
.../org/apache/metron/pcap/query/PcapCli.java | 23 +++--
.../metron/pcap/query/QueryCliConfig.java | 4 +
.../metron/pcap/query/QueryCliParser.java | 9 +-
.../src/main/scripts/pcap_zeppelin_run.sh | 88 ++++++++++++++++++++
.../apache/metron/pcap/query/PcapCliTest.java | 27 +++---
11 files changed, 158 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index b1ba637..c435c6c 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -288,7 +288,9 @@ This package installs the Metron PCAP files %{metron_home}
%{metron_home}/bin/pcap_inspector.sh
%{metron_home}/bin/pcap_query.sh
%{metron_home}/bin/start_pcap_topology.sh
+%{metron_home}/bin/pcap_zeppelin_run.sh
%{metron_home}/flux/pcap/remote.yaml
+%{metron_home}/config/zeppelin/metron/metron-pcap.json
%attr(0644,root,root) %{metron_home}/lib/metron-pcap-backend-%{full_version}.jar
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
new file mode 100644
index 0000000..8c853a5
--- /dev/null
+++ b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
@@ -0,0 +1 @@
+{"paragraphs":[{"text":"%md\n# Execute Packet Capture Queries\n\nSpecify the following to filter the packet capture query:\n* *end time* - The ending time of the query in yyyyMMdd format (e.g. 20170428)\n* *start time* - The starting time of the query in yyyyMMdd format (e.g. 20170428)\n* *query* - The [Stellar](https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-language) query (i.e. a Stellar expression that returns `true` or `false`) to specify the packets.\n\nThe available fields to use in the queries are as follows:\n* `ip_src_addr` - The source IP address of the packets filtered\n* `ip_src_port` - The source port of the packets filtered\n* `ip_dst_addr` - The destination IP address of the packets filtered\n* `ip_dst_port` - The destination port of the packets filtered\n* `packet` - The raw packet (for use with the `BYTEARRAY_MATCHER` function)\n\nYou can use any [Stellar function](https://github.com/apache/incubator-metron/tree/maste
r/metron-platform/metron-common#stellar-core-functions)\n\n## Simple Boolean Expressions\n\nFor example:\n* `ip_dst_port == 8080` would return all packets where the destination port is `8080`\n* `ip_dst_port in [ 8080, 80 ]` would return all packets where the destination port is either `8080` or `80`\n\n## Common Network Functions on Metadata\n\nFor example:\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24')` would return all packets whose destination conforms to the CIDR `192.168.0.0/24`\n* `IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080` would return all packets matching the CIDR and whose destination port is `8080`\n \n## Filtering based on the Packet Contents\n\nWe use byteseek regular expressions to filter packets. The syntax for these is described [here](https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md).\n* `BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` in them anywhere.\n* `ip_ds
t_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)` would return all packets that contain the string `/api/v1` and have a destination port of `8080`\n* `BYTEARRAY_MATCHER('ff(.){5}ff', packet)` would return all packets containing a binary regex with `0xff` followed by any 5 bytes and then `0xff`\n\n# The Output\nThe output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.","dateUpdated":"2017-05-01T17:36:27+0000","config":{"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"query":"ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\\\'/api/v1/\\\\\\'', packet)"},"forms":{}},"jobName":"paragraph_1493658709556_1010155640","id":"20170428-183346_777875025","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Execute Packet Capture Queries</h1>\n<p>Specify the fo
llowing to filter the packet capture query:</p>\n<ul>\n<li><em>end time</em> - The ending time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>start time</em> - The starting time of the query in yyyyMMdd format (e.g. 20170428)</li>\n<li><em>query</em> - The <a href=\"https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-language\">Stellar</a> query (i.e. a Stellar expression that returns <code>true</code> or <code>false</code>) to specify the packets.</li>\n</ul>\n<p>The available fields to use in the queries are as follows:</p>\n<ul>\n<li><code>ip_src_addr</code> - The source IP address of the packets filtered</li>\n<li><code>ip_src_port</code> - The source port of the packets filtered</li>\n<li><code>ip_dst_addr</code> - The destination IP address of the packets filtered</li>\n<li><code>ip_dst_port</code> - The destination port of the packets filtered</li>\n<li><code>packet</code> - The raw packet (for use with the <code>BYTEARR
AY_MATCHER</code> function)</li>\n</ul>\n<p>You can use any <a href=\"https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-core-functions\">Stellar function</a></p>\n<h2>Simple Boolean Expressions</h2>\n<p>For example:</p>\n<ul>\n<li><code>ip_dst_port == 8080</code> would return all packets where the destination port is <code>8080</code></li>\n<li><code>ip_dst_port in [ 8080, 80 ]</code> would return all packets where the destination port is either <code>8080</code> or <code>80</code></li>\n</ul>\n<h2>Common Network Functions on Metadata</h2>\n<p>For example:</p>\n<ul>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24')</code> would return all packets whose destination conforms to the CIDR <code>192.168.0.0/24</code></li>\n<li><code>IN_SUBNET(ip_dst_addr, '192.168.0.0/24') && ip_dst_port == 8080</code> would return all packets matching the CIDR and whose destination port is <code>8080</code></li>\n</ul>\n<h2>Filtering based on the Pack
et Contents</h2>\n<p>We use byteseek regular expressions to filter packets. The syntax for these is described <a href=\"https://github.com/nishihatapalmer/byteseek/blob/master/sequencesyntax.md\">here</a>.</p>\n<ul>\n<li><code>BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> in them anywhere.</li>\n<li><code>ip_dst_port==8080 && BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet)</code> would return all packets that contain the string <code>/api/v1</code> and have a destination port of <code>8080</code></li>\n<li><code>BYTEARRAY_MATCHER('ff(.){5}ff', packet)</code> would return all packets containing a binary regex with <code>0xff</code> followed by any 5 bytes and then <code>0xff</code></li>\n</ul>\n<h1>The Output</h1>\n<p>The output will be a table of links to the various parts of the packet capture files. The files will be named in temporal order.</p>\n"},"dateCreated":"2017-05-01T17:11:49+0000","stat
us":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1724","dateFinished":"2017-05-01T17:36:26+0000","dateStarted":"2017-05-01T17:36:26+0000","focus":true},{"text":"%sh\nexport PCAP_ZEPPELIN_RUN=$(find /usr -name pcap_zeppelin_run.sh)\nexport RECORDS_PER_FILE=10000\nexport NUMBER_OF_REDUCERS=10\nexport DATE_FORMAT=\"yyyyMMdd\"\nexport PCAP_DATA_PATH=\"/apps/metron/pcap\"\n\n$PCAP_ZEPPELIN_RUN \"${query}\" \"${start time}\" \"${optional end time}\"","dateUpdated":"2017-05-01T17:35:04+0000","config":{"colWidth":12,"editorMode":"ace/mode/sh","editorHide":false,"graph":{"mode":"table","height":164,"optionOpen":false,"keys":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"values":[],"groups":[],"scatter":{"xAxis":{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}}},"e
nabled":true},"settings":{"params":{"start time":"20170428","end time":"","query":"BYTEARRAY_MATCHER('\\\\'/api/v1\\\\'', packet) && ip_dst_port == 8080","optional end time":""},"forms":{"optional end time":{"name":"optional end time","defaultValue":"","hidden":false},"query":{"name":"query","defaultValue":"","hidden":false},"start time":{"name":"start time","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493658709562_1009386142","id":"20170428-181957_829993114","result":{"code":"SUCCESS","type":"TABLE","msg":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now\n%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>\n","comment":"","msgTable":[[{"value":"%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578
%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>"}]],"columnNames":[{"name":"Packets conforming to BYTEARRAY_MATCHER('\\'/api/v1\\'', packet) && ip_dst_port == 8080 starting at 20170428 ending now","index":0,"aggr":"sum"}],"rows":[["%html <a href=\"http://node1:50070/webhdfs/v1/user/zeppelin/queries/pcap-data-20170501173442578/pcap-data-20170501173442578%2B0000.pcap?op=OPEN\">pcap-data-20170501173442578+0000.pcap</a>"]]},"dateCreated":"2017-05-01T17:11:49+0000","dateStarted":"2017-05-01T17:35:04+0000","dateFinished":"2017-05-01T17:34:55+0000","status":"RUNNING","progressUpdateIntervalMs":500,"$$hashKey":"object:1725","focus":true},{"dateUpdated":"2017-05-01T17:34:55+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown","editorHide":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493658709563_1009001393","id":"20170428-184335_
1604096389","dateCreated":"2017-05-01T17:11:49+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:1726","dateFinished":"2017-05-01T17:34:54+0000","dateStarted":"2017-05-01T17:34:54+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h1>Troubleshooting</h1>\n<p>If you are having problems with the above form, the following may help.</p>\n<h2>I see <code>Terminated by SIGINTERRUPT</code> or something similar in the output!</h2>\n<p>PCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\n<br />They can do this by changing the <code>shell.command.timeout.millisecs</code> property for the <code>sh</code> interpreter to a larger value, likely <code>100000</code>.</p>\n<h2>I do not see a table of URLs to pcap files in my output, what happened?</h2>\n<p>If an error happens, the log of the pcap querying utility will be displayed instead of an output. Please co
ntact an administrator with this output to debug further.</p>\n"},"text":"%md\n# Troubleshooting\n\nIf you are having problems with the above form, the following may help.\n\n## I see `Terminated by SIGINTERRUPT` or something similar in the output!\n\nPCAP filtering happens via a batch process and on busy systems, this can take some time. You very well may need to request this of your system administrator.\nThey can do this by changing the `shell.command.timeout.millisecs` property for the `sh` interpreter to a larger value, likely `100000`.\n\n## I do not see a table of URLs to pcap files in my output, what happened?\n\nIf an error happens, the log of the pcap querying utility will be displayed instead of an output. Please contact an administrator with this output to debug further.","focus":true},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobN
ame":"paragraph_1493659778398_1082698332","id":"20170501-172938_213921861","dateCreated":"2017-05-01T17:29:38+0000","status":"READY","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:1972"}],"name":"metron/pcap","id":"2CEEXVR1W","angularObjects":{"2CFNGE6TP:shared_process":[],"2CGUX5TSW:shared_process":[],"2CETVR2AB:shared_process":[],"2CF163WFX:shared_process":[],"2CG4YXKUV:shared_process":[],"2CG6QDFF7:shared_process":[]},"config":{"looknfeel":"default"},"info":{}}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliConfig.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliConfig.java
index 294844f..1d8e3f3 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliConfig.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliConfig.java
@@ -18,12 +18,19 @@
package org.apache.metron.pcap.query;
import org.apache.commons.lang3.StringUtils;
+import org.apache.metron.common.system.Clock;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
+import java.util.UUID;
+import java.util.function.Consumer;
+import java.util.function.Function;
public class CliConfig {
+ public interface PrefixStrategy extends Function<Clock, String>{}
+
private boolean showHelp;
+ private String prefix;
private String basePath;
private String baseOutputPath;
private long startTime;
@@ -32,13 +39,23 @@ public class CliConfig {
private int numRecordsPerFile;
private DateFormat dateFormat;
- public CliConfig() {
+
+ public CliConfig(PrefixStrategy prefixStrategy) {
showHelp = false;
basePath = "";
baseOutputPath = "";
startTime = -1L;
endTime = -1L;
numReducers = 0;
+ prefix = prefixStrategy.apply(new Clock());
+ }
+
+ public String getPrefix() {
+ return prefix;
+ }
+
+ public void setPrefix(String prefix) {
+ this.prefix = prefix;
}
public int getNumReducers() {
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
index 83e9fcf..d5976ae 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/CliParser.java
@@ -29,8 +29,10 @@ public class CliParser {
public static final int NUM_REDUCERS_DEFAULT = 10;
public static final int NUM_RECORDS_PER_FILE_DEFAULT = 10000;
private CommandLineParser parser;
+ protected CliConfig.PrefixStrategy prefixStrategy;
- public CliParser() {
+ public CliParser(CliConfig.PrefixStrategy prefixStrategy) {
+ this.prefixStrategy = prefixStrategy;
parser = new PosixParser();
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
index df653e1..03caed7 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliConfig.java
@@ -27,7 +27,8 @@ public class FixedCliConfig extends CliConfig {
private Map<String, String> fixedFields;
- public FixedCliConfig() {
+ public FixedCliConfig(PrefixStrategy prefixStrategy) {
+ super(prefixStrategy);
this.fixedFields = new LinkedHashMap<>();
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
index fda8692..4e1bfcf 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/FixedCliParser.java
@@ -26,7 +26,8 @@ import org.apache.metron.pcap.PcapHelper;
public class FixedCliParser extends CliParser {
private Options fixedOptions;
- public FixedCliParser() {
+ public FixedCliParser(CliConfig.PrefixStrategy prefixStrategy) {
+ super(prefixStrategy);
fixedOptions = buildFixedOptions();
}
@@ -38,6 +39,7 @@ public class FixedCliParser extends CliParser {
options.addOption(newOption("dp", "ip_dst_port", true, "Destination port"));
options.addOption(newOption("p", "protocol", true, "IP Protocol"));
options.addOption(newOption("pf", "packet_filter", true, "Packet Filter regex"));
+ options.addOption(newOption("pre", "prefix", true, "Result file prefix to use"));
options.addOption(newOption("ir", "include_reverse", false, "Indicates if filter should check swapped src/dest addresses and IPs"));
return options;
}
@@ -51,7 +53,7 @@ public class FixedCliParser extends CliParser {
*/
public FixedCliConfig parse(String[] args) throws ParseException, java.text.ParseException {
CommandLine commandLine = getParser().parse(fixedOptions, args);
- FixedCliConfig config = new FixedCliConfig();
+ FixedCliConfig config = new FixedCliConfig(prefixStrategy);
super.parse(commandLine, config);
config.putFixedField(Constants.Fields.SRC_ADDR.getName(), commandLine.getOptionValue("ip_src_addr"));
config.putFixedField(Constants.Fields.DST_ADDR.getName(), commandLine.getOptionValue("ip_dst_addr"));
@@ -60,6 +62,9 @@ public class FixedCliParser extends CliParser {
config.putFixedField(Constants.Fields.PROTOCOL.getName(), commandLine.getOptionValue("protocol"));
config.putFixedField(Constants.Fields.INCLUDES_REVERSE_TRAFFIC.getName(), Boolean.toString(commandLine.hasOption("include_reverse")));
config.putFixedField(PcapHelper.PacketFields.PACKET_FILTER.getName(), commandLine.getOptionValue("packet_filter"));
+ if(commandLine.hasOption("prefix")) {
+ config.setPrefix(commandLine.getOptionValue("prefix"));
+ }
return config;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
index d2e6807..b02e2e2 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/PcapCli.java
@@ -36,22 +36,28 @@ import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
+import java.util.UUID;
public class PcapCli {
private static final Logger LOGGER = LoggerFactory.getLogger(PcapCli.class);
+ public static final CliConfig.PrefixStrategy PREFIX_STRATEGY = clock -> {
+ String timestamp = new Clock().currentTimeFormatted("yyyyMMddHHmm");
+ String uuid = UUID.randomUUID().toString().replaceAll("-", "");
+ return String.format("%s-%s", timestamp, uuid);
+ };
private final PcapJob jobRunner;
private final ResultsWriter resultsWriter;
- private final Clock clock;
+ private final CliConfig.PrefixStrategy prefixStrategy;
public static void main(String[] args) {
- int status = new PcapCli(new PcapJob(), new ResultsWriter(), new Clock()).run(args);
+ int status = new PcapCli(new PcapJob(), new ResultsWriter(), PREFIX_STRATEGY).run(args);
System.exit(status);
}
- public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, Clock clock) {
+ public PcapCli(PcapJob jobRunner, ResultsWriter resultsWriter, CliConfig.PrefixStrategy prefixStrategy) {
this.jobRunner = jobRunner;
this.resultsWriter = resultsWriter;
- this.clock = clock;
+ this.prefixStrategy = prefixStrategy;
}
public int run(String[] args) {
@@ -72,7 +78,7 @@ public class PcapCli {
}
CliConfig commonConfig = null;
if ("fixed".equals(jobType)) {
- FixedCliParser fixedParser = new FixedCliParser();
+ FixedCliParser fixedParser = new FixedCliParser(prefixStrategy);
FixedCliConfig config = null;
try {
config = fixedParser.parse(otherArgs);
@@ -110,7 +116,7 @@ public class PcapCli {
return -1;
}
} else if ("query".equals(jobType)) {
- QueryCliParser queryParser = new QueryCliParser();
+ QueryCliParser queryParser = new QueryCliParser(prefixStrategy);
QueryCliConfig config = null;
try {
config = queryParser.parse(otherArgs);
@@ -151,11 +157,12 @@ public class PcapCli {
return -1;
}
try {
+
Iterable<List<byte[]>> partitions = Iterables.partition(results, commonConfig.getNumRecordsPerFile());
+ int part = 1;
if (partitions.iterator().hasNext()) {
for (List<byte[]> data : partitions) {
- String timestamp = clock.currentTimeFormatted("yyyyMMddHHmmssSSSZ");
- String outFileName = String.format("pcap-data-%s.pcap", timestamp);
+ String outFileName = String.format("pcap-data-%s+%04d.pcap", commonConfig.getPrefix(), part++);
if(data.size() > 0) {
resultsWriter.write(data, outFileName);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliConfig.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliConfig.java
index 3d06e1d..67f045f 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliConfig.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliConfig.java
@@ -20,6 +20,10 @@ package org.apache.metron.pcap.query;
public class QueryCliConfig extends CliConfig {
private String query;
+ public QueryCliConfig(PrefixStrategy prefixStrategy) {
+ super(prefixStrategy);
+ }
+
public String getQuery() {
return query;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java
index 72b5a95..d6e5cd1 100644
--- a/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java
+++ b/metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/pcap/query/QueryCliParser.java
@@ -24,13 +24,15 @@ import org.apache.commons.cli.ParseException;
public class QueryCliParser extends CliParser {
private Options queryOptions;
- public QueryCliParser() {
+ public QueryCliParser(CliConfig.PrefixStrategy prefixStrategy) {
+ super(prefixStrategy);
queryOptions = setupOptions();
}
private Options setupOptions() {
Options options = buildOptions();
options.addOption(newOption("q", "query", true, "Query string to use as a filter"));
+ options.addOption(newOption("pre", "prefix", true, "Result file prefix to use"));
return options;
}
@@ -43,11 +45,14 @@ public class QueryCliParser extends CliParser {
*/
public QueryCliConfig parse(String[] args) throws ParseException, java.text.ParseException {
CommandLine commandLine = getParser().parse(queryOptions, args);
- QueryCliConfig config = new QueryCliConfig();
+ QueryCliConfig config = new QueryCliConfig(prefixStrategy);
super.parse(commandLine, config);
if (commandLine.hasOption("query")) {
config.setQuery(commandLine.getOptionValue("query"));
}
+ if(commandLine.hasOption("prefix")) {
+ config.setPrefix(commandLine.getOptionValue("prefix"));
+ }
return config;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh b/metron-platform/metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh
new file mode 100755
index 0000000..3561c9d
--- /dev/null
+++ b/metron-platform/metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh
@@ -0,0 +1,88 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+METRON_VERSION=0.4.0
+METRON_HOME=${METRON_HOME:-"/usr/metron/$METRON_VERSION"}
+DATE_FORMAT=${DATE_FORMAT:-"yyyyMMdd"}
+USER=$(whoami)
+USER_HOMEDIR=${USER_HOMEDIR:-`hdfs getconf -confKey dfs.user.home.dir.prefix`/$USER}
+QUERY_HOME=${QUERY_HOME:-"$USER_HOMEDIR/queries"}
+WEBHDFS_HOSTNAME=${WEBHDFS_HOSTNAME:-`hdfs getconf -confKey dfs.namenode.http-address`}
+QUERY=${QUERY:-$1}
+START_TIME=${START_TIME:-$2}
+END_TIME=${END_TIME:-$3}
+RECORDS_PER_FILE=${RECORDS_PER_FILE:-10000}
+NUMBER_OF_REDUCERS=${NUMBER_OF_REDUCERS:-10}
+PCAP_DATA_PATH=${PCAP_DATA_PATH:-/apps/metron/pcap}
+if [ -z "$QUERY" ]; then
+ echo "You must specify a query."
+ exit 1
+fi
+if [ -z "$START_TIME" ]; then
+ echo "You must specify a start time."
+ exit 2
+fi
+TIMESTAMP_EPOCH=$(date +%s%N | cut -b1-13)
+if [ -f /proc/sys/kernel/random/uuid ];then
+ UUID=$(cat /proc/sys/kernel/random/uuid | sed 's/-//g')
+ PREFIX="$TIMESTAMP_EPOCH-$UUID"
+else
+ QUERY_HASH=$(echo $QUERY | md5sum | awk '{print $1}')
+ PREFIX="$TIMESTAMP_EPOCH-$QUERY_HASH"
+fi
+if [ -z "$END_TIME" ]; then
+ CMD=$($METRON_HOME/bin/pcap_query.sh query --prefix "$PREFIX" --query "$QUERY" -st "$START_TIME" -df "$DATE_FORMAT" -rpf "$RECORDS_PER_FILE" -nr "$NUMBER_OF_REDUCERS" -bp "$PCAP_DATA_PATH" \"2>&1)
+ SUMMARY="Packets conforming to $QUERY starting at $START_TIME ending now"
+else
+ CMD=$($METRON_HOME/bin/pcap_query.sh query --prefix "$PREFIX" --query "$QUERY" -st "$START_TIME" -et "$END_TIME" -df "$DATE_FORMAT" -rpf "$RECORDS_PER_FILE" -nr "$NUMBER_OF_REDUCERS" -bp "$PCAP_DATA_PATH" 2>&1)
+ SUMMARY="Packets conforming to $QUERY starting at $START_TIME ending at $END_TIME"
+fi
+
+FAILED=$(echo $CMD | grep "Unable to complete query due to errors")
+if [ -z "$FAILED" ];then
+ PATTERN="pcap-data-$PREFIX"
+ hadoop fs -mkdir -p $QUERY_HOME/$PATTERN && hadoop fs -put $PATTERN* $QUERY_HOME/$PATTERN
+ FAILED=$?
+ if [ $FAILED -eq 0 ];then
+ echo "%html"
+ echo "<h4>$SUMMARY</h4>"
+ echo "<ul>"
+ for i in $(ls $PATTERN*.pcap | sort -n);do
+ FILENAME=$(echo $i | sed 's/+/%2B/g')
+ SIZE=$(du -h $i | awk '{print $1}')
+ echo "<li><a href=\"http://$WEBHDFS_HOSTNAME/webhdfs/v1$QUERY_HOME/$PATTERN/$FILENAME?op=OPEN\">$i</a> ($SIZE)</li>"
+ rm $i
+ done
+ echo "</ul>"
+ echo "<small>NOTE: There are $RECORDS_PER_FILE records per file</small>"
+ else
+ echo "Unable to create $QUERY_HOME/$PATTERN"
+ exit 3
+ fi
+else
+ echo "%html <pre>FAILED JOB:"
+ echo "QUERY: $QUERY"
+ echo "START_TIME: $START_TIME"
+ echo "DATE_FORMAT: $DATE_FORMAT"
+ echo "METRON_HOME: $METRON_HOME"
+ echo "DATE_FORMAT: $DATE_FORMAT"
+ echo "Output:"
+ echo "$CMD"
+ echo "</pre>"
+fi
http://git-wip-us.apache.org/repos/asf/metron/blob/1d27a32d/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
index 4f441f1..7202819 100644
--- a/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
+++ b/metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/query/PcapCliTest.java
@@ -92,11 +92,10 @@ public class PcapCliTest {
}};
when(jobRunner.query(eq(base_path), eq(base_output_path), anyLong(), anyLong(), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(FixedPcapFilter.Configurator.class))).thenReturn(iterable);
- when(clock.currentTimeFormatted("yyyyMMddHHmmssSSSZ")).thenReturn("20160615183527162+0000");
- PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock);
+ PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock -> "random_prefix");
assertThat("Expect no errors on run", cli.run(args), equalTo(0));
- Mockito.verify(resultsWriter).write(pcaps, "pcap-data-20160615183527162+0000.pcap");
+ Mockito.verify(resultsWriter).write(pcaps, "pcap-data-random_prefix+0001.pcap");
}
@Test
@@ -133,11 +132,10 @@ public class PcapCliTest {
}};
when(jobRunner.query(eq(base_path), eq(base_output_path), anyLong(), anyLong(), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(FixedPcapFilter.Configurator.class))).thenReturn(iterable);
- when(clock.currentTimeFormatted("yyyyMMddHHmmssSSSZ")).thenReturn("20160615183527162+0000");
- PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock);
+ PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock -> "random_prefix");
assertThat("Expect no errors on run", cli.run(args), equalTo(0));
- Mockito.verify(resultsWriter).write(pcaps, "pcap-data-20160615183527162+0000.pcap");
+ Mockito.verify(resultsWriter).write(pcaps, "pcap-data-random_prefix+0001.pcap");
}
@Test
@@ -177,11 +175,10 @@ public class PcapCliTest {
long startAsNanos = asNanos("2016-06-13-18:35.00", "yyyy-MM-dd-HH:mm.ss");
long endAsNanos = asNanos("2016-06-15-18:35.00", "yyyy-MM-dd-HH:mm.ss");
when(jobRunner.query(eq(base_path), eq(base_output_path), eq(startAsNanos), eq(endAsNanos), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(FixedPcapFilter.Configurator.class))).thenReturn(iterable);
- when(clock.currentTimeFormatted("yyyyMMddHHmmssSSSZ")).thenReturn("20160615183527162+0000");
- PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock);
+ PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock -> "random_prefix");
assertThat("Expect no errors on run", cli.run(args), equalTo(0));
- Mockito.verify(resultsWriter).write(pcaps, "pcap-data-20160615183527162+0000.pcap");
+ Mockito.verify(resultsWriter).write(pcaps, "pcap-data-random_prefix+0001.pcap");
}
private long asNanos(String inDate, String format) throws ParseException {
@@ -211,11 +208,10 @@ public class PcapCliTest {
String query = "some query string";
when(jobRunner.query(eq(base_path), eq(base_output_path), anyLong(), anyLong(), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(QueryPcapFilter.Configurator.class))).thenReturn(iterable);
- when(clock.currentTimeFormatted("yyyyMMddHHmmssSSSZ")).thenReturn("20160615183527162+0000");
- PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock);
+ PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock -> "random_prefix");
assertThat("Expect no errors on run", cli.run(args), equalTo(0));
- Mockito.verify(resultsWriter).write(pcaps, "pcap-data-20160615183527162+0000.pcap");
+ Mockito.verify(resultsWriter).write(pcaps, "pcap-data-random_prefix+0001.pcap");
}
@Test
@@ -240,11 +236,10 @@ public class PcapCliTest {
String query = "some query string";
when(jobRunner.query(eq(base_path), eq(base_output_path), anyLong(), anyLong(), anyInt(), eq(query), isA(Configuration.class), isA(FileSystem.class), isA(QueryPcapFilter.Configurator.class))).thenReturn(iterable);
- when(clock.currentTimeFormatted("yyyyMMddHHmmssSSSZ")).thenReturn("20160615183527162+0000");
- PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock);
+ PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock -> "random_prefix");
assertThat("Expect no errors on run", cli.run(args), equalTo(0));
- Mockito.verify(resultsWriter).write(pcaps, "pcap-data-20160615183527162+0000.pcap");
+ Mockito.verify(resultsWriter).write(pcaps, "pcap-data-random_prefix+0001.pcap");
}
// INVALID OPTION CHECKS
@@ -281,7 +276,7 @@ public class PcapCliTest {
PrintStream errOutStream = new PrintStream(new BufferedOutputStream(ebos));
System.setErr(errOutStream);
- PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock);
+ PcapCli cli = new PcapCli(jobRunner, resultsWriter, clock -> "random_prefix");
assertThat("Expect errors on run", cli.run(args), equalTo(-1));
assertThat("Expect missing required option error: " + ebos.toString(), ebos.toString().contains(optMsg), equalTo(true));
assertThat("Expect usage to be printed: " + bos.toString(), bos.toString().contains("usage: " + type + " filter options"), equalTo(true));
[43/44] metron git commit: METRON-937 Pycapa - Consume Messages from
Begin, End, or Stored Offsets (nickwallen) closes apache/metron#570
Posted by ma...@apache.org.
METRON-937 Pycapa - Consume Messages from Begin, End, or Stored Offsets (nickwallen) closes apache/metron#570
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/8779eb3f
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/8779eb3f
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/8779eb3f
Branch: refs/heads/Metron_0.4.0
Commit: 8779eb3fe6d2767d6b79d665adec735380cf2d61
Parents: 74bc236
Author: nickwallen <ni...@nickallen.org>
Authored: Thu Jun 1 11:15:42 2017 -0400
Committer: nickallen <ni...@apache.org>
Committed: Thu Jun 1 11:15:42 2017 -0400
----------------------------------------------------------------------
metron-sensors/pycapa/README.md | 169 +++++++++++++-----------
metron-sensors/pycapa/pycapa/consumer.py | 51 ++++++-
metron-sensors/pycapa/pycapa/producer.py | 24 +++-
metron-sensors/pycapa/pycapa/pycapa_cli.py | 34 +++--
4 files changed, 179 insertions(+), 99 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/8779eb3f/metron-sensors/pycapa/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md
index 6f41d63..fed1399 100644
--- a/metron-sensors/pycapa/README.md
+++ b/metron-sensors/pycapa/README.md
@@ -72,19 +72,22 @@ Pycapa has two primary runtime modes.
### Parameters
```
-$ pycapa -h
+$ pycapa --help
usage: pycapa [-h] [-p] [-c] [-k KAFKA_BROKERS] [-t KAFKA_TOPIC]
- [-i NETWORK_IFACE] [-m MAX_PACKETS] [-pp PRETTY_PRINT]
- [-ll LOG_LEVEL] [-X KAFKA_CONFIGS] [-s SNAPLEN]
+ [-o {begin,end,stored}] [-i NETWORK_IFACE] [-m MAX_PACKETS]
+ [-pp PRETTY_PRINT] [-ll LOG_LEVEL] [-X KAFKA_CONFIGS]
+ [-s SNAPLEN]
optional arguments:
-h, --help show this help message and exit
-p, --producer sniff packets and send to kafka
-c, --consumer read packets from kafka
-k KAFKA_BROKERS, --kafka-broker KAFKA_BROKERS
- kafka broker(s)
+ kafka broker(s) as host:port
-t KAFKA_TOPIC, --kafka-topic KAFKA_TOPIC
kafka topic
+ -o {begin,end,stored}, --kafka-offset {begin,end,stored}
+ kafka offset to consume from; default=end
-i NETWORK_IFACE, --interface NETWORK_IFACE
network interface to listen on
-m MAX_PACKETS, --max-packets MAX_PACKETS
@@ -92,91 +95,81 @@ optional arguments:
-pp PRETTY_PRINT, --pretty-print PRETTY_PRINT
pretty print every X packets
-ll LOG_LEVEL, --log-level LOG_LEVEL
- set the log level
+ set the log level; DEBUG, INFO, WARN
-X KAFKA_CONFIGS define a kafka client parameter; key=value
-s SNAPLEN, --snaplen SNAPLEN
- snapshot length
+ capture only the first X bytes of each packet;
+ default=65535
```
### Examples
-**Example**: Capture 10 packets from the `eth0` network interface and forward those to a Kafka topic called `pcap` running on `localhost:9092`.
- ```
- $ pycapa --producer \
- --interface eth0 \
- --kafka-broker localhost:9092 \
- --kafka-topic pcap \
- --max-packets 10
- INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'AWBHMIAESAHJ'}
- INFO:root:Starting packet capture
- INFO:root:Waiting for '10' message(s) to flush
- INFO:root:'10' packet(s) in, '10' packet(s) out
- ```
+#### Example 1
-**Example**: Capture packets until SIGINT is received. A SIGINT is the interrupt signal sent when entering CTRL-D in the console.
- ```
- $ pycapa --producer \
- --interface eth0 \
- --kafka-broker localhost:9092 \
- --kafka-topic pcap
- INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'EULLGDOMZDCT'}
- INFO:root:Starting packet capture
- ^C
- INFO:root:Clean shutdown process started
- INFO:root:Waiting for '0' message(s) to flush
- INFO:root:'7' packet(s) in, '7' packet(s) out
- ```
+Capture 10 packets from the `eth0` network interface and forward those to a Kafka topic called `pcap` running on `localhost:9092`. The process will not terminate until all messages have been delivered to Kafka.
-**Example**: While capturing packets, output diagnostic information every 10 packets.
- ```
- $ pycapa --producer \
- --interface en0 \
- --kafka-broker localhost:9092 \
- --kafka-topic pcap \
- --pretty-print 10
- INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'YMDSEEDIHVWD'}
+```
+$ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10
+INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'AWBHMIAESAHJ'}
+INFO:root:Starting packet capture
+INFO:root:Waiting for '6' message(s) to flush
+INFO:root:'10' packet(s) in, '10' packet(s) out
+```
+
+#### Example 2
+
+Capture packets until SIGINT is received (the interrupt signal sent when entering CTRL-C in the console.) In this example, nothing will be reported as packets are captured and delivered to Kafka. Simply wait a few seconds, then type CTRL-C and the number of packets will be reported.
+
+```
+$ pycapa --producer \
+ --interface en0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap
+INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'EULLGDOMZDCT'}
+INFO:root:Starting packet capture
+^C
+INFO:root:Clean shutdown process started
+INFO:root:Waiting for '2' message(s) to flush
+INFO:root:'21' packet(s) in, '21' packet(s) out
+```
+
+#### Example 3
+
+While capturing packets, output diagnostic information every 5 packets. Diagnostics will report when packets have been received from the network interface and when they have been successfully delivered to Kafka.
+
+```
+$ pycapa --producer \
+ --interface eth0 \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --pretty-print 5
+ INFO:root:Connecting to Kafka; {'bootstrap.servers': 'localhost:9092', 'group.id': 'UAWINMBDNQEH'}
INFO:root:Starting packet capture
- 10 packet(s) received
- ac bc 32 bf 0d 43 b8 3e 59 8b 8a 8a 08 00 45 00
- 00 3c 00 00 40 00 40 06 b9 66 c0 a8 00 02 c0 a8
- 00 03 1f 7c d7 14 5f 8b 82 b4 a8 c5 f6 63 a0 12
- 38 90 59 cc 00 00 02 04 05 b4 04 02 08 0a 00 51
- 44 17 39 43 3e 9b 01 03 03 04
- 20 packet(s) received
- 01 00 5e 00 00 fb ac bc 32 bf 0d 43 08 00 45 00
- 00 44 d2 09 00 00 ff 11 47 f8 c0 a8 00 03 e0 00
- 00 fb 14 e9 14 e9 00 30 69 fc 00 00 00 00 00 01
- 00 00 00 00 00 00 0b 5f 67 6f 6f 67 6c 65 63 61
- 73 74 04 5f 74 63 70 05 6c 6f 63 61 6c 00 00 0c
- 80 01
+ Packet received[5]
+ Packet delivered[5]: date=2017-05-08 14:48:54.474031 topic=pcap partition=0 offset=29086 len=42
+ Packet received[10]
+ Packet received[15]
+ Packet delivered[10]: date=2017-05-08 14:48:58.879710 topic=pcap partition=0 offset=0 len=187
+ Packet delivered[15]: date=2017-05-08 14:48:59.633127 topic=pcap partition=0 offset=0 len=43
+ Packet received[20]
+ Packet delivered[20]: date=2017-05-08 14:49:01.949628 topic=pcap partition=0 offset=29101 len=134
+ Packet received[25]
^C
INFO:root:Clean shutdown process started
- INFO:root:Waiting for '2' message(s) to flush
- INFO:root:'20' packet(s) in, '20' packet(s) out
- ```
+ Packet delivered[25]: date=2017-05-08 14:49:03.589940 topic=pcap partition=0 offset=0 len=142
+ INFO:root:Waiting for '1' message(s) to flush
+ INFO:root:'27' packet(s) in, '27' packet(s) out
-**Example**: Consume 10 packets from the Kafka topic `pcap` running on `localhost:9092`, then pipe those into Wireshark for DPI.
- ```
- $ pycapa --consumer \
- --kafka-broker localhost:9092 \
- --kafka-topic pcap \
- --max-packets 10 \
- | tshark -i -
- Capturing on 'Standard input'
- 1 0.000000 ArrisGro_0e:65:df → Apple_bf:0d:43 ARP 56 Who has 192.168.0.3? Tell 192.168.0.1
- 2 0.000044 Apple_bf:0d:43 → ArrisGro_0e:65:df ARP 42 192.168.0.3 is at ac:bc:32:bf:0d:43
- 3 0.203495 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
- 4 2.031988 192.168.0.3 → 96.27.183.249 TCP 54 55110 → 443 [ACK] Seq=1 Ack=1 Win=4108 Len=0
- 5 2.035816 192.30.253.125 → 192.168.0.3 TLSv1.2 97 Application Data
- 6 2.035892 192.168.0.3 → 192.30.253.125 TCP 66 54671 → 443 [ACK] Seq=1 Ack=32 Win=4095 Len=0 TSval=961120495 TSecr=2658503052
- 7 2.035994 192.168.0.3 → 192.30.253.125 TLSv1.2 101 Application Data
- 8 2.053866 96.27.183.249 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 55110 [ACK] Seq=1 Ack=2 Win=243 Len=0 TSval=728145145 TSecr=961030381
- 9 2.083872 192.30.253.125 → 192.168.0.3 TCP 66 443 → 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495
- 10 3.173189 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
- 10 packets captured
- ```
+```
+
+#### Example 4
+
+Consume 10 packets and create a libpcap-compliant pcap file.
-**Example**: Consume 10 packets and create a libpcap-compliant pcap file.
```
$ pycapa --consumer \
--kafka-broker localhost:9092 \
@@ -196,6 +189,30 @@ optional arguments:
10 2.494769 192.168.0.3 → 224.0.0.251 MDNS 82 Standard query 0x0000 PTR _googlecast._tcp.local, "QM" question
```
+#### Example 5
+
+Consume 10 packets from the Kafka topic `pcap` running on `localhost:9092`, then pipe those into Wireshark for DPI.
+
+```
+$ pycapa --consumer \
+ --kafka-broker localhost:9092 \
+ --kafka-topic pcap \
+ --max-packets 10 \
+ | tshark -i -
+Capturing on 'Standard input'
+ 1 0.000000 ArrisGro_0e:65:df → Apple_bf:0d:43 ARP 56 Who has 192.168.0.3? Tell 192.168.0.1
+ 2 0.000044 Apple_bf:0d:43 → ArrisGro_0e:65:df ARP 42 192.168.0.3 is at ac:bc:32:bf:0d:43
+ 3 0.203495 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+ 4 2.031988 192.168.0.3 → 96.27.183.249 TCP 54 55110 → 443 [ACK] Seq=1 Ack=1 Win=4108 Len=0
+ 5 2.035816 192.30.253.125 → 192.168.0.3 TLSv1.2 97 Application Data
+ 6 2.035892 192.168.0.3 → 192.30.253.125 TCP 66 54671 → 443 [ACK] Seq=1 Ack=32 Win=4095 Len=0 TSval=961120495 TSecr=2658503052
+ 7 2.035994 192.168.0.3 → 192.30.253.125 TLSv1.2 101 Application Data
+ 8 2.053866 96.27.183.249 → 192.168.0.3 TCP 66 [TCP ACKed unseen segment] 443 → 55110 [ACK] Seq=1 Ack=2 Win=243 Len=0 TSval=728145145 TSecr=961030381
+ 9 2.083872 192.30.253.125 → 192.168.0.3 TCP 66 443 → 54671 [ACK] Seq=32 Ack=36 Win=31 Len=0 TSval=2658503087 TSecr=961120495
+ 10 3.173189 fe80::1286:8cff:fe0e:65df → ff02::1 ICMPv6 134 Router Advertisement from 10:86:8c:0e:65:df
+10 packets captured
+```
+
### Kerberos
The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment.
http://git-wip-us.apache.org/repos/asf/metron/blob/8779eb3f/metron-sensors/pycapa/pycapa/consumer.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/consumer.py b/metron-sensors/pycapa/pycapa/consumer.py
index 7029f25..484ae3c 100644
--- a/metron-sensors/pycapa/pycapa/consumer.py
+++ b/metron-sensors/pycapa/pycapa/consumer.py
@@ -23,7 +23,7 @@ import random
import logging
import time
import struct
-from confluent_kafka import Consumer, KafkaException, KafkaError
+from confluent_kafka import Consumer, KafkaException, KafkaError, OFFSET_BEGINNING, OFFSET_END, OFFSET_STORED
from common import to_date, to_hex, unpack_ts
@@ -56,16 +56,51 @@ def packet_header(msg):
return hdr
+def seek_to_end(consumer, partitions):
+ """ Advance all partitions to the last offset. """
+
+ # advance to the end, ignoring any committed offsets
+ for p in partitions:
+ p.offset = OFFSET_END
+ consumer.assign(partitions)
+
+
+def seek_to_begin(consumer, partitions):
+ """ Advance all partitions to the first offset. """
+
+ # advance to the end, ignoring any committed offsets
+ for p in partitions:
+ p.offset = OFFSET_BEGINNING
+ consumer.assign(partitions)
+
+
+def seek_to_stored(consumer, partitions):
+ """ Advance all partitions to the stored offset. """
+
+ # advance to the end, ignoring any committed offsets
+ for p in partitions:
+ p.offset = OFFSET_STORED
+ consumer.assign(partitions)
+
+
def consumer(args, poll_timeout=3.0):
""" Consumes packets from a Kafka topic. """
# setup the signal handler
signal.signal(signal.SIGINT, signal_handler)
+ # where to start consuming messages from
+ kafka_offset_options = {
+ "begin": seek_to_begin,
+ "end": seek_to_end,
+ "stored": seek_to_stored
+ }
+ on_assign_cb = kafka_offset_options[args.kafka_offset]
+
# connect to kafka
logging.debug("Connecting to Kafka; %s", args.kafka_configs)
kafka_consumer = Consumer(args.kafka_configs)
- kafka_consumer.subscribe([args.kafka_topic])
+ kafka_consumer.subscribe([args.kafka_topic], on_assign=on_assign_cb)
# if 'pretty-print' not set, write libpcap global header
if args.pretty_print == 0:
@@ -85,8 +120,10 @@ def consumer(args, poll_timeout=3.0):
elif msg.error():
if msg.error().code() == KafkaError._PARTITION_EOF:
- logging.debug("reached end of topar: topic=%s, partition=%d, offset=%s", msg.topic(), msg.partition(), msg.offset())
- elif msg.error():
+ if args.pretty_print > 0:
+ print "Reached end of topar: topic=%s, partition=%d, offset=%s" % (
+ msg.topic(), msg.partition(), msg.offset())
+ else:
raise KafkaException(msg.error())
else:
@@ -103,9 +140,9 @@ def consumer(args, poll_timeout=3.0):
elif pkts_in % args.pretty_print == 0:
# pretty print
- print 'Packet: count=%s date=%s topic=%s' % (
- pkts_in, to_date(unpack_ts(msg.key())), args.kafka_topic)
- print to_hex(msg.value())
+ print 'Packet[%s]: date=%s topic=%s partition=%s offset=%s len=%s' % (
+ pkts_in, to_date(unpack_ts(msg.key())), args.kafka_topic,
+ msg.partition(), msg.offset(), len(msg.value()))
finally:
sys.stdout.close()
http://git-wip-us.apache.org/repos/asf/metron/blob/8779eb3f/metron-sensors/pycapa/pycapa/producer.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/producer.py b/metron-sensors/pycapa/pycapa/producer.py
index 7374522..ec21fdc 100644
--- a/metron-sensors/pycapa/pycapa/producer.py
+++ b/metron-sensors/pycapa/pycapa/producer.py
@@ -21,11 +21,11 @@ import pcapy
import argparse
import random
import logging
-from common import to_date, to_hex, pack_ts
+from common import to_date, to_hex, pack_ts, unpack_ts
from confluent_kafka import Producer
finished = threading.Event()
-
+producer_args = None
def signal_handler(signum, frame):
""" Initiates a clean shutdown for a SIGINT """
@@ -57,10 +57,18 @@ def delivery_callback(err, msg):
if err:
logging.error("message delivery failed: error=%s", err)
- else:
- logging.debug("message delivery succeeded: pkts_out=%d", delivery_callback.pkts_out)
+
+ elif msg is not None:
delivery_callback.pkts_out += 1
+ pretty_print = 0
+ pretty_print = producer_args.pretty_print
+
+ if pretty_print > 0 and delivery_callback.pkts_out % pretty_print == 0:
+ print 'Packet delivered[%s]: date=%s topic=%s partition=%s offset=%s len=%s' % (
+ delivery_callback.pkts_out, to_date(unpack_ts(msg.key())), msg.topic(),
+ msg.partition(), msg.offset(), len(msg.value()))
+
def producer(args, sniff_timeout_ms=500, sniff_promisc=True):
""" Captures packets from a network interface and sends them to a Kafka topic. """
@@ -68,6 +76,9 @@ def producer(args, sniff_timeout_ms=500, sniff_promisc=True):
# setup the signal handler
signal.signal(signal.SIGINT, signal_handler)
+ global producer_args
+ producer_args = args
+
# connect to kafka
logging.info("Connecting to Kafka; %s", args.kafka_configs)
kafka_producer = Producer(args.kafka_configs)
@@ -88,10 +99,9 @@ def producer(args, sniff_timeout_ms=500, sniff_promisc=True):
pkt_ts = timestamp(pkt_hdr)
kafka_producer.produce(args.kafka_topic, key=pack_ts(pkt_ts), value=pkt_raw, callback=delivery_callback)
- # debug messages, if needed
+ # pretty print, if needed
if args.pretty_print > 0 and pkts_in % args.pretty_print == 0:
- print '{} packet(s) received'.format(pkts_in)
- print to_hex(pkt_raw)
+ print 'Packet received[%s]' % (pkts_in)
# serve the callback queue
kafka_producer.poll(0)
http://git-wip-us.apache.org/repos/asf/metron/blob/8779eb3f/metron-sensors/pycapa/pycapa/pycapa_cli.py
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/pycapa/pycapa_cli.py b/metron-sensors/pycapa/pycapa/pycapa_cli.py
index f650280..609205a 100644
--- a/metron-sensors/pycapa/pycapa/pycapa_cli.py
+++ b/metron-sensors/pycapa/pycapa/pycapa_cli.py
@@ -39,13 +39,19 @@ def make_parser():
default=False)
parser.add_argument('-k', '--kafka-broker',
- help='kafka broker(s)',
+ help='kafka broker(s) as host:port',
dest='kafka_brokers')
parser.add_argument('-t', '--kafka-topic',
help='kafka topic',
dest='kafka_topic')
+ parser.add_argument('-o', '--kafka-offset',
+ help='kafka offset to consume from; default=end',
+ dest='kafka_offset',
+ choices=['begin','end','stored'],
+ default='end')
+
parser.add_argument('-i', '--interface',
help='network interface to listen on',
dest='interface',
@@ -64,7 +70,7 @@ def make_parser():
default=0)
parser.add_argument('-ll', '--log-level',
- help='set the log level',
+ help='set the log level; DEBUG, INFO, WARN',
dest='log_level',
default='INFO')
@@ -75,7 +81,7 @@ def make_parser():
action='append')
parser.add_argument('-s','--snaplen',
- help="snapshot length",
+ help="capture only the first X bytes of each packet; default=65535",
dest='snaplen',
type=int,
default=65535)
@@ -96,13 +102,21 @@ def keyval(input, delim="="):
def valid_args(args):
""" Validates the command-line arguments. """
- if args.producer and args.kafka_brokers and args.kafka_topic and args.interface:
- return True
- elif args.consumer and args.kafka_brokers and args.kafka_topic:
- return True
- else:
+ if not args.producer and not args.consumer:
+ print "error: expected either --consumer or --producer \n"
return False
+ elif args.producer and not (args.kafka_brokers and args.kafka_topic and args.interface):
+ print "error: missing required args: expected [--kafka-broker, --kafka-topic, --interface] \n"
+ return False
+
+ elif args.consumer and not (args.kafka_brokers and args.kafka_topic):
+ print "error: missing required args: expected [--kafka-broker, --kafka-topic] \n"
+ return False
+
+ else:
+ return True
+
def clean_kafka_configs(args):
""" Cleans and transforms the Kafka client configs. """
@@ -116,7 +130,7 @@ def clean_kafka_configs(args):
# boostrap servers can be set as a "-X bootstrap.servers=KAFKA:9092" or "-k KAFKA:9092"
bootstrap_key = "bootstrap.servers"
if(bootstrap_key not in configs):
- configs[bootstrap_key] = args.kafka_brokers;
+ configs[bootstrap_key] = args.kafka_brokers
# if no 'group.id', generate a random one
group_key = "group.id"
@@ -125,7 +139,9 @@ def clean_kafka_configs(args):
args.kafka_configs = configs
+
def main():
+
parser = make_parser()
args = parser.parse_args()
[09/44] metron git commit: METRON-840 All " ambari_*"
hosts need to have a /localrepo folder closes
apache/incubator-metron#563
Posted by ma...@apache.org.
METRON-840 All "ambari_*" hosts need to have a /localrepo folder closes apache/incubator-metron#563
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/082b218f
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/082b218f
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/082b218f
Branch: refs/heads/Metron_0.4.0
Commit: 082b218fb80ec8049cf7143d2a1c9a1537dd41dc
Parents: e08b7d4
Author: David Lyle <dl...@gmail.com>
Authored: Tue May 9 10:49:18 2017 -0400
Committer: David Lyle <dl...@gmail.com>
Committed: Tue May 9 10:49:18 2017 -0400
----------------------------------------------------------------------
.../roles/ambari_slave/tasks/main.yml | 4 ++-
.../roles/metron-rpms/tasks/copy_rpms.yml | 26 +++++++++++++++
.../metron-rpms/tasks/create_directory.yml | 28 +++++++++++++++++
.../roles/metron-rpms/tasks/create_repo.yml | 28 +++++++++++++++++
.../roles/metron-rpms/tasks/main.yml | 33 ++------------------
5 files changed, 88 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/082b218f/metron-deployment/roles/ambari_slave/tasks/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_slave/tasks/main.yml b/metron-deployment/roles/ambari_slave/tasks/main.yml
index 3fcd24f..ca81a7a 100644
--- a/metron-deployment/roles/ambari_slave/tasks/main.yml
+++ b/metron-deployment/roles/ambari_slave/tasks/main.yml
@@ -15,7 +15,9 @@
# limitations under the License.
#
---
-# tasks file for ambari_slave
+- include: ../roles/metron-rpms/tasks/create_directory.yml
+- include: ../roles/metron-rpms/tasks/create_repo.yml
+
- name: Install ambari-agent
yum:
name: ambari-agent
http://git-wip-us.apache.org/repos/asf/metron/blob/082b218f/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml b/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml
new file mode 100644
index 0000000..3410b7f
--- /dev/null
+++ b/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml
@@ -0,0 +1,26 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+- name: Copy Metron RPMs
+ copy:
+ src: "{{ item }}"
+ dest: /localrepo
+ owner: root
+ mode: 0755
+ with_fileglob:
+ - "{{ metron_rpm_glob }}"
+
http://git-wip-us.apache.org/repos/asf/metron/blob/082b218f/metron-deployment/roles/metron-rpms/tasks/create_directory.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/metron-rpms/tasks/create_directory.yml b/metron-deployment/roles/metron-rpms/tasks/create_directory.yml
new file mode 100644
index 0000000..0601625
--- /dev/null
+++ b/metron-deployment/roles/metron-rpms/tasks/create_directory.yml
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+
+- name: Clean out localrepo
+ file:
+ state: absent
+ path: /localrepo
+
+- name: Create localrepo directory
+ file:
+ path: /localrepo
+ state: directory
+ mode: 0755
http://git-wip-us.apache.org/repos/asf/metron/blob/082b218f/metron-deployment/roles/metron-rpms/tasks/create_repo.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/metron-rpms/tasks/create_repo.yml b/metron-deployment/roles/metron-rpms/tasks/create_repo.yml
new file mode 100644
index 0000000..e2ab00a
--- /dev/null
+++ b/metron-deployment/roles/metron-rpms/tasks/create_repo.yml
@@ -0,0 +1,28 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+---
+- name: Install createrepo
+ package:
+ name: createrepo
+ state: installed
+
+- name: Create local repo with new packages
+ shell: "createrepo /localrepo"
+ args:
+ creates: /localrepo/repodata
+
+
http://git-wip-us.apache.org/repos/asf/metron/blob/082b218f/metron-deployment/roles/metron-rpms/tasks/main.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/metron-rpms/tasks/main.yml b/metron-deployment/roles/metron-rpms/tasks/main.yml
index 8ea3088..9ebb452 100644
--- a/metron-deployment/roles/metron-rpms/tasks/main.yml
+++ b/metron-deployment/roles/metron-rpms/tasks/main.yml
@@ -16,33 +16,6 @@
#
---
-- name: Clean out localrepo
- file:
- state: absent
- path: /localrepo
-
-- name: Create localrepo directory
- file:
- path: /localrepo
- state: directory
- mode: 0755
-
-- name: Copy Metron RPMs
- copy:
- src: "{{ item }}"
- dest: /localrepo
- owner: root
- mode: 0755
- with_fileglob:
- - "{{ metron_rpm_glob }}"
-
-- name: Install createrepo
- package:
- name: createrepo
- state: installed
-
-- name: Create local repo with new packages
- shell: "createrepo /localrepo"
- args:
- creates: /localrepo/repodata
-
+- include: create_directory.yml
+- include: copy_rpms.yml
+- include: create_repo.yml
[06/44] metron git commit: METRON-903 Create a connections report in
Zeppelin (justinleet) closes apache/incubator-metron#556
Posted by ma...@apache.org.
METRON-903 Create a connections report in Zeppelin (justinleet) closes apache/incubator-metron#556
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/38d26d43
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/38d26d43
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/38d26d43
Branch: refs/heads/Metron_0.4.0
Commit: 38d26d4328eee058b3515dae38e02565a4669f4e
Parents: 1d27a32
Author: justinleet <ju...@gmail.com>
Authored: Thu May 4 07:55:52 2017 -0400
Committer: leet <le...@apache.org>
Committed: Thu May 4 07:55:52 2017 -0400
----------------------------------------------------------------------
metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec | 3 +++
.../src/main/config/zeppelin/metron/metron-connection-report.json | 1 +
2 files changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/38d26d43/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index c435c6c..eb30cac 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -264,6 +264,7 @@ This package installs the Metron Indexing files
%{metron_home}/config/zookeeper/indexing/asa.json
%{metron_home}/config/zookeeper/indexing/error.json
%{metron_home}/config/zeppelin/metron/metron-yaf-telemetry.json
+%{metron_home}/config/zeppelin/metron/metron-connection-report.json
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -365,6 +366,8 @@ This package installs the Metron Management UI %{metron_home}
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
%changelog
+* Fri Apr 28 2017 Apache Metron <de...@metron.apache.org> - 0.4.0
+- Add Zeppelin Connection Report Dashboard
* Thu Jan 19 2017 Justin Leet <ju...@gmail.com> - 0.3.1
- Replace GeoIP files with new implementation
* Thu Nov 03 2016 David Lyle <dl...@gmail.com> - 0.2.1
http://git-wip-us.apache.org/repos/asf/metron/blob/38d26d43/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-report.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-report.json b/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-report.json
new file mode 100644
index 0000000..0f14d5c
--- /dev/null
+++ b/metron-platform/metron-indexing/src/main/config/zeppelin/metron/metron-connection-report.json
@@ -0,0 +1 @@
+{"paragraphs":[{"text":"%spark.sql\n\n#\n# load the Yaf telemetry that has been archived by Metron\n#\ncreate temporary table yaf\n using org.apache.spark.sql.json\n options (path \"hdfs:///apps/metron/indexing/indexed/yaf\")","dateUpdated":"2017-05-02T00:17:09+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038892_607547570","id":"20170428-112507_2084067902","result":{"code":"SUCCESS","type":"TEXT","msg":""},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:09+0000","dateFinished":"2017-05-02T00:17:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:109","focus":true},{"text":"%spark.sql\n\n#\n# load the Bro telemetry that has been archived by Metron\n#\ncreate temporary table bro\n using org.apache.spark.sql.json\n options (path \"hd
fs:///apps/metron/indexing/indexed/bro\")","dateUpdated":"2017-05-02T00:17:09+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038899_618705289","id":"20170428-112518_1452220159","result":{"code":"SUCCESS","type":"TEXT","msg":""},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:09+0000","dateFinished":"2017-05-02T00:17:10+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:110","focus":true},{"text":"%spark.sql\n\n#\n# load the Snort telemetry that has been archived by Metron\n#\ncreate temporary table snort\n using org.apache.spark.sql.json\n options (path \"hdfs:///apps/metron/indexing/indexed/snort\")","dateUpdated":"2017-05-02T00:17:09+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":
[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038899_618705289","id":"20170428-112536_146360703","result":{"code":"SUCCESS","type":"TEXT","msg":""},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:10+0000","dateFinished":"2017-05-02T00:17:11+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:111","focus":true},{"text":"%md\n\n### Top Connections - Yaf\n\nThe number of connections made between IPs, ordered from highest to lowest.\n\nThis may be filtered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html).\nThe default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T11:39:56+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"grou
ps":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038902_617551042","id":"20170428-124415_505253984","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Top Connections - Yaf</h3>\n<p>The number of connections made between IPs, ordered from highest to lowest.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:09+0000","dateFinished":"2017-05-02T00:17:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:112"},{"text":"%spark.sql\n\nSELECT\n ip_src_addr,\n ip_dst_addr,\n COUNT(*) AS count\nFROM\n yaf\nWHERE timestamp BETWEEN\n (unix_timestamp(CASE WHEN '${start}' = '' then '1900-01-01 00:00:00' else '
${start}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000) AND\n (unix_timestamp(CASE WHEN '${end}' = '' then from_unixtime(unix_timestamp()) else '${end}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000)\nGROUP BY ip_src_addr, ip_dst_addr\nORDER BY COUNT(*) DESC\n\n","dateUpdated":"2017-05-02T00:17:09+0000","config":{"colWidth":12,"editorMode":"ace/mode/sql","graph":{"mode":"table","height":300,"optionOpen":false,"keys":[{"name":"ip_src_addr","index":0,"aggr":"sum"}],"values":[{"name":"ip_dst_addr","index":1,"aggr":"sum"}],"groups":[],"scatter":{"xAxis":{"name":"ip_src_addr","index":0,"aggr":"sum"},"yAxis":{"name":"ip_dst_addr","index":1,"aggr":"sum"}}},"enabled":true},"settings":{"params":{"start":"","end":"","date_format":"yyyy-MM-dd HH:mm:ss","yyyy-MM-dd HH:mm:ss":""},"forms":{"start":{"name":"start","defaultValue":"","hidden":false},"date_format":{"name":"date_format","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"end":{"name":"end","defaultValue":"","hidden":fals
e}}},"jobName":"paragraph_1493670038902_617551042","id":"20170428-112604_1206608049","result":{"code":"SUCCESS","type":"TABLE","msg":"ip_src_addr\tip_dst_addr\tcount\n62.75.195.236\t192.168.138.158\t210\n192.168.66.1\t192.168.66.121\t182\n192.168.66.121\t192.168.66.1\t131\n192.168.138.158\t62.75.195.236\t112\n192.168.138.158\t95.163.121.204\t37\n72.34.49.86\t192.168.138.158\t37\n192.168.138.158\t72.34.49.86\t28\n95.163.121.204\t192.168.138.158\t27\n192.168.138.158\t204.152.254.221\t16\n204.152.254.221\t192.168.138.158\t14\n192.168.138.158\t188.165.164.184\t7\n192.168.138.158\t192.168.138.2\t6\n192.168.138.2\t192.168.138.158\t4\n188.165.164.184\t192.168.138.158\t3\n192.168.66.1\t224.0.0.251\t1\n","comment":"","msgTable":[[{"key":"ip_dst_addr","value":"62.75.195.236"},{"key":"ip_dst_addr","value":"192.168.138.158"},{"key":"ip_dst_addr","value":"210"}],[{"key":"count","value":"192.168.66.1"},{"key":"count","value":"192.168.66.121"},{"key":"count","value":"182"}],[{"value":"192.168.66.1
21"},{"value":"192.168.66.1"},{"value":"131"}],[{"value":"192.168.138.158"},{"value":"62.75.195.236"},{"value":"112"}],[{"value":"192.168.138.158"},{"value":"95.163.121.204"},{"value":"37"}],[{"value":"72.34.49.86"},{"value":"192.168.138.158"},{"value":"37"}],[{"value":"192.168.138.158"},{"value":"72.34.49.86"},{"value":"28"}],[{"value":"95.163.121.204"},{"value":"192.168.138.158"},{"value":"27"}],[{"value":"192.168.138.158"},{"value":"204.152.254.221"},{"value":"16"}],[{"value":"204.152.254.221"},{"value":"192.168.138.158"},{"value":"14"}],[{"value":"192.168.138.158"},{"value":"188.165.164.184"},{"value":"7"}],[{"value":"192.168.138.158"},{"value":"192.168.138.2"},{"value":"6"}],[{"value":"192.168.138.2"},{"value":"192.168.138.158"},{"value":"4"}],[{"value":"188.165.164.184"},{"value":"192.168.138.158"},{"value":"3"}],[{"value":"192.168.66.1"},{"value":"224.0.0.251"},{"value":"1"}]],"columnNames":[{"name":"ip_src_addr","index":0,"aggr":"sum"},{"name":"ip_dst_addr","index":1,"aggr":
"sum"},{"name":"count","index":2,"aggr":"sum"}],"rows":[["62.75.195.236","192.168.138.158","210"],["192.168.66.1","192.168.66.121","182"],["192.168.66.121","192.168.66.1","131"],["192.168.138.158","62.75.195.236","112"],["192.168.138.158","95.163.121.204","37"],["72.34.49.86","192.168.138.158","37"],["192.168.138.158","72.34.49.86","28"],["95.163.121.204","192.168.138.158","27"],["192.168.138.158","204.152.254.221","16"],["204.152.254.221","192.168.138.158","14"],["192.168.138.158","188.165.164.184","7"],["192.168.138.158","192.168.138.2","6"],["192.168.138.2","192.168.138.158","4"],["188.165.164.184","192.168.138.158","3"],["192.168.66.1","224.0.0.251","1"]]},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:11+0000","dateFinished":"2017-05-02T00:17:14+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:113"},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}
},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493684751112_1504866931","id":"20170502-002551_1551961648","dateCreated":"2017-05-02T00:25:51+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:2780","dateUpdated":"2017-05-02T11:20:23+0000","dateFinished":"2017-05-02T11:19:43+0000","dateStarted":"2017-05-02T11:19:43+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Connections Histogram - Yaf</h3>\n<p>A histogram of connections made between IPs, binned into groups of time with a configurable lookback.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"},"text":"%md\n\n### Connections Histogram - Yaf\n\nA histogram of connections made between IPs, binned into configurable groups of time.\n\nT
his may be filtered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\nThe default format is yyyy-MM-dd HH:mm:ss"},{"text":"%spark\nimport org.apache.spark.sql.Row\nimport scala.concurrent.duration._\nimport java.util.concurrent.TimeUnit\n\nval timeunits = Seq((\"SECONDS\", \"Seconds\"), (\"MINUTES\", \"Minutes\"), (\"HOURS\", \"Hours\"), (\"DAYS\", \"Days\"))\nval sourceIp = z.input(\"SourceIp\").toString\nval destIp = z.input(\"DestIp\").toString\nval start = z.input(\"start\").toString\nval end = z.input(\"end\").toString\nval date_format = z.input(\"date_format\", \"yyyy-MM-dd HH:mm:ss\")\nval durationAmount = z.input(\"BinSize\", \"5\").toString.toInt\nval durationUnit = z.select(\"BinUnit\", \"MINUTES\", timeunits).toString\nval durationSize = Duration.create(durationAmount, TimeUnit.valueOf(durationUnit)).toMillis\n\nval results = sqlContext.sql(\ns\"\
"\"SELECT\n CONCAT(from_unixtime(($durationSize*FLOOR(timestamp/$durationSize))/1000)) AS time,\n COUNT(*) AS count\nFROM\n yaf\nWHERE\n ip_src_addr = '$sourceIp' AND\n ip_dst_addr = '$destIp' AND\n timestamp BETWEEN\n (unix_timestamp(CASE WHEN '$start' = '' then '1900-01-01 00:00:00' else '$start' END, '$date_format') * 1000) AND\n (unix_timestamp(CASE WHEN '$end' = '' then from_unixtime(unix_timestamp()) else '$end' END, '$date_format') * 1000)\nGROUP BY FLOOR(timestamp/$durationSize)\n\"\"\").map {\n case Row(time: String, count: Long) => {\n\t\ttime + \"\\t\" + count\n }\n }.collect()\n\nprint(\"%table time\\tcount\\n\" + results.mkString(\"\\n\"))","dateUpdated":"2017-05-02T12:53:25+0000","config":{"colWidth":12,"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"keys":[{"name":"time","index":0,"aggr":"sum"}],"values":[{"name":"count","index":1,"aggr":"sum"}],"groups":[],"scatter":{"yAxis":{"name":"count","index":1,"aggr":"sum"}}},"enab
led":true,"editorMode":"ace/mode/scala","tableHide":false,"editorHide":false},"settings":{"params":{"SourceIp":"62.75.195.236","DestIp":"192.168.138.158","LookBackAmount":"24","LookBackUnit":"MINUTES","BinSize":"1","BinUnit":"MINUTES","start":"","end":"","date_format":"yyyy-MM-dd HH:mm:ss"},"forms":{"SourceIp":{"name":"SourceIp","displayName":"SourceIp","type":"input","defaultValue":"","hidden":false},"DestIp":{"name":"DestIp","displayName":"DestIp","type":"input","defaultValue":"","hidden":false},"start":{"name":"start","displayName":"start","type":"input","defaultValue":"","hidden":false},"end":{"name":"end","displayName":"end","type":"input","defaultValue":"","hidden":false},"date_format":{"name":"date_format","displayName":"date_format","type":"input","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"BinSize":{"name":"BinSize","displayName":"BinSize","type":"input","defaultValue":5,"hidden":false},"BinUnit":{"name":"BinUnit","displayName":"BinUnit","type":"select","defaultVa
lue":"MINUTES","options":[{"value":"SECONDS","displayName":"Seconds","$$hashKey":"object:4090"},{"value":"MINUTES","displayName":"Minutes","$$hashKey":"object:4091"},{"value":"HOURS","displayName":"Hours","$$hashKey":"object:4092"},{"value":"DAYS","displayName":"Days","$$hashKey":"object:4093"}],"hidden":false}}},"jobName":"paragraph_1493672984960_2113296590","id":"20170501-210944_251279181","result":{"code":"SUCCESS","type":"TABLE","msg":"time\tcount\n2017-05-01 20:37:00\t2\n2017-05-01 20:38:00\t7\n2017-05-01 20:39:00\t18\n2017-05-01 20:40:00\t30\n2017-05-01 20:41:00\t15\n2017-05-01 20:42:00\t30\n2017-05-01 20:43:00\t27\n2017-05-01 20:44:00\t26\n2017-05-01 20:45:00\t39\n2017-05-01 20:46:00\t16","comment":"","msgTable":[[{"key":"count","value":"2017-05-01 20:37:00"},{"key":"count","value":"2"}],[{"value":"2017-05-01 20:38:00"},{"value":"7"}],[{"value":"2017-05-01 20:39:00"},{"value":"18"}],[{"value":"2017-05-01 20:40:00"},{"value":"30"}],[{"value":"2017-05-01 20:41:00"},{"value":"15
"}],[{"value":"2017-05-01 20:42:00"},{"value":"30"}],[{"value":"2017-05-01 20:43:00"},{"value":"27"}],[{"value":"2017-05-01 20:44:00"},{"value":"26"}],[{"value":"2017-05-01 20:45:00"},{"value":"39"}],[{"value":"2017-05-01 20:46:00"},{"value":"16"}]],"columnNames":[{"name":"time","index":0,"aggr":"sum"},{"name":"count","index":1,"aggr":"sum"}],"rows":[["2017-05-01 20:37:00","2"],["2017-05-01 20:38:00","7"],["2017-05-01 20:39:00","18"],["2017-05-01 20:40:00","30"],["2017-05-01 20:41:00","15"],["2017-05-01 20:42:00","30"],["2017-05-01 20:43:00","27"],["2017-05-01 20:44:00","26"],["2017-05-01 20:45:00","39"],["2017-05-01 20:46:00","16"]]},"dateCreated":"2017-05-01T09:09:44+0000","dateStarted":"2017-05-02T11:19:39+0000","dateFinished":"2017-05-02T11:19:47+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:114","focus":true},{"text":"%md\n\n### Top Requests - Bro DNS\n\nThe number of DNS requests made between IPs, ordered from highest to lowest.\n\nThis may be fi
ltered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\nThe default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T12:59:13+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038902_617551042","id":"20170428-124500_925401848","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Top Requests - Bro DNS</h3>\n<p>The number of DNS queries made between IPs, ordered from highest to lowest.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"}
,"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:12+0000","dateFinished":"2017-05-02T00:17:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:115"},{"text":"%spark.sql\n\nSELECT\n ip_src_addr,\n ip_dst_addr,\n COUNT(*) AS count\nFROM\n bro\nWHERE timestamp BETWEEN\n (unix_timestamp(CASE WHEN '${start}' = '' then '1900-01-01 00:00:00' else '${start}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000) AND\n (unix_timestamp(CASE WHEN '${end}' = '' then from_unixtime(unix_timestamp()) else '${end}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000) AND\n protocol = 'dns'\nGROUP BY ip_src_addr, ip_dst_addr\nORDER BY COUNT(*) DESC\n \n ","dateUpdated":"2017-05-02T00:17:10+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[{"name":"ip_src_addr","index":0,"aggr":"sum"}],"values":[{"name":"ip_dst_addr","index":1,"aggr":"sum"}],"groups":[],"scatter":{"xAxis":{"name":"ip_
src_addr","index":0,"aggr":"sum"},"yAxis":{"name":"ip_dst_addr","index":1,"aggr":"sum"}}},"enabled":true,"editorMode":"ace/mode/sql"},"settings":{"params":{"start":"","end":"","date_format":"yyyy-MM-dd HH:mm:ss","yyyy-MM-dd HH:mm:ss":""},"forms":{"start":{"name":"start","defaultValue":"","hidden":false},"date_format":{"name":"date_format","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"end":{"name":"end","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493670038902_617551042","id":"20170428-114040_87496728","result":{"code":"SUCCESS","type":"TABLE","msg":"ip_src_addr\tip_dst_addr\tcount\n192.168.138.158\t192.168.138.2\t78\n192.168.66.1\t224.0.0.251\t53\n","comment":"","msgTable":[[{"key":"ip_dst_addr","value":"192.168.138.158"},{"key":"ip_dst_addr","value":"192.168.138.2"},{"key":"ip_dst_addr","value":"78"}],[{"key":"count","value":"192.168.66.1"},{"key":"count","value":"224.0.0.251"},{"key":"count","value":"53"}]],"columnNames":[{"name":"ip_src_addr","index":0,"aggr"
:"sum"},{"name":"ip_dst_addr","index":1,"aggr":"sum"},{"name":"count","index":2,"aggr":"sum"}],"rows":[["192.168.138.158","192.168.138.2","78"],["192.168.66.1","224.0.0.251","53"]]},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:14+0000","dateFinished":"2017-05-02T00:17:20+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:116"},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493729863774_-980328101","id":"20170502-125743_777301519","dateCreated":"2017-05-02T12:57:43+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:4571","text":"%md\n\n### Requests Histogram - Bro DNS\n\nA histogram of DNS requests made between IPs, binned into configurable groups of time.\n\nThis may be filtered by a providing a start and end
filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\nThe default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T12:59:10+0000","dateFinished":"2017-05-02T12:59:10+0000","dateStarted":"2017-05-02T12:59:10+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Requests Histogram - Bro DNS</h3>\n<p>A histogram of DNS requests made between IPs, binned into configurable groups of time.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"}},{"config":{"colWidth":12,"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"keys":[{"name":"time","index":0,"aggr":"sum"}],"values":[{"name":"count","index":1,"aggr":"sum"}],"groups":[],"scatter":{"xAxis":{"name":"t
ime","index":0,"aggr":"sum"},"yAxis":{"name":"count","index":1,"aggr":"sum"}}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{"SourceIp":"192.168.138.158","DestIp":"192.168.138.2","start":"","end":"","date_format":"yyyy-MM-dd HH:mm:ss","BinSize":"1","BinUnit":"MINUTES"},"forms":{"SourceIp":{"name":"SourceIp","displayName":"SourceIp","type":"input","defaultValue":"","hidden":false},"DestIp":{"name":"DestIp","displayName":"DestIp","type":"input","defaultValue":"","hidden":false},"start":{"name":"start","displayName":"start","type":"input","defaultValue":"","hidden":false},"end":{"name":"end","displayName":"end","type":"input","defaultValue":"","hidden":false},"date_format":{"name":"date_format","displayName":"date_format","type":"input","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"BinSize":{"name":"BinSize","displayName":"BinSize","type":"input","defaultValue":"5","hidden":false},"BinUnit":{"name":"BinUnit","displayName":"BinUnit","type":"select","default
Value":"MINUTES","options":[{"value":"SECONDS","displayName":"Seconds","$$hashKey":"object:4133"},{"value":"MINUTES","displayName":"Minutes","$$hashKey":"object:4134"},{"value":"HOURS","displayName":"Hours","$$hashKey":"object:4135"},{"value":"DAYS","displayName":"Days","$$hashKey":"object:4136"}],"hidden":false}}},"jobName":"paragraph_1493729434074_-1929699562","id":"20170502-125034_1944008091","dateCreated":"2017-05-02T12:50:34+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:3683","text":"%spark\nimport org.apache.spark.sql.Row\nimport scala.concurrent.duration._\nimport java.util.concurrent.TimeUnit\n\nval timeunits = Seq((\"SECONDS\", \"Seconds\"), (\"MINUTES\", \"Minutes\"), (\"HOURS\", \"Hours\"), (\"DAYS\", \"Days\"))\nval sourceIp = z.input(\"SourceIp\").toString\nval destIp = z.input(\"DestIp\").toString\nval start = z.input(\"start\").toString\nval end = z.input(\"end\").toString\nval date_format = z.input(\"date_format\", \"yyyy-M
M-dd HH:mm:ss\")\nval durationAmount = z.input(\"BinSize\", \"5\").toString.toInt\nval durationUnit = z.select(\"BinUnit\", \"MINUTES\", timeunits).toString\nval durationSize = Duration.create(durationAmount, TimeUnit.valueOf(durationUnit)).toMillis\n\nval results = sqlContext.sql(\ns\"\"\"SELECT\n CONCAT(from_unixtime(($durationSize*FLOOR(timestamp/$durationSize))/1000)) AS time,\n COUNT(*) AS count\nFROM\n bro\nWHERE\n ip_src_addr = '$sourceIp' AND\n ip_dst_addr = '$destIp' AND\n timestamp BETWEEN\n (unix_timestamp(CASE WHEN '$start' = '' then '1900-01-01 00:00:00' else '$start' END, '$date_format') * 1000) AND\n (unix_timestamp(CASE WHEN '$end' = '' then from_unixtime(unix_timestamp()) else '$end' END, '$date_format') * 1000) AND\n protocol = 'dns'\nGROUP BY FLOOR(timestamp/$durationSize)\n\"\"\").map {\n case Row(time: String, count: Long) => {\n\t\ttime + \"\\t\" + count\n }\n }.collect()\n\nprint(\"%table time\\tcount\\n\" + results.mkString(\"\
\n\"))","dateUpdated":"2017-05-02T12:53:24+0000","dateFinished":"2017-05-02T12:53:33+0000","dateStarted":"2017-05-02T12:53:24+0000","result":{"code":"SUCCESS","type":"TABLE","msg":"time\tcount\n2017-05-01 20:02:00\t4\n2017-05-01 20:03:00\t15\n2017-05-01 20:04:00\t21\n2017-05-01 20:05:00\t18\n2017-05-01 20:06:00\t17\n2017-05-01 20:07:00\t3","comment":"","msgTable":[[{"key":"count","value":"2017-05-01 20:02:00"},{"key":"count","value":"4"}],[{"value":"2017-05-01 20:03:00"},{"value":"15"}],[{"value":"2017-05-01 20:04:00"},{"value":"21"}],[{"value":"2017-05-01 20:05:00"},{"value":"18"}],[{"value":"2017-05-01 20:06:00"},{"value":"17"}],[{"value":"2017-05-01 20:07:00"},{"value":"3"}]],"columnNames":[{"name":"time","index":0,"aggr":"sum"},{"name":"count","index":1,"aggr":"sum"}],"rows":[["2017-05-01 20:02:00","4"],["2017-05-01 20:03:00","15"],["2017-05-01 20:04:00","21"],["2017-05-01 20:05:00","18"],["2017-05-01 20:06:00","17"],["2017-05-01 20:07:00","3"]]}},{"text":"%md\n\n### Top Request
s - Bro HTTP\n\nThe number of HTTP requests made between IPs, ordered from highest to lowest.\n\nThis may be filtered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\nThe default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T00:17:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","graph":{"mode":"table","height":86,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038900_616781544","id":"20170428-123858_1869250606","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Top Requests - Bro HTTP</h3>\n<p>The number of HTTP requests made between IPs, ordered from highest to lowest.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/f
ormat/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:12+0000","dateFinished":"2017-05-02T00:17:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:117"},{"text":"%spark.sql\n\nSELECT\n ip_src_addr,\n ip_dst_addr,\n COUNT(*) AS count\nFROM\n bro\nWHERE timestamp BETWEEN\n (unix_timestamp(CASE WHEN '${start}' = '' then '1900-01-01 00:00:00' else '${start}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000) AND\n (unix_timestamp(CASE WHEN '${end}' = '' then from_unixtime(unix_timestamp()) else '${end}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000) AND\n protocol = 'http'\nGROUP BY ip_src_addr, ip_dst_addr\nORDER BY COUNT(*) DESC\n","dateUpdated":"2017-05-02T00:17:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","graph":{"mode":"table","height":278,"optionOpen":false,"keys":[{"name":"ip_src
_addr","index":0,"aggr":"sum"}],"values":[{"name":"ip_dst_addr","index":1,"aggr":"sum"}],"groups":[],"scatter":{"xAxis":{"name":"ip_src_addr","index":0,"aggr":"sum"},"yAxis":{"name":"ip_dst_addr","index":1,"aggr":"sum"}}},"enabled":true},"settings":{"params":{"start":"","date_format":"yyyy-MM-dd HH:mm:ss","end":""},"forms":{"start":{"name":"start","defaultValue":"","hidden":false},"date_format":{"name":"date_format","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"end":{"name":"end","defaultValue":"","hidden":false}}},"jobName":"paragraph_1493670038900_616781544","id":"20170428-123208_869982701","result":{"code":"SUCCESS","type":"TABLE","msg":"ip_src_addr\tip_dst_addr\tcount\n192.168.138.158\t95.163.121.204\t208\n192.168.66.1\t192.168.66.121\t147\n192.168.138.158\t62.75.195.236\t107\n192.168.138.158\t72.34.49.86\t44\n192.168.138.158\t204.152.254.221\t36\n192.168.138.158\t188.165.164.184\t17\n","comment":"","msgTable":[[{"key":"ip_dst_addr","value":"192.168.138.158"},{"key":"ip_
dst_addr","value":"95.163.121.204"},{"key":"ip_dst_addr","value":"208"}],[{"key":"count","value":"192.168.66.1"},{"key":"count","value":"192.168.66.121"},{"key":"count","value":"147"}],[{"value":"192.168.138.158"},{"value":"62.75.195.236"},{"value":"107"}],[{"value":"192.168.138.158"},{"value":"72.34.49.86"},{"value":"44"}],[{"value":"192.168.138.158"},{"value":"204.152.254.221"},{"value":"36"}],[{"value":"192.168.138.158"},{"value":"188.165.164.184"},{"value":"17"}]],"columnNames":[{"name":"ip_src_addr","index":0,"aggr":"sum"},{"name":"ip_dst_addr","index":1,"aggr":"sum"},{"name":"count","index":2,"aggr":"sum"}],"rows":[["192.168.138.158","95.163.121.204","208"],["192.168.66.1","192.168.66.121","147"],["192.168.138.158","62.75.195.236","107"],["192.168.138.158","72.34.49.86","44"],["192.168.138.158","204.152.254.221","36"],["192.168.138.158","188.165.164.184","17"]]},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:19+0000","dateFinished":"2017-05-02T00:17:2
1+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:118"},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493729961028_741593438","id":"20170502-125921_2097846713","dateCreated":"2017-05-02T12:59:21+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:4649","text":"%md\n\n### Requests Histogram - Bro HTTP\n\nA histogram of HTTP requests made between IPs, binned into configurable groups of time.\n\nThis may be filtered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\nThe default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T13:00:10+0000","dateFinished":"2017-05-02T13:00:10+0000","dateStarted":"2017-05-
02T13:00:10+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Requests Histogram - Bro HTTP</h3>\n<p>A histogram of HTTP requests made between IPs, binned into configurable groups of time.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"}},{"config":{"colWidth":12,"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"keys":[{"name":"time","index":0,"aggr":"sum"}],"values":[{"name":"count","index":1,"aggr":"sum"}],"groups":[],"scatter":{"xAxis":{"name":"time","index":0,"aggr":"sum"},"yAxis":{"name":"count","index":1,"aggr":"sum"}}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{"SourceIp":"192.168.138.158","DestIp":"95.163.121.204","start":"","end":"","date_format":"yyyy-MM-dd HH:mm:ss","BinSize":"1","BinUnit":"MINUTES"},"
forms":{"SourceIp":{"name":"SourceIp","displayName":"SourceIp","type":"input","defaultValue":"","hidden":false},"DestIp":{"name":"DestIp","displayName":"DestIp","type":"input","defaultValue":"","hidden":false},"start":{"name":"start","displayName":"start","type":"input","defaultValue":"","hidden":false},"end":{"name":"end","displayName":"end","type":"input","defaultValue":"","hidden":false},"date_format":{"name":"date_format","displayName":"date_format","type":"input","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"BinSize":{"name":"BinSize","displayName":"BinSize","type":"input","defaultValue":"5","hidden":false},"BinUnit":{"name":"BinUnit","displayName":"BinUnit","type":"select","defaultValue":"MINUTES","options":[{"value":"SECONDS","displayName":"Seconds","$$hashKey":"object:4331"},{"value":"MINUTES","displayName":"Minutes","$$hashKey":"object:4332"},{"value":"HOURS","displayName":"Hours","$$hashKey":"object:4333"},{"value":"DAYS","displayName":"Days","$$hashKey":"object:43
34"}],"hidden":false}}},"jobName":"paragraph_1493729623175_-771160972","id":"20170502-125343_190565024","dateCreated":"2017-05-02T12:53:43+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:4137","text":"%spark\nimport org.apache.spark.sql.Row\nimport scala.concurrent.duration._\nimport java.util.concurrent.TimeUnit\n\nval timeunits = Seq((\"SECONDS\", \"Seconds\"), (\"MINUTES\", \"Minutes\"), (\"HOURS\", \"Hours\"), (\"DAYS\", \"Days\"))\nval sourceIp = z.input(\"SourceIp\").toString\nval destIp = z.input(\"DestIp\").toString\nval start = z.input(\"start\").toString\nval end = z.input(\"end\").toString\nval date_format = z.input(\"date_format\", \"yyyy-MM-dd HH:mm:ss\")\nval durationAmount = z.input(\"BinSize\", \"5\").toString.toInt\nval durationUnit = z.select(\"BinUnit\", \"MINUTES\", timeunits).toString\nval durationSize = Duration.create(durationAmount, TimeUnit.valueOf(durationUnit)).toMillis\n\nval results = sqlContext.sql(\ns\"\"\"SELE
CT\n CONCAT(from_unixtime(($durationSize*FLOOR(timestamp/$durationSize))/1000)) AS time,\n COUNT(*) AS count\nFROM\n bro\nWHERE\n ip_src_addr = '$sourceIp' AND\n ip_dst_addr = '$destIp' AND\n timestamp BETWEEN\n (unix_timestamp(CASE WHEN '$start' = '' then '1900-01-01 00:00:00' else '$start' END, '$date_format') * 1000) AND\n (unix_timestamp(CASE WHEN '$end' = '' then from_unixtime(unix_timestamp()) else '$end' END, '$date_format') * 1000) AND\n protocol = 'http'\nGROUP BY FLOOR(timestamp/$durationSize)\n\"\"\").map {\n case Row(time: String, count: Long) => {\n\t\ttime + \"\\t\" + count\n }\n }.collect()\n\nprint(\"%table time\\tcount\\n\" + results.mkString(\"\\n\"))","dateUpdated":"2017-05-02T12:54:35+0000","dateFinished":"2017-05-02T12:54:31+0000","dateStarted":"2017-05-02T12:54:22+0000","result":{"code":"SUCCESS","type":"TABLE","msg":"time\tcount\n2017-05-01 20:02:00\t10\n2017-05-01 20:03:00\t39\n2017-05-01 20:04:00\t48\n2017-05-01 20:05:00\t58\n
2017-05-01 20:06:00\t43\n2017-05-01 20:07:00\t10","comment":"","msgTable":[[{"key":"count","value":"2017-05-01 20:02:00"},{"key":"count","value":"10"}],[{"value":"2017-05-01 20:03:00"},{"value":"39"}],[{"value":"2017-05-01 20:04:00"},{"value":"48"}],[{"value":"2017-05-01 20:05:00"},{"value":"58"}],[{"value":"2017-05-01 20:06:00"},{"value":"43"}],[{"value":"2017-05-01 20:07:00"},{"value":"10"}]],"columnNames":[{"name":"time","index":0,"aggr":"sum"},{"name":"count","index":1,"aggr":"sum"}],"rows":[["2017-05-01 20:02:00","10"],["2017-05-01 20:03:00","39"],["2017-05-01 20:04:00","48"],["2017-05-01 20:05:00","58"],["2017-05-01 20:06:00","43"],["2017-05-01 20:07:00","10"]]}},{"text":"%md\n\n### Top Alerts - Snort\n\nThe number of alerts triggered between IPs, ordered from highest to lowest.\n\nThis may be filtered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\n
The default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T00:17:10+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/markdown"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038903_617166293","id":"20170428-124511_1307410313","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Top Alerts - Snort</h3>\n<p>The number of alerts triggered between IPs, ordered from highest to lowest.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:12+0000","dateFinished":"2017-05-02T00:17:12+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"
object:119"},{"text":"%spark.sql\n\nSELECT\n ip_src_addr,\n ip_dst_addr,\n COUNT(*) AS count\nFROM\n snort\nWHERE timestamp BETWEEN\n (unix_timestamp(CASE WHEN '${start}' = '' then '1900-01-01 00:00:00' else '${start}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000) AND\n (unix_timestamp(CASE WHEN '${end}' = '' then from_unixtime(unix_timestamp()) else '${end}' END, '${date_format=yyyy-MM-dd HH:mm:ss}') * 1000)\nGROUP BY ip_src_addr, ip_dst_addr\nORDER BY COUNT(*) DESC\n ","dateUpdated":"2017-05-02T00:17:10+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true},"settings":{"params":{"date_format":"yyyy-MM-dd HH:mm:ss","start":"","end":""},"forms":{"start":{"name":"start","defaultValue":"","hidden":false},"date_format":{"name":"date_format","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"end":{"name":"end","defaultValue":"","hidden
":false}}},"jobName":"paragraph_1493670038903_617166293","id":"20170428-123118_1636321684","result":{"code":"SUCCESS","type":"TABLE","msg":"ip_src_addr\tip_dst_addr\tcount\n192.168.66.1\t192.168.66.121\t227\n62.75.195.236\t192.168.138.158\t174\n192.168.138.158\t62.75.195.236\t81\n192.168.66.121\t192.168.66.1\t60\n192.168.138.158\t95.163.121.204\t30\n72.34.49.86\t192.168.138.158\t29\n95.163.121.204\t192.168.138.158\t26\n192.168.138.158\t72.34.49.86\t25\n192.168.138.158\t204.152.254.221\t14\n204.152.254.221\t192.168.138.158\t14\n","comment":"","msgTable":[[{"key":"ip_dst_addr","value":"192.168.66.1"},{"key":"ip_dst_addr","value":"192.168.66.121"},{"key":"ip_dst_addr","value":"227"}],[{"key":"count","value":"62.75.195.236"},{"key":"count","value":"192.168.138.158"},{"key":"count","value":"174"}],[{"value":"192.168.138.158"},{"value":"62.75.195.236"},{"value":"81"}],[{"value":"192.168.66.121"},{"value":"192.168.66.1"},{"value":"60"}],[{"value":"192.168.138.158"},{"value":"95.163.121.204
"},{"value":"30"}],[{"value":"72.34.49.86"},{"value":"192.168.138.158"},{"value":"29"}],[{"value":"95.163.121.204"},{"value":"192.168.138.158"},{"value":"26"}],[{"value":"192.168.138.158"},{"value":"72.34.49.86"},{"value":"25"}],[{"value":"192.168.138.158"},{"value":"204.152.254.221"},{"value":"14"}],[{"value":"204.152.254.221"},{"value":"192.168.138.158"},{"value":"14"}]],"columnNames":[{"name":"ip_src_addr","index":0,"aggr":"sum"},{"name":"ip_dst_addr","index":1,"aggr":"sum"},{"name":"count","index":2,"aggr":"sum"}],"rows":[["192.168.66.1","192.168.66.121","227"],["62.75.195.236","192.168.138.158","174"],["192.168.138.158","62.75.195.236","81"],["192.168.66.121","192.168.66.1","60"],["192.168.138.158","95.163.121.204","30"],["72.34.49.86","192.168.138.158","29"],["95.163.121.204","192.168.138.158","26"],["192.168.138.158","72.34.49.86","25"],["192.168.138.158","204.152.254.221","14"],["204.152.254.221","192.168.138.158","14"]]},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted
":"2017-05-02T00:17:21+0000","dateFinished":"2017-05-02T00:17:23+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:120"},{"config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493729982351_281449077","id":"20170502-125942_58259658","dateCreated":"2017-05-02T12:59:42+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:4719","text":"%md\n\n### Alerts Histogram - Snort\n\nA histogram of alerts triggered between IPs, binned into configurable groups of time.\n\nThis may be filtered by a providing a start and end filter, along with a date_format as specified by [Customizing Formats](http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html)\nThe default format is yyyy-MM-dd HH:mm:ss","dateUpdated":"2017-05-02T13:00:05+0000","dateFinish
ed":"2017-05-02T13:00:05+0000","dateStarted":"2017-05-02T13:00:05+0000","result":{"code":"SUCCESS","type":"HTML","msg":"<h3>Alerts Histogram - Snort</h3>\n<p>A histogram of alerts triggered between IPs, binned into configurable groups of time.</p>\n<p>This may be filtered by a providing a start and end filter, along with a date_format as specified by <a href=\"http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html\">Customizing Formats</a>\n<br />The default format is yyyy-MM-dd HH:mm:ss</p>\n"}},{"config":{"colWidth":12,"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"keys":[{"name":"time","index":0,"aggr":"sum"}],"values":[{"name":"count","index":1,"aggr":"sum"}],"groups":[],"scatter":{"xAxis":{"name":"time","index":0,"aggr":"sum"},"yAxis":{"name":"count","index":1,"aggr":"sum"}}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{"SourceIp":"192.168.66.1","DestIp":"192.168.66.121","start":"","end":"","date_format":"yyyy-MM-dd HH
:mm:ss","BinSize":"1","BinUnit":"MINUTES"},"forms":{"SourceIp":{"name":"SourceIp","displayName":"SourceIp","type":"input","defaultValue":"","hidden":false},"DestIp":{"name":"DestIp","displayName":"DestIp","type":"input","defaultValue":"","hidden":false},"start":{"name":"start","displayName":"start","type":"input","defaultValue":"","hidden":false},"end":{"name":"end","displayName":"end","type":"input","defaultValue":"","hidden":false},"date_format":{"name":"date_format","displayName":"date_format","type":"input","defaultValue":"yyyy-MM-dd HH:mm:ss","hidden":false},"BinSize":{"name":"BinSize","displayName":"BinSize","type":"input","defaultValue":"5","hidden":false},"BinUnit":{"name":"BinUnit","displayName":"BinUnit","type":"select","defaultValue":"MINUTES","options":[{"value":"SECONDS","displayName":"Seconds","$$hashKey":"object:4567"},{"value":"MINUTES","displayName":"Minutes","$$hashKey":"object:4568"},{"value":"HOURS","displayName":"Hours","$$hashKey":"object:4569"},{"value":"DAYS"
,"displayName":"Days","$$hashKey":"object:4570"}],"hidden":false}}},"jobName":"paragraph_1493729702158_612719525","id":"20170502-125502_1681465417","dateCreated":"2017-05-02T12:55:02+0000","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:4366","text":"%spark\nimport org.apache.spark.sql.Row\nimport scala.concurrent.duration._\nimport java.util.concurrent.TimeUnit\n\nval timeunits = Seq((\"SECONDS\", \"Seconds\"), (\"MINUTES\", \"Minutes\"), (\"HOURS\", \"Hours\"), (\"DAYS\", \"Days\"))\nval sourceIp = z.input(\"SourceIp\").toString\nval destIp = z.input(\"DestIp\").toString\nval start = z.input(\"start\").toString\nval end = z.input(\"end\").toString\nval date_format = z.input(\"date_format\", \"yyyy-MM-dd HH:mm:ss\")\nval durationAmount = z.input(\"BinSize\", \"5\").toString.toInt\nval durationUnit = z.select(\"BinUnit\", \"MINUTES\", timeunits).toString\nval durationSize = Duration.create(durationAmount, TimeUnit.valueOf(durationUnit)).toMillis\n
\nval results = sqlContext.sql(\ns\"\"\"SELECT\n CONCAT(from_unixtime(($durationSize*FLOOR(timestamp/$durationSize))/1000)) AS time,\n COUNT(*) AS count\nFROM\n snort\nWHERE\n ip_src_addr = '$sourceIp' AND\n ip_dst_addr = '$destIp' AND\n timestamp BETWEEN\n (unix_timestamp(CASE WHEN '$start' = '' then '1900-01-01 00:00:00' else '$start' END, '$date_format') * 1000) AND\n (unix_timestamp(CASE WHEN '$end' = '' then from_unixtime(unix_timestamp()) else '$end' END, '$date_format') * 1000)\nGROUP BY FLOOR(timestamp/$durationSize)\n\"\"\").map {\n case Row(time: String, count: Long) => {\n\t\ttime + \"\\t\" + count\n }\n }.collect()\n\nprint(\"%table time\\tcount\\n\" + results.mkString(\"\\n\"))","dateUpdated":"2017-05-02T12:55:54+0000","dateFinished":"2017-05-02T12:56:04+0000","dateStarted":"2017-05-02T12:55:54+0000","result":{"code":"SUCCESS","type":"TABLE","msg":"time\tcount\n2017-05-01 20:02:00\t4\n2017-05-01 20:03:00\t27\n2017-05-01 20:04:00\t47\n2017-05
-01 20:05:00\t68\n2017-05-01 20:06:00\t56\n2017-05-01 20:07:00\t25","comment":"","msgTable":[[{"key":"count","value":"2017-05-01 20:02:00"},{"key":"count","value":"4"}],[{"value":"2017-05-01 20:03:00"},{"value":"27"}],[{"value":"2017-05-01 20:04:00"},{"value":"47"}],[{"value":"2017-05-01 20:05:00"},{"value":"68"}],[{"value":"2017-05-01 20:06:00"},{"value":"56"}],[{"value":"2017-05-01 20:07:00"},{"value":"25"}]],"columnNames":[{"name":"time","index":0,"aggr":"sum"},{"name":"count","index":1,"aggr":"sum"}],"rows":[["2017-05-01 20:02:00","4"],["2017-05-01 20:03:00","27"],["2017-05-01 20:04:00","47"],["2017-05-01 20:05:00","68"],["2017-05-01 20:06:00","56"],["2017-05-01 20:07:00","25"]]}},{"dateUpdated":"2017-05-02T00:17:10+0000","config":{"colWidth":12,"graph":{"mode":"table","height":300,"optionOpen":false,"keys":[],"values":[],"groups":[],"scatter":{}},"enabled":true,"editorMode":"ace/mode/scala"},"settings":{"params":{},"forms":{}},"jobName":"paragraph_1493670038903_617166293","id":
"20170428-112735_164601815","result":{"code":"SUCCESS","type":"TEXT","msg":""},"dateCreated":"2017-05-01T08:20:38+0000","dateStarted":"2017-05-02T00:17:22+0000","dateFinished":"2017-05-02T00:17:23+0000","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:121"}],"name":"Metron - Connection Report","id":"2CJ36HA8C","angularObjects":{"2CFJZNPTR:shared_process":[],"2CHQEKJX6:shared_process":[],"2CGXQM678:shared_process":[],"2CHVX3RDD:shared_process":[],"2CFCW5XXT:shared_process":[],"2CFQKHR9K:shared_process":[]},"config":{"looknfeel":"default"},"info":{}}
\ No newline at end of file