You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by ma...@apache.org on 2017/06/26 17:27:22 UTC
[01/18] metron git commit: METRON-979 markdown errors in site-book
(mattf-horton) closes apache/metron#605
Repository: metron
Updated Branches:
refs/heads/Metron_0.4.0 85872bd68 -> b76bcd5f2
METRON-979 markdown errors in site-book (mattf-horton) closes apache/metron#605
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e31ed2f7
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e31ed2f7
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e31ed2f7
Branch: refs/heads/Metron_0.4.0
Commit: e31ed2f7d2420fc9f68b0dbf5b6a9a5b53ac6453
Parents: 85872bd
Author: mattf-horton <mf...@hortonworks.com>
Authored: Fri Jun 2 10:37:54 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Fri Jun 2 10:37:54 2017 -0700
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 202 +++++++++----------
.../packaging/docker/ansible-docker/README.md | 6 +-
.../packaging/docker/rpm-docker/README.md | 2 +-
metron-deployment/roles/README.md | 4 +-
metron-interface/metron-config/README.md | 79 ++++----
metron-platform/metron-pcap-backend/README.md | 2 +-
metron-sensors/pycapa/README.md | 4 +-
7 files changed, 155 insertions(+), 144 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index 1540231..4efebc6 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -42,75 +42,75 @@ Setup
1. Stop all Metron topologies. They will be restarted again once Kerberos has been enabled.
- ```
- for topology in bro snort enrichment indexing; do
- storm kill $topology;
- done
- ```
+ ```
+ for topology in bro snort enrichment indexing; do
+ storm kill $topology;
+ done
+ ```
1. Create the `metron` user's home directory in HDFS.
- ```
- sudo -u hdfs hdfs dfs -mkdir /user/metron
- sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
- sudo -u hdfs hdfs dfs -chmod 770 /user/metron
- ```
+ ```
+ sudo -u hdfs hdfs dfs -mkdir /user/metron
+ sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
+ sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+ ```
Setup a KDC
-----------
1. Install dependencies.
- ```
- yum -y install krb5-server krb5-libs krb5-workstation
- ```
+ ```
+ yum -y install krb5-server krb5-libs krb5-workstation
+ ```
1. Define the host, `node1`, as the KDC.
- ```
- sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
- cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
- ```
+ ```
+ sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+ cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+ ```
1. Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section
- ```
- max_renewable_life = 7d
- ```
-
-
-1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
- ```
- kdb5_util create -s
+ ```
+ max_renewable_life = 7d
+ ```
- /etc/rc.d/init.d/krb5kdc start
- chkconfig krb5kdc on
+1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database.
- /etc/rc.d/init.d/kadmin start
- chkconfig kadmin on
- ```
+ ```
+ kdb5_util create -s
+ /etc/rc.d/init.d/krb5kdc start
+ chkconfig krb5kdc on
+ /etc/rc.d/init.d/kadmin start
+ chkconfig kadmin on
+ ```
1. Setup the `admin` and `metron` principals. You'll `kinit` as the `metron` principal when running topologies. Make sure to remember the passwords.
- ```
- kadmin.local -q "addprinc admin/admin"
- kadmin.local -q "addprinc metron"
- ```
+ ```
+ kadmin.local -q "addprinc admin/admin"
+ kadmin.local -q "addprinc metron"
+ ```
Verify KDC
----------
Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies:
- ```
- Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
- ```
+
+```
+Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable
+```
Ensure the Metron keytab is renewable. Look for the 'R' flag from the following command
- ```
- klist -f
- ```
+
+```
+klist -f
+```
If the 'R' flags are present, you may skip to next section.
@@ -118,10 +118,11 @@ If the 'R' flags are absent, you will need to follow the below steps:
If the KDC is already setup, then editing max_life and max_renewable_life in `/var/kerberos/krb5kdc/kdc.conf`, and restarting kadmin and krb5kdc services will not change the policies for existing users.
You need to set the renew lifetime for existing users and krbtgt realm. Modify the appropriate principals to allow renewable tickets using the following commands. Adjust the parameters to match your desired KDC parameters:
- ```
- kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
- kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
- ```
+
+```
+kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable krbtgt/EXAMPLE.COM@EXAMPLE.COM"
+kadmin.local -q "modprinc -maxlife 1days -maxrenewlife 7days +allow_renewable metron@EXAMPLE.COM"
+```
Enable Kerberos
@@ -164,11 +165,11 @@ Enable Kerberos
1. Create a Metron keytab
```
- kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
- cp metron.headless.keytab /etc/security/keytabs
- chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
- chmod 440 /etc/security/keytabs/metron.headless.keytab
- ```
+ kadmin.local -q "ktadd -k metron.headless.keytab metron@EXAMPLE.COM"
+ cp metron.headless.keytab /etc/security/keytabs
+ chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
+ chmod 440 /etc/security/keytabs/metron.headless.keytab
+ ```
Kafka Authorization
-------------------
@@ -176,60 +177,58 @@ Kafka Authorization
1. Acquire a Kerberos ticket using the `metron` principal.
```
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
1. Create any additional Kafka topics that you will need. We need to create the topics before adding the required ACLs. The current full dev installation will deploy bro, snort, enrichments, and indexing only. For example, you may want to add a topic for 'yaf' telemetry.
```
- ${KAFKA_HOME}/bin/kafka-topics.sh \
+ ${KAFKA_HOME}/bin/kafka-topics.sh \
--zookeeper ${ZOOKEEPER} \
--create \
--topic yaf \
--partitions 1 \
--replication-factor 1
- ```
+ ```
1. Setup Kafka ACLs for the `bro`, `snort`, `enrichments`, and `indexing` topics. Run the same command against any additional topics that you might be using; for example `yaf`.
```
- export KERB_USER=metron
-
- for topic in bro snort enrichments indexing; do
- ${KAFKA_HOME}/bin/kafka-acls.sh \
+ export KERB_USER=metron
+ for topic in bro snort enrichments indexing; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=${ZOOKEEPER} \
--add \
--allow-principal User:${KERB_USER} \
--topic ${topic}
- done
- ```
+ done
+ ```
1. Setup Kafka ACLs for the consumer groups. This command sets the ACLs for Bro, Snort, YAF, Enrichments, Indexing, and the Profiler. Execute the same command for any additional Parsers that you may be running.
```
export KERB_USER=metron
-
- for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
- ${KAFKA_HOME}/bin/kafka-acls.sh \
+ for group in bro_parser snort_parser yaf_parser enrichments indexing profiler; do
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=${ZOOKEEPER} \
--add \
--allow-principal User:${KERB_USER} \
--group ${group}
- done
- ```
+ done
+ ```
1. Add the `metron` principal to the `kafka-cluster` ACL.
```
- ${KAFKA_HOME}/bin/kafka-acls.sh \
+ ${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=${ZOOKEEPER} \
--add \
--allow-principal User:${KERB_USER} \
--cluster kafka-cluster
- ```
+ ```
HBase Authorization
-------------------
@@ -237,22 +236,22 @@ HBase Authorization
1. Acquire a Kerberos ticket using the `hbase` principal
```
- kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
- ```
+ kinit -kt /etc/security/keytabs/hbase.headless.keytab hbase-metron_cluster@EXAMPLE.COM
+ ```
1. Grant permissions for the HBase tables used in Metron.
```
- echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
- echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
- ```
+ echo "grant 'metron', 'RW', 'threatintel'" | hbase shell
+ echo "grant 'metron', 'RW', 'enrichment'" | hbase shell
+ ```
1. If you are using the Profiler, do the same for its HBase table.
```
- echo "create 'profiler', 'P'" | hbase shell
- echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
- ```
+ echo "create 'profiler', 'P'" | hbase shell
+ echo "grant 'metron', 'RW', 'profiler', 'P'" | hbase shell
+ ```
Storm Authorization
-------------------
@@ -260,19 +259,18 @@ Storm Authorization
1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
```
- su metron
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
1. Create the directory `/home/metron/.storm` and switch to that directory.
```
- mkdir /home/metron/.storm
- cd /home/metron/.storm
- ```
-
-1. Ensure the Metron keytab is renewable. See [Verify KDC](#verify-kdc) above
+ mkdir /home/metron/.storm
+ cd /home/metron/.storm
+ ```
+1. Ensure the Metron keytab is renewable. See [Verify KDC](#verify-kdc) above.
1. Create a client JAAS file at `/home/metron/.storm/client_jaas.conf`. This should look identical to the Storm client JAAS file located at `/etc/storm/conf/client_jaas.conf` except for the addition of a `Client` stanza. The `Client` stanza is used for Zookeeper. All quotes and semicolons are necessary.
@@ -325,18 +323,18 @@ Storm Authorization
EOF
```
-1. Configure the Enrichment, Indexing and the Profiler topologies to use the client JAAS file. Add the following properties to each of the topology properties files.
+1. Configure the Enrichment, Indexing and Profiler topologies to use the client JAAS file. To do this, the following key-value pairs:
+
+ * `kafka.security.protocol=PLAINTEXTSASL`
+ * `topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf`
- ```
- kafka.security.protocol=PLAINTEXTSASL
- topology.worker.childopts=-Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf
- ```
+ must be added to each of the topology properties files:
* `${METRON_HOME}/config/enrichment.properties`
* `${METRON_HOME}/config/elasticsearch.properties`
* `${METRON_HOME}/config/profiler.properties`
- Use the following command to automate this step.
+ You may use the following command to automate this step:
```
for file in enrichment.properties elasticsearch.properties profiler.properties; do
@@ -352,9 +350,9 @@ Start Metron
1. Switch to the `metron` user and acquire a Kerberos ticket for the `metron` principal.
```
- su metron
- kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
- ```
+ su metron
+ kinit -kt /etc/security/keytabs/metron.headless.keytab metron@EXAMPLE.COM
+ ```
1. Restart the parser topologies. Be sure to pass in the new parameter, `-ksp` or `--kafka_security_protocol`. The following command will start only the Bro and Snort topologies. Execute the same command for any other Parsers that you may need, for example `yaf`.
@@ -371,9 +369,9 @@ Start Metron
1. Restart the Enrichment and Indexing topologies.
```
- ${METRON_HOME}/bin/start_enrichment_topology.sh
- ${METRON_HOME}/bin/start_elasticsearch_topology.sh
- ```
+ ${METRON_HOME}/bin/start_enrichment_topology.sh
+ ${METRON_HOME}/bin/start_elasticsearch_topology.sh
+ ```
Metron should be ready to receive data.
@@ -382,18 +380,18 @@ Push Data
1. Push some sample data to one of the parser topics. E.g for Bro we took raw data from [metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
- cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
- --broker-list ${BROKERLIST}
- --security-protocol SASL_PLAINTEXT \
+ cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+ --broker-list ${BROKERLIST} \
+ --security-protocol SASL_PLAINTEXT \
--topic bro
- ```
+ ```
1. Wait a few moments for data to flow through the system and then check for data in the Elasticsearch indices. Replace yaf with whichever parser type you’ve chosen.
```
- curl -XGET "${ELASTICSEARCH}/bro*/_search"
- curl -XGET "${ELASTICSEARCH}/bro*/_count"
- ```
+ curl -XGET "${ELASTICSEARCH}/bro*/_search"
+ curl -XGET "${ELASTICSEARCH}/bro*/_count"
+ ```
1. You should have data flowing from the parsers all the way through to the indexes. This completes the Kerberization instructions
@@ -428,6 +426,7 @@ KVNO Timestamp Principal
A couple steps are required to produce data to a Kerberized Kafka topic. On the host you'll be setting up your sensor(s), switch to the metron user and create a client_jaas.conf file in the metron home directory if one doesn't already exist. It should be owned by metron:metron and
contain at least the following stanza that tells the Kafka client how to interact with Kerberos:
+
```
su - metron
cat ${METRON_HOME}/client_jaas.conf
@@ -444,6 +443,7 @@ KafkaClient {
```
You'll also need to set KAFKA_OPTS to tell the Kafka client how to interact with Kerberos.
+
```
export KAFKA_OPTS="-Djava.security.auth.login.config=${METRON_HOME}/client_jaas.conf"
```
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-deployment/packaging/docker/ansible-docker/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/ansible-docker/README.md b/metron-deployment/packaging/docker/ansible-docker/README.md
index ebd594a..3d50d23 100644
--- a/metron-deployment/packaging/docker/ansible-docker/README.md
+++ b/metron-deployment/packaging/docker/ansible-docker/README.md
@@ -3,14 +3,14 @@ The Metron ansible-docker container is provided in an effort reduce the installa
It is provisioned with software required to sucessfully run the deployment scripts.
## Building the Container
-1. Install Docker [https://www.docker.com/products/overview]
+1. Install Docker ( https://www.docker.com/products/overview )
2. Navigate to \<project-directory\>/metron-deployment/packaging/docker/ansible-docker
3. Build the container `docker build -t ansible-docker:2.0.0.2 .`
## Using the Container
-Full instructions are found on the wiki [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361].
+Full instructions are found on the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361
-tl;dr
+tl;dr:
1. docker run -it -v \<project-directory\>:/root/metron ansible-docker:2.0.0.2 bash
2. cd /root/metron
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-deployment/packaging/docker/rpm-docker/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/docker/rpm-docker/README.md b/metron-deployment/packaging/docker/rpm-docker/README.md
index 584e05f..d62b204 100644
--- a/metron-deployment/packaging/docker/rpm-docker/README.md
+++ b/metron-deployment/packaging/docker/rpm-docker/README.md
@@ -3,6 +3,6 @@ The Metron ansible-docker container is provided in an effort reduce the installa
It is provisioned with software required to sucessfully run the deployment scripts.
## Building the Container
-1. Install Docker [https://www.docker.com/products/overview]
+1. Install Docker ( https://www.docker.com/products/overview )
2. Navigate to \<project-directory\>/metron-deployment/packaging/rpm-docker
3. Build the container `docker build -t rpm-docker .`
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-deployment/roles/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/README.md b/metron-deployment/roles/README.md
index 32c4fbc..ce6c3ba 100644
--- a/metron-deployment/roles/README.md
+++ b/metron-deployment/roles/README.md
@@ -1,7 +1,7 @@
# Ansible Roles
-- Kibana
- Monit
- OpenTaxii
-- Pcap Relay
+- Pcap Replay
+- Sensor Stubs
- Sensor Test Mode
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-interface/metron-config/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/README.md b/metron-interface/metron-config/README.md
index 241adff..257faba 100644
--- a/metron-interface/metron-config/README.md
+++ b/metron-interface/metron-config/README.md
@@ -12,47 +12,53 @@ This module provides a user interface for management functions in Metron.
### From Source
1. Package the application with Maven:
-```
-cd metron-interface/metron-config
-mvn clean package
-```
+
+ ```
+ cd metron-interface/metron-config
+ mvn clean package
+ ```
1. Untar the archive in the $METRON_HOME directory. The directory structure will look like:
-```
-bin
- metron-management-ui
-web
- expressjs
+
+ ```
+ bin
+ metron-management-ui
+ web
+ expressjs
package.json
server.js
- management-ui
+ management-ui
web assets (html, css, js, ...)
-```
+ ```
1. Copy the `$METRON_HOME/bin/metron-management-ui` script to `/etc/init.d/metron-management-ui`
1. Install the [Express](https://expressjs.com/) web framework from the `package.json` file in `$METRON_HOME/web/expressjs`:
-```
-npm --prefix $METRON_HOME/web/expressjs/ install
-```
+
+ ```
+ npm --prefix $METRON_HOME/web/expressjs/ install
+ ```
### From Package Manager
1. Deploy the RPM at `/metron/metron-deployment/packaging/docker/rpm-docker/target/RPMS/noarch/metron-config-$METRON_VERSION-*.noarch.rpm`
1. Install the RPM with:
-```
-rpm -ih metron-config-$METRON_VERSION-*.noarch.rpm
-```
+
+ ```
+ rpm -ih metron-config-$METRON_VERSION-*.noarch.rpm
+ ```
1. Install the [Express](https://expressjs.com/) web framework from the `package.json` file in `$METRON_HOME/web/expressjs`:
-```
-npm --prefix $METRON_HOME/web/expressjs/ install
-```
+
+ ```
+ npm --prefix $METRON_HOME/web/expressjs/ install
+ ```
## Configuration
The Managment UI is configured in the `$METRON_HOME/config/management_ui.yml` file. Create this file and set the values to match your environment:
+
```
port: port the managment UI will run on
@@ -64,6 +70,7 @@ rest:
## Usage
After configuration is complete, the Management UI can be managed as a service:
+
```
service metron-management-ui start
```
@@ -75,30 +82,34 @@ The application will be available at http://host:4200 assuming the port is set t
The Management UI can also be started in development mode. This allows changes to web assets to be seen interactively.
1. Install the application with dev dependencies:
-```
-cd metron-interface/metron-config
-npm install
-```
+
+ ```
+ cd metron-interface/metron-config
+ npm install
+ ```
1. Start the application:
-```
-./scripts/start_dev.sh
-```
+
+ ```
+ ./scripts/start_dev.sh
+ ```
The application will be available at http://localhost:4200/. The REST application url defaults to `http://localhost:8080` but can be changed in the `proxy.conf.json` file.
## Testing
1. Install the application with dev dependencies:
-```
-cd metron-interface/metron-config
-npm install
-```
+
+ ```
+ cd metron-interface/metron-config
+ npm install
+ ```
1. Unit tests can be run with:
-```
-npm test
-```
+
+ ```
+ npm test
+ ```
## License
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-platform/metron-pcap-backend/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap-backend/README.md b/metron-platform/metron-pcap-backend/README.md
index da19611..69ffe52 100644
--- a/metron-platform/metron-pcap-backend/README.md
+++ b/metron-platform/metron-pcap-backend/README.md
@@ -1,7 +1,7 @@
# Metron PCAP Backend
The purpose of the Metron PCAP backend is to create a storm topology
-capable of ingesting rapidly raw packet capture data directly into HDFS
+capable of rapidly ingesting raw packet capture data directly into HDFS
from Kafka.
* [Sensors](#the-sensors-feeding-kafka)
http://git-wip-us.apache.org/repos/asf/metron/blob/e31ed2f7/metron-sensors/pycapa/README.md
----------------------------------------------------------------------
diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md
index fed1399..7b58ca1 100644
--- a/metron-sensors/pycapa/README.md
+++ b/metron-sensors/pycapa/README.md
@@ -37,7 +37,6 @@ General notes on the installation of Pycapa.
```
export PREFIX=/usr
-
wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz
cd librdkafka-0.9.4/
./configure --prefix=$PREFIX
@@ -251,7 +250,6 @@ The probe can be used in a Kerberized environment. Follow these additional step
--add \
--allow-principal User:metron \
--topic pcap
-
${KAFKA_HOME}/bin/kafka-acls.sh \
--authorizer kafka.security.auth.SimpleAclAuthorizer \
--authorizer-properties zookeeper.connect=zookeeper1:2181 \
@@ -264,6 +262,7 @@ The probe can be used in a Kerberized environment. Follow these additional step
* `security.protocol`
* `sasl.kerberos.keytab`
* `sasl.kerberos.principal`
+
```
$ pycapa --producer \
--interface eth0 \
@@ -284,6 +283,7 @@ FAQs
**Question**: How do I get more logs?
Use the following two command-line arguments to get detailed logging.
+
```
-X debug=all --log-level DEBUG
```
[05/18] metron git commit: METRON-569 Enrichment topology duplicates
messages (merrimanr) closes apache/metron#603
Posted by ma...@apache.org.
METRON-569 Enrichment topology duplicates messages (merrimanr) closes apache/metron#603
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/d0e1ba50
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/d0e1ba50
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/d0e1ba50
Branch: refs/heads/Metron_0.4.0
Commit: d0e1ba5089eb08187fd36c55fbe40539ef1f7890
Parents: 43e529f
Author: merrimanr <me...@gmail.com>
Authored: Mon Jun 5 16:51:18 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Mon Jun 5 16:51:18 2017 -0500
----------------------------------------------------------------------
.../src/main/flux/enrichment/remote.yaml | 4 +-
.../enrichment/bolt/EnrichmentJoinBolt.java | 7 +-
.../apache/metron/enrichment/bolt/JoinBolt.java | 29 +++---
.../enrichment/bolt/ThreatIntelJoinBolt.java | 6 +-
.../enrichment/bolt/EnrichmentJoinBoltTest.java | 18 +++-
.../metron/enrichment/bolt/JoinBoltTest.java | 98 ++++++++++++++------
.../bolt/ThreatIntelJoinBoltTest.java | 40 ++++----
7 files changed, 135 insertions(+), 67 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml b/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
index e4f119e..0e50f77 100644
--- a/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
+++ b/metron-platform/metron-enrichment/src/main/flux/enrichment/remote.yaml
@@ -312,7 +312,7 @@ bolts:
- "${kafka.zk}"
configMethods:
- name: "withMaxCacheSize"
- args: [10000]
+ args: [100000]
- name: "withMaxTimeRetain"
args: [10]
- id: "enrichmentErrorOutputBolt"
@@ -366,7 +366,7 @@ bolts:
- "${kafka.zk}"
configMethods:
- name: "withMaxCacheSize"
- args: [10000]
+ args: [100000]
- name: "withMaxTimeRetain"
args: [10]
- id: "threatIntelErrorOutputBolt"
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
index 2adf430..4b88399 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
@@ -17,11 +17,13 @@
*/
package org.apache.metron.enrichment.bolt;
+import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.storm.task.TopologyContext;
import com.google.common.base.Joiner;
import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
import org.apache.metron.common.utils.MessageUtils;
+import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -66,10 +68,11 @@ public class EnrichmentJoinBolt extends JoinBolt<JSONObject> {
@Override
- public JSONObject joinMessages(Map<String, JSONObject> streamMessageMap) {
+ public JSONObject joinMessages(Map<String, Tuple> streamMessageMap, MessageGetStrategy messageGetStrategy) {
JSONObject message = new JSONObject();
for (String key : streamMessageMap.keySet()) {
- JSONObject obj = streamMessageMap.get(key);
+ Tuple tuple = streamMessageMap.get(key);
+ JSONObject obj = (JSONObject) messageGetStrategy.get(tuple);
message.putAll(obj);
}
List<Object> emptyKeys = new ArrayList<>();
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
index a8e793d..f3fe52c 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/JoinBolt.java
@@ -51,11 +51,11 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
.getLogger(JoinBolt.class);
protected OutputCollector collector;
- protected transient CacheLoader<String, Map<String, V>> loader;
- protected transient LoadingCache<String, Map<String, V>> cache;
- private transient MessageGetStrategy keyGetStrategy;
- private transient MessageGetStrategy subgroupGetStrategy;
- private transient MessageGetStrategy messageGetStrategy;
+ protected transient CacheLoader<String, Map<String, Tuple>> loader;
+ protected transient LoadingCache<String, Map<String, Tuple>> cache;
+ protected transient MessageGetStrategy keyGetStrategy;
+ protected transient MessageGetStrategy subgroupGetStrategy;
+ protected transient MessageGetStrategy messageGetStrategy;
protected Long maxCacheSize;
protected Long maxTimeRetain;
@@ -86,9 +86,9 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
if (this.maxTimeRetain == null) {
throw new IllegalStateException("maxTimeRetain must be specified");
}
- loader = new CacheLoader<String, Map<String, V>>() {
+ loader = new CacheLoader<String, Map<String, Tuple>>() {
@Override
- public Map<String, V> load(String key) throws Exception {
+ public Map<String, Tuple> load(String key) throws Exception {
return new HashMap<>();
}
};
@@ -98,10 +98,10 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
prepare(map, topologyContext);
}
- class JoinRemoveListener implements RemovalListener<String, Map<String, V>> {
+ class JoinRemoveListener implements RemovalListener<String, Map<String, Tuple>> {
@Override
- public void onRemoval(RemovalNotification<String, Map<String, V>> removalNotification) {
+ public void onRemoval(RemovalNotification<String, Map<String, Tuple>> removalNotification) {
if (removalNotification.getCause() == RemovalCause.SIZE) {
String errorMessage = "Join cache reached max size limit. Increase the maxCacheSize setting or add more tasks to enrichment/threatintel join bolt.";
Exception exception = new Exception(errorMessage);
@@ -126,12 +126,12 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
streamId = Joiner.on(":").join("" + streamId, subgroup == null?"":subgroup);
V message = (V) messageGetStrategy.get(tuple);
try {
- Map<String, V> streamMessageMap = cache.get(key);
+ Map<String, Tuple> streamMessageMap = cache.get(key);
if (streamMessageMap.containsKey(streamId)) {
LOG.warn(String.format("Received key %s twice for " +
"stream %s", key, streamId));
}
- streamMessageMap.put(streamId, message);
+ streamMessageMap.put(streamId, tuple);
Set<String> streamIds = getStreamIds(message);
Set<String> streamMessageKeys = streamMessageMap.keySet();
if ( streamMessageKeys.size() == streamIds.size()
@@ -141,11 +141,12 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
collector.emit( "message"
, tuple
, new Values( key
- , joinMessages(streamMessageMap)
+ , joinMessages(streamMessageMap, this.messageGetStrategy)
)
);
cache.invalidate(key);
- collector.ack(tuple);
+ Tuple messageTuple = streamMessageMap.get("message:");
+ collector.ack(messageTuple);
LOG.trace("Emitted message for key: {}", key);
} else {
cache.put(key, streamMessageMap);
@@ -177,5 +178,5 @@ public abstract class JoinBolt<V> extends ConfiguredEnrichmentBolt {
public abstract Set<String> getStreamIds(V value);
- public abstract V joinMessages(Map<String, V> streamMessageMap);
+ public abstract V joinMessages(Map<String, Tuple> streamMessageMap, MessageGetStrategy messageGetStrategy);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
index 4d924c3..d4865e2 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
@@ -27,11 +27,13 @@ import org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriag
import org.apache.metron.common.dsl.Context;
import org.apache.metron.common.dsl.StellarFunctions;
import org.apache.metron.common.dsl.functions.resolver.FunctionResolver;
+import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.metron.common.utils.ConversionUtils;
import org.apache.metron.common.utils.MessageUtils;
import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
import org.apache.metron.threatintel.triage.ThreatTriageProcessor;
import org.apache.storm.task.TopologyContext;
+import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -132,8 +134,8 @@ public class ThreatIntelJoinBolt extends EnrichmentJoinBolt {
}
@Override
- public JSONObject joinMessages(Map<String, JSONObject> streamMessageMap) {
- JSONObject ret = super.joinMessages(streamMessageMap);
+ public JSONObject joinMessages(Map<String, Tuple> streamMessageMap, MessageGetStrategy messageGetStrategy) {
+ JSONObject ret = super.joinMessages(streamMessageMap, messageGetStrategy);
LOG.trace("Received joined messages: {}", ret);
boolean isAlert = ret.containsKey("is_alert");
if(!isAlert) {
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBoltTest.java
index 56ddf08..77dd4cf 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBoltTest.java
@@ -18,7 +18,9 @@
package org.apache.metron.enrichment.bolt;
import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
+import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
@@ -32,6 +34,9 @@ import java.util.HashMap;
import java.util.Map;
import java.util.Set;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
public class EnrichmentJoinBoltTest extends BaseEnrichmentBoltTest {
/**
@@ -75,10 +80,15 @@ public class EnrichmentJoinBoltTest extends BaseEnrichmentBoltTest {
enrichmentJoinBolt.prepare(new HashMap<>(), topologyContext, outputCollector);
Set<String> actualStreamIds = enrichmentJoinBolt.getStreamIds(sampleMessage);
Assert.assertEquals(joinStreamIds, actualStreamIds);
- Map<String, JSONObject> streamMessageMap = new HashMap<>();
- streamMessageMap.put("message", sampleMessage);
- streamMessageMap.put("enriched", enrichedMessage);
- JSONObject joinedMessage = enrichmentJoinBolt.joinMessages(streamMessageMap);
+ Map<String, Tuple> streamMessageMap = new HashMap<>();
+ MessageGetStrategy messageGetStrategy = mock(MessageGetStrategy.class);
+ Tuple sampleTuple = mock(Tuple.class);
+ when(messageGetStrategy.get(sampleTuple)).thenReturn(sampleMessage);
+ Tuple enrichedTuple = mock(Tuple.class);
+ when(messageGetStrategy.get(enrichedTuple)).thenReturn(enrichedMessage);
+ streamMessageMap.put("message", sampleTuple);
+ streamMessageMap.put("enriched", enrichedTuple);
+ JSONObject joinedMessage = enrichmentJoinBolt.joinMessages(streamMessageMap, messageGetStrategy);
removeTimingFields(joinedMessage);
Assert.assertEquals(expectedJoinedMessage, joinedMessage);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/JoinBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/JoinBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/JoinBoltTest.java
index 9f12fcd..e03dc71 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/JoinBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/JoinBoltTest.java
@@ -21,9 +21,11 @@ import com.google.common.cache.LoadingCache;
import org.adrianwalker.multilinestring.Multiline;
import org.apache.metron.common.Constants;
import org.apache.metron.common.error.MetronError;
+import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
import org.apache.metron.test.error.MetronErrorJSONMatcher;
import org.apache.storm.task.TopologyContext;
+import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
@@ -44,6 +46,7 @@ import static org.mockito.Matchers.eq;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;
public class JoinBoltTest extends BaseEnrichmentBoltTest {
@@ -65,11 +68,12 @@ public class JoinBoltTest extends BaseEnrichmentBoltTest {
for(String s : streamIds) {
ret.add(s + ":");
}
+ ret.add("message:");
return ret;
}
@Override
- public JSONObject joinMessages(Map<String, JSONObject> streamMessageMap) {
+ public JSONObject joinMessages(Map<String, Tuple> streamMessageMap, MessageGetStrategy messageGetStrategy) {
return joinedMessage;
}
}
@@ -83,6 +87,7 @@ public class JoinBoltTest extends BaseEnrichmentBoltTest {
private String joinedMessageString;
private JSONObject joinedMessage;
+ private JoinBolt<JSONObject> joinBolt;
@Before
public void parseMessages() {
@@ -92,13 +97,13 @@ public class JoinBoltTest extends BaseEnrichmentBoltTest {
} catch (ParseException e) {
e.printStackTrace();
}
+ joinBolt = new StandAloneJoinBolt("zookeeperUrl");
+ joinBolt.setCuratorFramework(client);
+ joinBolt.setTreeCache(cache);
}
@Test
- public void test() throws Exception {
- StandAloneJoinBolt joinBolt = new StandAloneJoinBolt("zookeeperUrl");
- joinBolt.setCuratorFramework(client);
- joinBolt.setTreeCache(cache);
+ public void testPrepare() {
try {
joinBolt.prepare(new HashMap(), topologyContext, outputCollector);
fail("Should fail if a maxCacheSize property is not set");
@@ -110,38 +115,79 @@ public class JoinBoltTest extends BaseEnrichmentBoltTest {
} catch(IllegalStateException e) {}
joinBolt.withMaxTimeRetain(10000);
joinBolt.prepare(new HashMap(), topologyContext, outputCollector);
+ }
+
+ @Test
+ public void testDeclareOutputFields() {
joinBolt.declareOutputFields(declarer);
verify(declarer, times(1)).declareStream(eq("message"), argThat(new FieldsMatcher("key", "message")));
- when(tuple.getValueByField("key")).thenReturn(key);
- when(tuple.getSourceStreamId()).thenReturn("geo");
- when(tuple.getValueByField("message")).thenReturn(geoMessage);
- joinBolt.execute(tuple);
- verify(outputCollector, times(0)).emit(eq("message"), any(tuple.getClass()), any(Values.class));
- verify(outputCollector, times(0)).ack(tuple);
- when(tuple.getSourceStreamId()).thenReturn("host");
- when(tuple.getValueByField("message")).thenReturn(hostMessage);
- joinBolt.execute(tuple);
- verify(outputCollector, times(0)).emit(eq("message"), any(tuple.getClass()), any(Values.class));
- verify(outputCollector, times(0)).ack(tuple);
- when(tuple.getSourceStreamId()).thenReturn("hbaseEnrichment");
- when(tuple.getValueByField("message")).thenReturn(hbaseEnrichmentMessage);
- joinBolt.execute(tuple);
- when(tuple.getSourceStreamId()).thenReturn("stellar");
- when(tuple.getValueByField("message")).thenReturn(new JSONObject());
- verify(outputCollector, times(0)).emit(eq("message"), any(tuple.getClass()), eq(new Values(key, joinedMessage)));
- joinBolt.execute(tuple);
+ verify(declarer, times(1)).declareStream(eq("error"), argThat(new FieldsMatcher("message")));
+ verifyNoMoreInteractions(declarer);
+ }
+
+ @Test
+ public void testExecute() {
+ joinBolt.withMaxCacheSize(100);
+ joinBolt.withMaxTimeRetain(10000);
+ joinBolt.prepare(new HashMap(), topologyContext, outputCollector);
+
+ Tuple geoTuple = mock(Tuple.class);
+ when(geoTuple.getValueByField("key")).thenReturn(key);
+ when(geoTuple.getSourceStreamId()).thenReturn("geo");
+ when(geoTuple.getValueByField("message")).thenReturn(geoMessage);
+ joinBolt.execute(geoTuple);
+
+ Tuple messageTuple = mock(Tuple.class);
+ when(messageTuple.getValueByField("key")).thenReturn(key);
+ when(messageTuple.getSourceStreamId()).thenReturn("message");
+ when(messageTuple.getValueByField("message")).thenReturn(sampleMessage);
+ joinBolt.execute(messageTuple);
+
+ Tuple hostTuple = mock(Tuple.class);
+ when(hostTuple.getValueByField("key")).thenReturn(key);
+ when(hostTuple.getSourceStreamId()).thenReturn("host");
+ when(hostTuple.getValueByField("message")).thenReturn(hostMessage);
+ joinBolt.execute(hostTuple);
+
+ Tuple hbaseEnrichmentTuple = mock(Tuple.class);
+ when(hbaseEnrichmentTuple.getValueByField("key")).thenReturn(key);
+ when(hbaseEnrichmentTuple.getSourceStreamId()).thenReturn("hbaseEnrichment");
+ when(hbaseEnrichmentTuple.getValueByField("message")).thenReturn(hbaseEnrichmentMessage);
+ joinBolt.execute(hbaseEnrichmentTuple);
+
+ Tuple stellarTuple = mock(Tuple.class);
+ when(stellarTuple.getValueByField("key")).thenReturn(key);
+ when(stellarTuple.getSourceStreamId()).thenReturn("stellar");
+ when(stellarTuple.getValueByField("message")).thenReturn(new JSONObject());
+ joinBolt.execute(stellarTuple);
+
verify(outputCollector, times(1)).emit(eq("message"), any(tuple.getClass()), eq(new Values(key, joinedMessage)));
- verify(outputCollector, times(1)).ack(tuple);
+ verify(outputCollector, times(1)).ack(messageTuple);
+
+ verifyNoMoreInteractions(outputCollector);
+ }
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testExecuteShouldReportError() throws ExecutionException {
+ joinBolt.withMaxCacheSize(100);
+ joinBolt.withMaxTimeRetain(10000);
+ joinBolt.prepare(new HashMap(), topologyContext, outputCollector);
+ when(tuple.getValueByField("key")).thenReturn(key);
+ when(tuple.getValueByField("message")).thenReturn(new JSONObject());
joinBolt.cache = mock(LoadingCache.class);
when(joinBolt.cache.get(key)).thenThrow(new ExecutionException(new Exception("join exception")));
- joinBolt.execute(tuple);
+ joinBolt.execute(tuple);
+ ExecutionException expectedExecutionException = new ExecutionException(new Exception("join exception"));
MetronError error = new MetronError()
.withErrorType(Constants.ErrorType.ENRICHMENT_ERROR)
.withMessage("Joining problem: {}")
- .withThrowable(new ExecutionException(new Exception("join exception")))
+ .withThrowable(expectedExecutionException)
.addRawMessage(new JSONObject());
verify(outputCollector, times(1)).emit(eq(Constants.ERROR_STREAM), argThat(new MetronErrorJSONMatcher(error.getJSONObject())));
+ verify(outputCollector, times(1)).reportError(any(ExecutionException.class));
+ verify(outputCollector, times(1)).ack(eq(tuple));
+ verifyNoMoreInteractions(outputCollector);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/d0e1ba50/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
index 6fe318e..0f3cc8c 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
@@ -19,16 +19,15 @@ package org.apache.metron.enrichment.bolt;
import com.fasterxml.jackson.databind.JsonMappingException;
import junit.framework.Assert;
-import junit.framework.TestCase;
import org.adrianwalker.multilinestring.Multiline;
-import org.apache.hadoop.fs.Path;
import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
-import org.apache.metron.common.configuration.enrichment.threatintel.ThreatScore;
import org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
+import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.metron.common.utils.JSONUtils;
import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
import org.apache.metron.test.utils.UnitTestHelper;
+import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
@@ -39,9 +38,13 @@ import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.HashMap;
-import java.util.List;
import java.util.Map;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
public class ThreatIntelJoinBoltTest extends BaseEnrichmentBoltTest {
/**
@@ -187,26 +190,29 @@ public class ThreatIntelJoinBoltTest extends BaseEnrichmentBoltTest {
fieldMap = threatIntelJoinBolt.getFieldMap(sensorType);
Assert.assertTrue(fieldMap.containsKey("hbaseThreatIntel"));
- Map<String, JSONObject> streamMessageMap = new HashMap<>();
- streamMessageMap.put("message", message);
- JSONObject joinedMessage = threatIntelJoinBolt.joinMessages(streamMessageMap);
- Assert.assertFalse(joinedMessage.containsKey("is_alert"));
+ MessageGetStrategy messageGetStrategy = mock(MessageGetStrategy.class);
+ Tuple messageTuple = mock(Tuple.class);
+ when(messageGetStrategy.get(messageTuple)).thenReturn(message);
+ Map<String, Tuple> streamMessageMap = new HashMap<>();
+ streamMessageMap.put("message", messageTuple);
+ JSONObject joinedMessage = threatIntelJoinBolt.joinMessages(streamMessageMap, messageGetStrategy);
+ assertFalse(joinedMessage.containsKey("is_alert"));
- streamMessageMap.put("message", messageWithTiming);
- joinedMessage = threatIntelJoinBolt.joinMessages(streamMessageMap);
- Assert.assertFalse(joinedMessage.containsKey("is_alert"));
+ when(messageGetStrategy.get(messageTuple)).thenReturn(messageWithTiming);
+ joinedMessage = threatIntelJoinBolt.joinMessages(streamMessageMap, messageGetStrategy);
+ assertFalse(joinedMessage.containsKey("is_alert"));
- streamMessageMap.put("message", alertMessage);
- joinedMessage = threatIntelJoinBolt.joinMessages(streamMessageMap);
- Assert.assertTrue(joinedMessage.containsKey("is_alert") && "true".equals(joinedMessage.get("is_alert")));
+ when(messageGetStrategy.get(messageTuple)).thenReturn(alertMessage);
+ joinedMessage = threatIntelJoinBolt.joinMessages(streamMessageMap, messageGetStrategy);
+ assertTrue(joinedMessage.containsKey("is_alert") && "true".equals(joinedMessage.get("is_alert")));
if(withThreatTriage && !badConfig) {
- Assert.assertTrue(joinedMessage.containsKey("threat.triage.score"));
+ assertTrue(joinedMessage.containsKey("threat.triage.score"));
Double score = (Double) joinedMessage.get("threat.triage.score");
- Assert.assertTrue(Math.abs(10d - score) < 1e-10);
+ assertTrue(Math.abs(10d - score) < 1e-10);
}
else {
- Assert.assertFalse(joinedMessage.containsKey("threat.triage.score"));
+ assertFalse(joinedMessage.containsKey("threat.triage.score"));
}
}
}
[12/18] metron git commit: METRON-987: Allow stellar enrichments to
be specified by a list as well as a map closes apache/metron#609
Posted by ma...@apache.org.
METRON-987: Allow stellar enrichments to be specified by a list as well as a map closes apache/metron#609
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/7d6121bf
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/7d6121bf
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/7d6121bf
Branch: refs/heads/Metron_0.4.0
Commit: 7d6121bfd3bd622940e9f411ac87b56d29755596
Parents: 61cbab4
Author: cstella <ce...@gmail.com>
Authored: Fri Jun 9 00:52:34 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Fri Jun 9 00:52:34 2017 -0400
----------------------------------------------------------------------
.../profiler/client/stellar/GetProfile.java | 2 +-
.../enrichment/handler/Config.java | 41 +++-
.../enrichment/handler/ConfigHandler.java | 12 +-
.../enrichment/handler/Configs.java | 9 +-
.../enrichment/handler/ListConfig.java | 16 +-
.../enrichment/handler/StellarConfig.java | 82 ++++++--
.../metron/common/message/JSONFromPosition.java | 7 +-
.../common/stellar/BaseStellarProcessor.java | 1 +
.../common/stellar/StellarAssignment.java | 135 +++++++++++++
.../common/stellar/shell/StellarShell.java | 19 +-
.../StellarEnrichmentConfigTest.java | 144 ++++++++++++++
.../configuration/StellarEnrichmentTest.java | 189 +++++++++++++++++++
.../common/stellar/StellarAssignmentTest.java | 63 +++++++
metron-platform/metron-enrichment/README.md | 91 +++++++--
metron-platform/metron-enrichment/pom.xml | 7 +
.../adapters/stellar/StellarAdapter.java | 98 +++++++---
.../enrichment/bolt/EnrichmentJoinBolt.java | 3 +-
.../enrichment/bolt/EnrichmentSplitterBolt.java | 2 +-
.../adapters/stellar/StellarAdapterTest.java | 134 +++++++++++++
19 files changed, 975 insertions(+), 80 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
index ecce7e0..87232aa 100644
--- a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
+++ b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/GetProfile.java
@@ -247,7 +247,7 @@ public class GetProfile implements StellarFunction {
return provider.getTable(HBaseConfiguration.create(), tableName);
} catch (IOException e) {
- throw new IllegalArgumentException(String.format("Unable to access table: %s", tableName));
+ throw new IllegalArgumentException(String.format("Unable to access table: %s", tableName), e);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Config.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Config.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Config.java
index f447326..bbadeac 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Config.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Config.java
@@ -21,16 +21,53 @@ package org.apache.metron.common.configuration.enrichment.handler;
import com.google.common.collect.ImmutableList;
import org.json.simple.JSONObject;
+import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
public interface Config {
+
+ /**
+ * Split a message by the fields. Certain configs will do this differently than others, but
+ * these are the messages sent to the enrichment adapter downstream.
+ * @param message
+ * @param fields
+ * @param fieldToEnrichmentKey
+ * @param config The config to use
+ * @return
+ */
List<JSONObject> splitByFields( JSONObject message
, Object fields
, Function<String, String> fieldToEnrichmentKey
- , Map<String, Object> config
+ , Iterable<Map.Entry<String, Object>> config
);
- List<String> getSubgroups(Map<String, Object> config);
+ default List<JSONObject> splitByFields( JSONObject message
+ , Object fields
+ , Function<String, String> fieldToEnrichmentKey
+ , ConfigHandler handler
+ ) {
+ return splitByFields(message, fields, fieldToEnrichmentKey, handler.getType().toConfig(handler.getConfig()));
+ }
+
+ /**
+ *
+ * Return the subgroups for a given enrichment. This will allow the join bolt to know when the join is complete.
+ * NOTE: this implies that a given enrichment may have a 1 to many relationship with subgroups.
+ * @param config
+ * @return The list of subgroups
+ */
+ List<String> getSubgroups(Iterable<Map.Entry<String, Object>> config);
+
+ default List<String> getSubgroups(ConfigHandler handler) {
+ return getSubgroups(handler.getType().toConfig(handler.getConfig()));
+ }
+
+ /**
+ * Convert a config object (currently either a map or list is supported) to a list of configs.
+ * @param c Either a map or list representing the enrichment adapter configuration.
+ * @return
+ */
+ Iterable<Map.Entry<String, Object>> toConfig(Object c);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ConfigHandler.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ConfigHandler.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ConfigHandler.java
index 996b33f..11a4852 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ConfigHandler.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ConfigHandler.java
@@ -17,15 +17,13 @@
*/
package org.apache.metron.common.configuration.enrichment.handler;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
+import java.util.*;
public class ConfigHandler {
- private Map<String, Object> config;
+ private Object config;
private Configs type = Configs.LIST;
public ConfigHandler(String enrichment, Map<String, Object> obj) {
- config = (Map<String, Object>) obj.get("config");
+ config = obj.get("config");
if(obj.containsKey("type")) {
type = Configs.valueOf((String) obj.get("type"));
}
@@ -39,11 +37,11 @@ public class ConfigHandler {
config = new HashMap<>();
type = Configs.LIST;
}
- public Map<String, Object> getConfig() {
+ public Object getConfig() {
return config;
}
- public void setConfig(Map<String, Object> config) {
+ public void setConfig(Object config) {
this.config = config;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Configs.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Configs.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Configs.java
index 210caf4..dced564 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Configs.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/Configs.java
@@ -19,6 +19,7 @@ package org.apache.metron.common.configuration.enrichment.handler;
import org.json.simple.JSONObject;
+import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
@@ -36,16 +37,20 @@ public enum Configs implements Config {
public List<JSONObject> splitByFields(JSONObject message
, Object fields
, Function<String, String> fieldToEnrichmentKey
- , Map<String, Object> config
+ , Iterable<Map.Entry<String, Object>> config
)
{
return configCreator.splitByFields(message, fields, fieldToEnrichmentKey, config);
}
@Override
- public List<String> getSubgroups(Map<String, Object> config) {
+ public List<String> getSubgroups(Iterable<Map.Entry<String, Object>> config) {
return configCreator.getSubgroups(config);
}
+ @Override
+ public Iterable<Map.Entry<String, Object>> toConfig(Object c) {
+ return configCreator.toConfig(c);
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ListConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ListConfig.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ListConfig.java
index 7803947..852418a 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ListConfig.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/ListConfig.java
@@ -20,6 +20,8 @@ package org.apache.metron.common.configuration.enrichment.handler;
import com.google.common.collect.ImmutableList;
import org.json.simple.JSONObject;
+import java.util.ArrayList;
+import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.function.Function;
@@ -30,7 +32,7 @@ public class ListConfig implements Config {
public List<JSONObject> splitByFields( JSONObject message
, Object fieldsObj
, Function<String, String> fieldToEnrichmentKey
- , Map<String, Object> config
+ , Iterable<Map.Entry<String, Object>> config
)
{
List<String> fields = (List<String>)fieldsObj;
@@ -45,7 +47,17 @@ public class ListConfig implements Config {
}
@Override
- public List<String> getSubgroups(Map<String, Object> config) {
+ public List<String> getSubgroups(Iterable<Map.Entry<String, Object>> config) {
return ImmutableList.of("");
}
+
+ @Override
+ public Iterable<Map.Entry<String, Object>> toConfig(Object c) {
+ if(c instanceof Map) {
+ return ((Map<String, Object>)c).entrySet();
+ }
+ else {
+ return new ArrayList<>();
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
index 9a26873..7d49fbd 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
@@ -17,7 +17,10 @@
*/
package org.apache.metron.common.configuration.enrichment.handler;
+import com.google.common.base.Splitter;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.stellar.StellarAssignment;
import org.apache.metron.common.stellar.StellarProcessor;
import org.json.simple.JSONObject;
@@ -27,14 +30,14 @@ import java.util.function.Function;
public class StellarConfig implements Config {
@Override
- public List<String> getSubgroups(Map<String, Object> config) {
+ public List<String> getSubgroups(Iterable<Map.Entry<String, Object>> config) {
boolean includeEmpty = false;
List<String> ret = new ArrayList<>();
- for(Map.Entry<String, Object> kv : config.entrySet()) {
+ for(Map.Entry<String, Object> kv : config) {
if(kv.getValue() instanceof String) {
includeEmpty = true;
}
- else if(kv.getValue() instanceof Map) {
+ else if(kv.getValue() instanceof Map || kv.getValue() instanceof List) {
ret.add(kv.getKey());
}
}
@@ -45,46 +48,99 @@ public class StellarConfig implements Config {
}
@Override
+ public Iterable<Map.Entry<String, Object>> toConfig(Object c) {
+ if(c instanceof Map) {
+ return ((Map<String, Object>)c).entrySet();
+ }
+ else if(c instanceof Collection) {
+ List<Map.Entry<String, Object>> ret = new ArrayList<>();
+ for(Object o : (Collection)c) {
+ if(o instanceof String) {
+ StellarAssignment assignment = StellarAssignment.from((String)o);
+ ret.add(assignment);
+ }
+ else if(o instanceof Map.Entry) {
+ ret.add((Map.Entry<String, Object>)o);
+ }
+ else {
+ throw new IllegalStateException("Expected " + c + " to be a list of strings, but got non-string.");
+ }
+ }
+ return ret;
+ }
+ throw new IllegalStateException("Unable to convert config " + c
+ + " to stellar config. Expected List<String> or Map<String, Object>");
+ }
+
+ @Override
public List<JSONObject> splitByFields( JSONObject message
, Object fields
, Function<String, String> fieldToEnrichmentKey
- , Map<String, Object> config
+ , Iterable<Map.Entry<String, Object>> config
)
{
StellarProcessor processor = new StellarProcessor();
List<JSONObject> messages = new ArrayList<>();
Map<String, String> defaultStellarStatementGroup = new HashMap<>();
- for(Map.Entry<String, Object> kv : config.entrySet()) {
+ for(Map.Entry<String, Object> kv : config) {
if(kv.getValue() instanceof String) {
defaultStellarStatementGroup.put(kv.getKey(), (String)kv.getValue());
}
else if(kv.getValue() instanceof Map) {
JSONObject ret = new JSONObject();
- ret.put(kv.getKey(), getMessage(processor, (Map<String, String>) kv.getValue(), message));
+ ret.put(kv.getKey(), getMessage(getFields(processor, (Map)kv.getValue()), message));
+ messages.add(ret);
+ }
+ else if(kv.getValue() instanceof List) {
+ JSONObject ret = new JSONObject();
+ ret.put(kv.getKey(), getMessage(getFields(processor, (List)kv.getValue()), message));
messages.add(ret);
}
}
if(defaultStellarStatementGroup.size() > 0)
{
JSONObject ret = new JSONObject();
- ret.put("", getMessage(processor, defaultStellarStatementGroup, message));
+ ret.put("", getMessage(getFields(processor, defaultStellarStatementGroup), message));
messages.add(ret);
}
return messages;
}
- private Map<String, Object> getMessage( StellarProcessor processor
- , Map<String, String> stellarStatementGroup
- , JSONObject message
- )
+ private Set<String> getFields(StellarProcessor processor
+ , List<String> stellarStatementGroup
+ )
{
Set<String> stellarFields = new HashSet<>();
- for(String stellarStatement: stellarStatementGroup.values()) {
+ for(String stellarStatementExpr: stellarStatementGroup) {
+ StellarAssignment assignment = StellarAssignment.from(stellarStatementExpr);
+ if(assignment.getStatement() != null) {
+ Set<String> variables = processor.variablesUsed(assignment.getStatement());
+ if (variables != null) {
+ stellarFields.addAll(variables);
+ }
+ }
+ }
+ return stellarFields;
+ }
+
+ private Set<String> getFields( StellarProcessor processor
+ , Map<String, String> stellarStatementGroup
+ ) {
+ Set<String> stellarFields = new HashSet<>();
+ for (String stellarStatement : stellarStatementGroup.values()) {
Set<String> variables = processor.variablesUsed(stellarStatement);
- if(variables != null) {
+ if (variables != null) {
stellarFields.addAll(variables);
}
}
+ return stellarFields;
+ }
+
+ private Map<String, Object> getMessage( Set<String> stellarFields
+ , JSONObject message
+ )
+ {
+
Map<String, Object> messageSegment = new HashMap<>();
for(String variable : stellarFields) {
messageSegment.put(variable, message.get(variable));
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java
index 4407d4f..c91a262 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/message/JSONFromPosition.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.common.message;
+import org.apache.commons.io.Charsets;
import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
@@ -40,10 +41,12 @@ public class JSONFromPosition implements MessageGetStrategy {
@Override
public JSONObject get(Tuple tuple) {
+ String s = null;
try {
- return (JSONObject) parser.get().parse(new String(tuple.getBinary(position), "UTF8"));
+ s = new String(tuple.getBinary(position), Charsets.UTF_8);
+ return (JSONObject) parser.get().parse(s);
} catch (Exception e) {
- throw new IllegalStateException(e.getMessage(), e);
+ throw new IllegalStateException("Unable to parse " + s + " due to " + e.getMessage(), e);
}
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
index a04ee43..8f4af93 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
@@ -234,4 +234,5 @@ public class BaseStellarProcessor<T> {
return true;
}
+
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarAssignment.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarAssignment.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarAssignment.java
new file mode 100644
index 0000000..c28ebfb
--- /dev/null
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarAssignment.java
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.stellar;
+
+
+import java.util.Map;
+
+public class StellarAssignment implements Map.Entry<String, Object>{
+ private String variable;
+ private String statement;
+
+ public StellarAssignment(String variable, String statement) {
+ this.variable = variable;
+ this.statement = statement;
+ }
+
+ public String getVariable() {
+ return variable;
+ }
+
+ public String getStatement() {
+ return statement;
+ }
+
+ public static boolean isAssignment(String statement) {
+ return statement != null && statement.contains(":=");
+ }
+
+ public static StellarAssignment from(String statement) {
+ if(statement == null || statement.length() == 0) {
+ return new StellarAssignment(null, null);
+ }
+ char prev = statement.charAt(0);
+ char curr;
+ String variable = "" + prev;
+ String s = null;
+ boolean isAssignment = false;
+ for(int i = 1;i < statement.length();++i,prev=curr) {
+ curr = statement.charAt(i);
+ if(prev == ':' && curr == '=') {
+ isAssignment = true;
+ variable = variable.substring(0, variable.length() - 1);
+ s = "";
+ continue;
+ }
+ if(!isAssignment) {
+ variable += curr;
+ }
+ else {
+ s += curr;
+ }
+ }
+
+ if(!isAssignment) {
+ s = variable;
+ variable = null;
+ }
+
+ if(s != null) {
+ s = s.trim();
+ }
+ if(variable != null) {
+ variable = variable.trim();
+ }
+ return new StellarAssignment(variable, s);
+ }
+
+ /**
+ * Returns the key corresponding to this entry.
+ *
+ * @return the key corresponding to this entry
+ * @throws IllegalStateException implementations may, but are not
+ * required to, throw this exception if the entry has been
+ * removed from the backing map.
+ */
+ @Override
+ public String getKey() {
+ return variable;
+ }
+
+ /**
+ * Returns the value corresponding to this entry. If the mapping
+ * has been removed from the backing map (by the iterator's
+ * <tt>remove</tt> operation), the results of this call are undefined.
+ *
+ * @return the value corresponding to this entry
+ * @throws IllegalStateException implementations may, but are not
+ * required to, throw this exception if the entry has been
+ * removed from the backing map.
+ */
+ @Override
+ public Object getValue() {
+ return statement;
+ }
+
+ /**
+ * Replaces the value corresponding to this entry with the specified
+ * value (optional operation). (Writes through to the map.) The
+ * behavior of this call is undefined if the mapping has already been
+ * removed from the map (by the iterator's <tt>remove</tt> operation).
+ *
+ * @param value new value to be stored in this entry
+ * @return old value corresponding to the entry
+ * @throws UnsupportedOperationException if the <tt>put</tt> operation
+ * is not supported by the backing map
+ * @throws ClassCastException if the class of the specified value
+ * prevents it from being stored in the backing map
+ * @throws NullPointerException if the backing map does not permit
+ * null values, and the specified value is null
+ * @throws IllegalArgumentException if some property of this value
+ * prevents it from being stored in the backing map
+ * @throws IllegalStateException implementations may, but are not
+ * required to, throw this exception if the entry has been
+ * removed from the backing map.
+ */
+ @Override
+ public String setValue(Object value) {
+ throw new UnsupportedOperationException("Assignments are immutable.");
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java
index 4e24212..f13f1e3 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java
@@ -33,6 +33,7 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.metron.common.dsl.Context;
import org.apache.metron.common.dsl.StellarFunctionInfo;
import org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolver;
+import org.apache.metron.common.stellar.StellarAssignment;
import org.apache.metron.common.utils.JSONUtils;
import org.jboss.aesh.complete.CompleteOperation;
import org.jboss.aesh.complete.Completion;
@@ -247,19 +248,17 @@ public class StellarShell extends AeshConsoleCallback implements Completion {
*/
private void handleStellar(String expression) {
- Iterable<String> assignmentSplit = Splitter.on(":=").split(expression);
String stellarExpression = expression;
String variable = null;
- if(Iterables.size(assignmentSplit) == 2) {
- //assignment
- variable = Iterables.getFirst(assignmentSplit, null);
- if(variable != null) {
- variable = variable.trim();
- }
- stellarExpression = Iterables.getLast(assignmentSplit, null);
+ if(StellarAssignment.isAssignment(expression)) {
+ StellarAssignment expr = StellarAssignment.from(expression);
+ variable = expr.getVariable();
+ stellarExpression = expr.getStatement();
}
- if(!stellarExpression.isEmpty()) {
- stellarExpression = stellarExpression.trim();
+ else {
+ if (!stellarExpression.isEmpty()) {
+ stellarExpression = stellarExpression.trim();
+ }
}
Object result = executeStellar(stellarExpression);
if(result != null && variable == null) {
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
new file mode 100644
index 0000000..0883672
--- /dev/null
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
@@ -0,0 +1,144 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.configuration;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.configuration.enrichment.handler.Configs;
+import org.apache.metron.common.utils.JSONUtils;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+public class StellarEnrichmentConfigTest extends StellarEnrichmentTest {
+
+
+ @Test
+ public void testSplitter_default() throws IOException {
+ JSONObject message = getMessage();
+ for(String c : DEFAULT_CONFIGS) {
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<JSONObject> splits = Configs.STELLAR.splitByFields(message, null, x -> null, handler );
+ Assert.assertEquals(1, splits.size());
+ Map<String, Object> split = (Map<String, Object>) splits.get(0).get("");
+ Assert.assertEquals(3, split.size());
+ Assert.assertEquals("stellar_test", split.get("source.type"));
+ Assert.assertEquals("foo", split.get("string"));
+ Assert.assertNull(split.get("stmt1"));
+ }
+ }
+
+ @Test
+ public void testGetSubgroups_default() throws IOException {
+ for(String c : DEFAULT_CONFIGS) {
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<String> subgroups = Configs.STELLAR.getSubgroups(handler);
+ Assert.assertEquals("", subgroups.get(0));
+ Assert.assertEquals(1, subgroups.size());
+ }
+ }
+
+ @Test
+ public void testSplitter_grouped() throws IOException {
+ JSONObject message = getMessage();
+ for(String c : GROUPED_CONFIGS) {
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<JSONObject> splits = Configs.STELLAR.splitByFields(message, null, x -> null, handler );
+ Assert.assertEquals(2, splits.size());
+ {
+ Map<String, Object> split = (Map<String, Object>) splits.get(0).get("group1");
+ Assert.assertEquals(2, split.size());
+ Assert.assertEquals("stellar_test", split.get("source.type"));
+ Assert.assertNull(split.get("stmt1"));
+ }
+ {
+ Map<String, Object> split = (Map<String, Object>) splits.get(1).get("group2");
+ Assert.assertEquals(1, split.size());
+ Assert.assertEquals("foo", split.get("string"));
+ }
+ }
+ }
+
+ @Test
+ public void testGetSubgroups_grouped() throws IOException {
+ for(String c : GROUPED_CONFIGS) {
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<String> subgroups = Configs.STELLAR.getSubgroups(handler);
+ Assert.assertEquals("group1", subgroups.get(0));
+ Assert.assertEquals("group2", subgroups.get(1));
+ Assert.assertEquals(2, subgroups.size());
+ }
+ }
+
+
+ @Test
+ public void testSplitter_mixed() throws IOException {
+ JSONObject message = getMessage();
+ for(String c : Iterables.concat(MIXED_CONFIGS, ImmutableList.of(tempVarStellarConfig_list))) {
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<JSONObject> splits = Configs.STELLAR.splitByFields(message, null, x -> null, handler );
+ Assert.assertEquals(3, splits.size());
+ {
+ Map<String, Object> split = (Map<String, Object>) splits.get(0).get("group1");
+ Assert.assertEquals(2, split.size());
+ Assert.assertEquals("stellar_test", split.get("source.type"));
+ Assert.assertNull(split.get("stmt1"));
+ }
+ {
+ Map<String, Object> split = (Map<String, Object>) splits.get(1).get("group2");
+ Assert.assertEquals(1, split.size());
+ Assert.assertEquals("foo", split.get("string"));
+ }
+ {
+ Map<String, Object> split = (Map<String, Object>) splits.get(2).get("");
+ Assert.assertEquals(1, split.size());
+ Assert.assertEquals("stellar_test", split.get("source.type"));
+ }
+ }
+ }
+
+ @Test
+ public void testGetSubgroups_mixed() throws IOException {
+ for(String c : MIXED_CONFIGS) {
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<String> subgroups = Configs.STELLAR.getSubgroups(handler);
+ Assert.assertEquals("group1", subgroups.get(0));
+ Assert.assertEquals("group2", subgroups.get(1));
+ Assert.assertEquals("", subgroups.get(2));
+ Assert.assertEquals(3, subgroups.size());
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
new file mode 100644
index 0000000..3af15cd
--- /dev/null
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.configuration;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.google.common.collect.ImmutableList;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.common.utils.JSONUtils;
+import org.json.simple.JSONObject;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+
+public class StellarEnrichmentTest {
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "stmt1" : "TO_UPPER(source.type)",
+ "stmt2" : "TO_LOWER(stmt1)",
+ "stmt3" : "TO_LOWER(string)"
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String defaultStellarConfig_map;
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := TO_LOWER(stmt1)",
+ "stmt3 := TO_LOWER(string)"
+ ]
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String defaultStellarConfig_list;
+ public static List<String> DEFAULT_CONFIGS = ImmutableList.of(defaultStellarConfig_list, defaultStellarConfig_map);
+
+/**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : {
+ "stmt1" : "TO_UPPER(source.type)",
+ "stmt2" : "TO_LOWER(stmt1)"
+ },
+ "group2" : {
+ "stmt3" : "TO_LOWER(string)"
+ }
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String groupedStellarConfig_map;
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := TO_LOWER(stmt1)"
+ ],
+ "group2" : [
+ "stmt3 := TO_LOWER(string)"
+ ]
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String groupedStellarConfig_list;
+ public static List<String> GROUPED_CONFIGS = ImmutableList.of(groupedStellarConfig_list, groupedStellarConfig_map);
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : {
+ "stmt1" : "TO_UPPER(source.type)",
+ "stmt2" : "TO_LOWER(stmt1)"
+ },
+ "group2" : {
+ "stmt3" : "TO_LOWER(string)"
+ },
+ "stmt4" : "1 + 1",
+ "stmt5" : "FORMAT('%s', source.type)"
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String mixedStellarConfig_map;
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := TO_LOWER(stmt1)"
+ ],
+ "group2" : [
+ "stmt3 := TO_LOWER(string)"
+ ],
+ "stmt4" : "1 + 1",
+ "stmt5" : "FORMAT('%s', source.type)"
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String mixedStellarConfig_list;
+ public static List<String> MIXED_CONFIGS = ImmutableList.of(mixedStellarConfig_list, mixedStellarConfig_map);
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := TO_LOWER(stmt1)",
+ "stmt1 := null"
+ ],
+ "group2" : [
+ "stmt3 := TO_LOWER(string)"
+ ],
+ "stmt4" : "1 + 1",
+ "stmt5" : "FORMAT('%s', source.type)"
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String tempVarStellarConfig_list;
+
+ /**
+ {
+ "string" : "foo"
+ ,"number" : 2
+ ,"source.type" : "stellar_test"
+ }
+ */
+ @Multiline
+ public static String message;
+
+ public static JSONObject getMessage() throws IOException {
+ Map<String, Object> ret = JSONUtils.INSTANCE.load(message, new TypeReference<Map<String, Object>>() {
+ });
+ return new JSONObject(ret);
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarAssignmentTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarAssignmentTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarAssignmentTest.java
new file mode 100644
index 0000000..c4a6060
--- /dev/null
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarAssignmentTest.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.stellar;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class StellarAssignmentTest {
+
+ @Test
+ public void testAssignment() {
+ for(String statement : ImmutableList.of( "foo := bar + grok"
+ , "foo := bar + grok"
+ , "foo := bar + grok "
+ )
+ )
+ {
+ StellarAssignment assignment = StellarAssignment.from(statement);
+ Assert.assertEquals("foo", assignment.getKey());
+ Assert.assertEquals("foo", assignment.getVariable());
+ Assert.assertEquals("bar + grok", assignment.getStatement());
+ Assert.assertEquals("bar + grok", assignment.getValue());
+ }
+ }
+
+ @Test
+ public void testNonAssignment() {
+ for(String statement : ImmutableList.of( "bar + grok"
+ , " bar + grok"
+ , "bar + grok "
+ )
+ )
+ {
+ StellarAssignment assignment = StellarAssignment.from(statement);
+ Assert.assertNull( assignment.getKey());
+ Assert.assertNull( assignment.getVariable());
+ Assert.assertEquals("bar + grok", assignment.getStatement());
+ Assert.assertEquals("bar + grok", assignment.getValue());
+ }
+ }
+
+ @Test(expected=UnsupportedOperationException.class)
+ public void testImmutability() {
+ StellarAssignment assignment = StellarAssignment.from("foo := bar");
+ assignment.setValue("myval");
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-enrichment/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md
index a789f0f..e9d4dde 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -54,7 +54,9 @@ The `config` map is intended to house enrichment specific configuration.
For instance, for the `hbaseEnrichment`, the mappings between the
enrichment types to the column families is specified.
-The `fieldMap`contents are of interest because they contain the routing and configuration information for the enrichments. When we say 'routing', we mean how the messages get split up and sent to the enrichment adapter bolts. The simplest, by far, is just providing a simple list as in
+The `fieldMap`contents are of interest because they contain the routing and configuration information for the enrichments.
+When we say 'routing', we mean how the messages get split up and sent to the enrichment adapter bolts.
+The simplest, by far, is just providing a simple list as in
```
"fieldMap": {
"geo": [
@@ -71,40 +73,94 @@ The `fieldMap`contents are of interest because they contain the routing and conf
]
}
```
-Based on this sample config, both ip_src_addr and ip_dst_addr will go to the `geo`, `host`, and `hbaseEnrichment` adapter bolts. For the `geo`, `host` and `hbaseEnrichment`, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the `stellar` enrichment requires a more complex configuration, such as:
+Based on this sample config, both `ip_src_addr` and `ip_dst_addr` will go to the `geo`, `host`, and
+`hbaseEnrichment` adapter bolts.
+
+#### Stellar Enrichment Configuration
+For the `geo`, `host` and `hbaseEnrichment`, this is sufficient. However, more complex enrichments
+may contain their own configuration. Currently, the `stellar` enrichment is more adaptable and thus
+requires a more nuanced configuration.
+
+At its most basic, we want to take a message and apply a couple of enrichments, such as converting the
+`hostname` field to lowercase. We do this by specifying the transformation inside of the
+`config` for the `stellar` fieldMap. There are two syntaxes that are supported, specifying the transformations
+as a map with the key as the field and the value the stellar expression:
```
"fieldMap": {
...
"stellar" : {
"config" : {
- "numeric" : {
- "foo": "1 + 1"
- }
- ,"ALL_CAPS" : "TO_UPPER(source.type)"
+ "hostname" : "TO_LOWER(hostname)"
}
}
}
```
-Whereas the simpler enrichments just need a set of fields explicitly stated so they can be separated from the message and sent to the enrichment adapter bolt for enrichment and ultimately joined back in the join bolt, the stellar enrichment has its set of required fields implicitly stated through usage. For instance, if your stellar statement references a field, it should be included and if not, then it should not be included. We did not want to require users to make explicit the implicit.
+Another approach is to make the transformations as a list with the same `var := expr` syntax as is used
+in the Stellar REPL, such as:
+```
+ "fieldMap": {
+ ...
+ "stellar" : {
+ "config" : [
+ "hostname := TO_LOWER(hostname)"
+ ]
+ }
+ }
+```
+
+Sometimes arbitrary stellar enrichments may take enough time that you would prefer to split some of them
+into groups and execute the groups of stellar enrichments in parallel. Take, for instance, if you wanted
+to do an HBase enrichment and a profiler call which were independent of one another. This usecase is
+supported by splitting the enrichments up as groups.
-The other way in which the stellar enrichment is somewhat more complex is in how the statements are executed. In the general purpose case for a list of fields, those fields are used to create a message to send to the enrichment adapter bolt and that bolt's worker will handle the fields one by one in serial for a given message. For stellar enrichment, we wanted to have a more complex design so that users could specify the groups of stellar statements sent to the same worker in the same message (and thus executed sequentially). Consider the following configuration:
+Consider the following example:
```
"fieldMap": {
+ ...
"stellar" : {
"config" : {
- "numeric" : {
- "foo": "1 + 1"
- "bar" : TO_LOWER(source.type)"
- }
- ,"text" : {
- "ALL_CAPS" : "TO_UPPER(source.type)"
- }
+ "malicious_domain_enrichment" : {
+ "is_bad_domain" : "ENRICHMENT_EXISTS('malicious_domains', ip_dst_addr, 'enrichments', 'cf')"
+ },
+ "login_profile" : [
+ "profile_window := PROFILE_WINDOW('from 6 months ago')",
+ "global_login_profile := PROFILE_GET('distinct_login_attempts', 'global', profile_window)",
+ "stats := STATS_MERGE(global_login_profile)",
+ "auth_attempts_median := STATS_PERCENTILE(stats, 0.5)",
+ "auth_attempts_sd := STATS_SD(stats)",
+ "profile_window := null",
+ "global_login_profile := null",
+ "stats := null"
+ ]
}
}
}
```
-We have a group called `numeric` whose stellar statements will be executed sequentially. In parallel to that, we have the group of stellar statements under the group `text` executing. The intent here is to allow you to not force higher latency operations to be done sequentially. You can use any name for your groupings you like. Be aware that the configuration is a map and duplicate configuration keys' values are not combined, so the duplicate configuration value will be overwritten.
+
+Here we want to perform two enrichments that hit HBase and we would rather not run in sequence. These
+enrichments are entirely independent of one another (i.e. neither relies on the output of the other). In
+this case, we've created a group called `malicious_domain_enrichment` to inquire about whether the destination
+address exists in the HBase enrichment table in the `malicious_domains` enrichment type. This is a simple
+enrichment, so we can express the enrichment group as a map with the new field `is_bad_domain` being a key
+and the stellar expression associated with that operation being the associated value.
+
+In contrast, the stellar enrichment group `login_profile` is interacting with the profiler, has multiple temporary
+expressions (i.e. `profile_window`, `global_login_profile`, and `stats`) that are useful only within the context
+of this group of stellar expressions. In this case, we would need to ensure that we use the list construct
+when specifying the group and remember to set the temporary variables to `null` so they are not passed along.
+
+In general, things to note from this section are as follows:
+* The stellar enrichments for the `stellar` enrichment adapter are specified in the `config` for the `stellar` enrichment
+adapter in the `fieldMap`
+* Groups of independent (i.e. no expression in any group depend on the output of an expression from an other group) may be executed in parallel
+* If you have the need to use temporary variables, you may use the list construct. Ensure that you assign the variables to `null` before the end of the group.
+* **Ensure that you do not assign a field to a stellar expression which returns an object which JSON cannot represent.**
+* Fields assigned to Maps as part of stellar enrichments have the maps unfolded, similar to the HBase Enrichment
+ * For example the stellar enrichment for field `foo` which assigns a map such as `foo := { 'grok' : 1, 'bar' : 'baz'}`
+ would yield the following fields:
+ * `foo.grok` == `1`
+ * `foo.bar` == `'baz'`
### The `threatIntel` Configuration
@@ -117,7 +173,8 @@ We have a group called `numeric` whose stellar statements will be executed seque
The `config` map is intended to house threat intel specific configuration.
For instance, for the `hbaseThreatIntel` threat intel adapter, the mappings between the
-enrichment types to the column families is specified.
+enrichment types to the column families is specified. The `fieldMap` configuration is similar to the `enrichment`
+configuration in that the adapters available are the same.
The `triageConfig` field is also a complex field and it bears some description:
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-enrichment/pom.xml
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/pom.xml b/metron-platform/metron-enrichment/pom.xml
index 29de74f..d525a64 100644
--- a/metron-platform/metron-enrichment/pom.xml
+++ b/metron-platform/metron-enrichment/pom.xml
@@ -40,6 +40,13 @@
</dependency>
<dependency>
<groupId>org.apache.metron</groupId>
+ <artifactId>metron-common</artifactId>
+ <version>${project.parent.version}</version>
+ <scope>test</scope>
+ <type>test-jar</type>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.metron</groupId>
<artifactId>metron-storm-kafka</artifactId>
<version>${project.parent.version}</version>
</dependency>
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
index bc5f2b6..53ef16f 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
@@ -24,6 +24,7 @@ import org.apache.metron.common.dsl.MapVariableResolver;
import org.apache.metron.common.dsl.StellarFunctions;
import org.apache.metron.common.dsl.VariableResolver;
import org.apache.metron.common.stellar.StellarProcessor;
+import org.apache.metron.common.utils.ConversionUtils;
import org.apache.metron.enrichment.bolt.CacheKey;
import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
import org.json.simple.JSONObject;
@@ -37,7 +38,11 @@ import java.util.function.Function;
import static org.apache.metron.enrichment.bolt.GenericEnrichmentBolt.STELLAR_CONTEXT_CONF;
public class StellarAdapter implements EnrichmentAdapter<CacheKey>,Serializable {
+ public static class Perf {}
protected static final Logger _LOG = LoggerFactory.getLogger(StellarAdapter.class);
+ protected static final Logger _PERF_LOG = LoggerFactory.getLogger(Perf.class);
+ public static final String STELLAR_SLOW_LOG = "stellar.slow.threshold.ms";
+ public static final Long STELLAR_SLOW_LOG_DEFAULT = 1000l;
private enum EnrichmentType implements Function<SensorEnrichmentConfig, ConfigHandler>{
ENRICHMENT(config -> config.getEnrichment().getEnrichmentConfigs().get("stellar"))
@@ -74,6 +79,63 @@ public class StellarAdapter implements EnrichmentAdapter<CacheKey>,Serializable
return field;
}
+ public static Iterable<Map.Entry<String, Object>> getStellarStatements(ConfigHandler handler, String field) {
+ if(field.length() == 0) {
+ return handler.getType().toConfig(handler.getConfig());
+ }
+ else {
+ Map<String, Object> groupStatements = (Map<String, Object>)handler.getConfig();
+ return handler.getType().toConfig(groupStatements.get(field));
+ }
+ }
+
+ public static JSONObject process( Map<String, Object> message
+ , ConfigHandler handler
+ , String field
+ , Long slowLogThreshold
+ , StellarProcessor processor
+ , VariableResolver resolver
+ , Context stellarContext
+ )
+ {
+ JSONObject ret = new JSONObject();
+ Iterable<Map.Entry<String, Object>> stellarStatements = getStellarStatements(handler, field);
+
+ if(stellarStatements != null) {
+ for (Map.Entry<String, Object> kv : stellarStatements) {
+ if(kv.getKey() != null && kv.getValue() != null) {
+ if (kv.getValue() instanceof String) {
+ long startTime = System.currentTimeMillis();
+ String stellarStatement = (String) kv.getValue();
+ Object o = processor.parse(stellarStatement, resolver, StellarFunctions.FUNCTION_RESOLVER(), stellarContext);
+ if (slowLogThreshold != null && _PERF_LOG.isDebugEnabled()) {
+ long duration = System.currentTimeMillis() - startTime;
+ if (duration > slowLogThreshold) {
+ _PERF_LOG.debug("SLOW LOG: " + stellarStatement + " took" + duration + "ms");
+ }
+ }
+ if (o != null && o instanceof Map) {
+ for (Map.Entry<Object, Object> valueKv : ((Map<Object, Object>) o).entrySet()) {
+ String newKey = ((kv.getKey().length() > 0) ? kv.getKey() + "." : "") + valueKv.getKey();
+ message.put(newKey, valueKv.getValue());
+ ret.put(newKey, valueKv.getValue());
+ }
+ }
+ else if(o == null) {
+ message.remove(kv.getKey());
+ ret.remove(kv.getKey());
+ }
+ else {
+ message.put(kv.getKey(), o);
+ ret.put(kv.getKey(), o);
+ }
+ }
+ }
+ }
+ }
+ return ret;
+ }
+
@Override
public JSONObject enrich(CacheKey value) {
Context stellarContext = (Context) value.getConfig().getConfiguration().get(STELLAR_CONTEXT_CONF);
@@ -81,33 +143,25 @@ public class StellarAdapter implements EnrichmentAdapter<CacheKey>,Serializable
Map<String, Object> globalConfig = value.getConfig().getConfiguration();
Map<String, Object> sensorConfig = value.getConfig().getEnrichment().getConfig();
if(handler == null) {
- _LOG.trace("Stellar ConfigHandler is null.");
+ _LOG.trace("Stellar ConfigHandler is null.");
return new JSONObject();
}
+ Long slowLogThreshold = null;
+ if(_PERF_LOG.isDebugEnabled()) {
+ slowLogThreshold = ConversionUtils.convert(globalConfig.getOrDefault(STELLAR_SLOW_LOG, STELLAR_SLOW_LOG_DEFAULT), Long.class);
+ }
Map<String, Object> message = value.getValue(Map.class);
VariableResolver resolver = new MapVariableResolver(message, sensorConfig, globalConfig);
StellarProcessor processor = new StellarProcessor();
- Map<String, Object> stellarStatements = value.getField().length() == 0? handler.getConfig()
- : (Map)handler.getConfig().get(value.getField());
- if(stellarStatements != null) {
- for (Map.Entry<String, Object> kv : stellarStatements.entrySet()) {
- if(kv.getValue() instanceof String) {
- String stellarStatement = (String) kv.getValue();
- Object o = processor.parse(stellarStatement, resolver, StellarFunctions.FUNCTION_RESOLVER(), stellarContext);
- if(o != null && o instanceof Map) {
- for(Map.Entry<Object, Object> valueKv : ((Map<Object, Object>)o).entrySet()) {
- String newKey = ((kv.getKey().length() > 0)?kv.getKey() + "." : "" )+ valueKv.getKey();
- message.put(newKey, valueKv.getValue());
- }
- }
- else {
- message.put(kv.getKey(), o);
- }
- }
- }
- }
- JSONObject enriched = new JSONObject(message);
- _LOG.trace("Stellar Enrichment Success: " + enriched);
+ JSONObject enriched = process(message
+ , handler
+ , value.getField()
+ , slowLogThreshold
+ , processor
+ , resolver
+ , stellarContext
+ );
+ _LOG.trace("Stellar Enrichment Success: {}", enriched);
return enriched;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
index 4b88399..61e0b22 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java
@@ -57,7 +57,8 @@ public class EnrichmentJoinBolt extends JoinBolt<JSONObject> {
if(fieldMap != null) {
for (String enrichmentType : fieldMap.keySet()) {
ConfigHandler handler = handlerMap.get(enrichmentType);
- for(String subgroup : handler.getType().getSubgroups(handler.getConfig())) {
+ List<String> subgroups = handler.getType().getSubgroups(handler.getType().toConfig(handler.getConfig()));
+ for(String subgroup : subgroups) {
streamIds.add(Joiner.on(":").join(enrichmentType, subgroup));
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentSplitterBolt.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentSplitterBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentSplitterBolt.java
index f9cad80..6529296 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentSplitterBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentSplitterBolt.java
@@ -124,7 +124,7 @@ public class EnrichmentSplitterBolt extends SplitBolt<JSONObject> {
.splitByFields( message
, fields
, field -> getKeyName(enrichmentType, field)
- , retriever.getConfig()
+ , retriever
);
for(JSONObject eo : enrichmentObject) {
eo.put(Constants.SENSOR_TYPE, sensorType);
http://git-wip-us.apache.org/repos/asf/metron/blob/7d6121bf/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
new file mode 100644
index 0000000..5624d4b
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.adapters.stellar;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.metron.common.configuration.StellarEnrichmentTest;
+import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
+import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.MapVariableResolver;
+import org.apache.metron.common.dsl.VariableResolver;
+import org.apache.metron.common.stellar.StellarProcessor;
+import org.apache.metron.common.utils.JSONUtils;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class StellarAdapterTest extends StellarEnrichmentTest {
+ StellarProcessor processor = new StellarProcessor();
+
+ private JSONObject enrich(JSONObject message, String field, ConfigHandler handler) {
+ VariableResolver resolver = new MapVariableResolver(message);
+ return StellarAdapter.process( message
+ , handler
+ , field
+ , 1000L
+ , processor
+ , resolver
+ , Context.EMPTY_CONTEXT()
+ );
+ }
+
+ @Test
+ public void test_default() throws Exception {
+ for(String c : DEFAULT_CONFIGS) {
+ JSONObject message = getMessage();
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ JSONObject enriched = enrich(message, "", handler);
+ Assert.assertEquals("STELLAR_TEST", enriched.get("stmt1"));
+ Assert.assertEquals("stellar_test", enriched.get("stmt2"));
+ Assert.assertEquals("foo", enriched.get("stmt3"));
+ Assert.assertEquals(3, enriched.size());
+ }
+ }
+
+ @Test
+ public void test_grouped() throws Exception {
+ for(String c : GROUPED_CONFIGS) {
+ JSONObject message = getMessage();
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ {
+ JSONObject enriched = enrich(message, "group1", handler);
+ Assert.assertEquals("STELLAR_TEST", enriched.get("stmt1"));
+ Assert.assertEquals("stellar_test", enriched.get("stmt2"));
+ Assert.assertEquals(2, enriched.size());
+ }
+ {
+ JSONObject enriched = enrich(message, "group2", handler);
+ Assert.assertEquals("foo", enriched.get("stmt3"));
+ Assert.assertEquals(1, enriched.size());
+ }
+ }
+ }
+
+ @Test
+ public void test_mixed() throws Exception {
+ for(String c : MIXED_CONFIGS) {
+ JSONObject message = getMessage();
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(c, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ {
+ JSONObject enriched = enrich(message, "group1", handler);
+ Assert.assertEquals("STELLAR_TEST", enriched.get("stmt1"));
+ Assert.assertEquals("stellar_test", enriched.get("stmt2"));
+ Assert.assertEquals(2, enriched.size());
+ }
+ {
+ JSONObject enriched = enrich(message, "group2", handler);
+ Assert.assertEquals("foo", enriched.get("stmt3"));
+ Assert.assertEquals(1, enriched.size());
+ }
+ {
+ JSONObject enriched = enrich(message, "", handler);
+ Assert.assertEquals(2, enriched.get("stmt4"));
+ Assert.assertEquals("stellar_test", enriched.get("stmt5"));
+ Assert.assertEquals(2, enriched.size());
+ }
+ }
+ }
+
+ @Test
+ public void test_tempVariable() throws Exception {
+ JSONObject message = getMessage();
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(tempVarStellarConfig_list, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ {
+ JSONObject enriched = enrich(message, "group1", handler);
+ Assert.assertEquals("stellar_test", enriched.get("stmt2"));
+ Assert.assertEquals(1, enriched.size());
+ }
+ {
+ JSONObject enriched = enrich(message, "group2", handler);
+ Assert.assertEquals("foo", enriched.get("stmt3"));
+ Assert.assertEquals(1, enriched.size());
+ }
+ {
+ JSONObject enriched = enrich(message, "", handler);
+ Assert.assertEquals(2, enriched.get("stmt4"));
+ Assert.assertEquals("stellar_test", enriched.get("stmt5"));
+ Assert.assertEquals(2, enriched.size());
+ }
+ }
+}
[07/18] metron git commit: METRON-962 Configuration Based Unit Tests
and Add integration tests (justinleet via leet) closes apache/metron#612
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/resources/logData/LancopeParserTest.txt
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/logData/LancopeParserTest.txt b/metron-platform/metron-parsers/src/test/resources/logData/LancopeParserTest.txt
new file mode 100644
index 0000000..0e4bf74
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/logData/LancopeParserTest.txt
@@ -0,0 +1 @@
+{"message":"<131>Jul 17 15:59:01 smc-01 StealthWatch[12365]: 2014-07-17T15:58:30Z 10.40.10.254 0.0.0.0 Minor High Concern Index The host's concern index has either exceeded the CI threshold or rapidly increased. Observed 36.55M points. Policy maximum allows up to 20M points.","@version":"1","@timestamp":"2014-07-17T15:56:05.992Z","type":"syslog","host":"10.122.196.201"}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/resources/logData/PaloAltoFirewallParserTest.txt
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/logData/PaloAltoFirewallParserTest.txt b/metron-platform/metron-parsers/src/test/resources/logData/PaloAltoFirewallParserTest.txt
new file mode 100644
index 0000000..c58bcc8
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/logData/PaloAltoFirewallParserTest.txt
@@ -0,0 +1,2 @@
+<11>Jan 5 05:38:59 PAN1.exampleCustomer.com 1,2015/01/05 05:38:58,0006C110285,THREAT,vulnerability,1,2015/01/05 05:38:58,10.0.0.115,216.0.10.198,0.0.0.0,0.0.0.0,EX-Allow,example\\user.name,,web-browsing,vsys1,internal,external,ethernet1/2,ethernet1/1,LOG-Default,2015/01/05 05:38:58,12031,1,54180,80,0,0,0x80004000,tcp,reset-both,\"ad.aspx?f=300x250&id=12;tile=1;ord=67AF705D60B1119C0F18BEA336F9\",HTTP: IIS Denial Of Service Attempt(40019),any,high,client-to-server,347368099,0x0,10.0.0.0-10.255.255.255,US,0,,1200568889751109656,,
+<14>Jan 5 12:51:34 PAN1.exampleCustomer.com 1,2015/01/05 12:51:33,0011C103117,TRAFFIC,end,1,2015/01/05 12:51:33,10.0.0.39,10.1.0.163,0.0.0.0,0.0.0.0,EX-Allow,,example\\user.name,ms-ds-smb,vsys1,v_external,v_internal,ethernet1/2,ethernet1/1,LOG-Default,2015/01/05 12:51:33,33760927,1,52688,445,0,0,0x401a,tcp,allow,2229,1287,942,10,2015/01/05 12:51:01,30,any,0,17754932062,0x0,10.0.0.0-10.255.255.255,10.0.0.0-10.255.255.255,0,6,4
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/resources/logData/SourcefireParserTest.txt
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/logData/SourcefireParserTest.txt b/metron-platform/metron-parsers/src/test/resources/logData/SourcefireParserTest.txt
new file mode 100644
index 0000000..af257aa
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/logData/SourcefireParserTest.txt
@@ -0,0 +1,3 @@
+SFIMS: [Primary Detection Engine (a7213248-6423-11e3-8537-fac6a92b7d9d)][MTD Access Control] Connection Type: Start, User: Unknown, Client: Unknown, Application Protocol: Unknown, Web App: Unknown, Firewall Rule Name: MTD Access Control, Firewall Rule Action: Allow, Firewall Rule Reasons: Unknown, URL Category: Unknown, URL_Reputation: Risk unknown, URL: Unknown, Interface Ingress: s1p1, Interface Egress: N/A, Security Zone Ingress: Unknown, Security Zone Egress: N/A, Security Intelligence Matching IP: None, Security Intelligence Category: None, {TCP} 72.163.0.129:60517 -> 10.1.128.236:443
+snort: [1:3192:2] WEB-CLIENT Windows Media Player directory traversal via Content-Disposition attempt [Classification: Attempted User Privilege Gain] [Priority: 1] {TCP} 46.149.110.103:80 -> 192.168.56.102:1073
+SFIMS: Correlation Event: Open Soc Log Forwarding/Opensoc Log Forwarding at Thu Oct 23 04:55:39 2014 UTC: [1:19123:7] \"MALWARE-CNC Dropper Win.Trojan.Cefyns.A variant outbound connection\" [Impact: Unknown] From \"172.19.50.7\" at Thu Oct 23 04:55:38 2014 UTC [Classification: A Network Trojan was Detected] [Priority: 1] {tcp} 139.230.245.23:52078->72.52.4.91:80
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/utils/PcapUtilsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/utils/PcapUtilsTest.java b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/utils/PcapUtilsTest.java
index 39fa5fd..5704d76 100644
--- a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/utils/PcapUtilsTest.java
+++ b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/utils/PcapUtilsTest.java
@@ -17,7 +17,7 @@
*/
package org.apache.metron.pcap.utils;
-import junit.framework.Assert;
+import org.junit.Assert;
import org.junit.Test;
public class PcapUtilsTest {
[15/18] metron git commit: METRON-980: Short circuit operations for
Stellar closes apache/metron#606
Posted by ma...@apache.org.
METRON-980: Short circuit operations for Stellar closes apache/metron#606
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/30d0e2a6
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/30d0e2a6
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/30d0e2a6
Branch: refs/heads/Metron_0.4.0
Commit: 30d0e2a6faa1c1205522758e5fd15f35ebc9fcbb
Parents: 7d6121b
Author: cstella <ce...@gmail.com>
Authored: Fri Jun 9 11:48:27 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Fri Jun 9 11:48:27 2017 -0400
----------------------------------------------------------------------
.../profiler/client/window/WindowProcessor.java | 2 +-
metron-platform/metron-common/README.md | 1 +
.../metron/common/stellar/generated/Stellar.g4 | 16 +-
.../org/apache/metron/common/dsl/Token.java | 14 +
.../common/stellar/BaseStellarProcessor.java | 1 +
.../metron/common/stellar/FrameContext.java | 45 ++
.../metron/common/stellar/StellarCompiler.java | 287 +++++--
.../stellar/evaluators/ArithmeticEvaluator.java | 42 +-
...mparisonExpressionWithOperatorEvaluator.java | 9 +-
.../evaluators/DoubleLiteralEvaluator.java | 5 +-
.../evaluators/FloatLiteralEvaluator.java | 5 +-
.../stellar/evaluators/IntLiteralEvaluator.java | 5 +-
.../evaluators/LongLiteralEvaluator.java | 5 +-
.../stellar/evaluators/NumberEvaluator.java | 3 +-
.../evaluators/NumberLiteralEvaluator.java | 8 +-
.../stellar/generated/StellarBaseListener.java | 36 +
.../stellar/generated/StellarListener.java | 30 +
.../common/stellar/generated/StellarParser.java | 774 +++++++++++--------
.../dsl/functions/FunctionalFunctionsTest.java | 17 +
.../common/stellar/StellarInterpreterTest.java | 24 +-
.../metron/common/stellar/StellarTest.java | 101 ++-
.../evaluators/ArithmeticEvaluatorTest.java | 58 +-
...isonExpressionWithOperatorEvaluatorTest.java | 10 +-
.../evaluators/DoubleLiteralEvaluatorTest.java | 8 +-
.../evaluators/FloatLiteralEvaluatorTest.java | 8 +-
.../evaluators/IntLiteralEvaluatorTest.java | 8 +-
.../evaluators/LongLiteralEvaluatorTest.java | 8 +-
.../evaluators/NumberLiteralEvaluatorTest.java | 18 +-
28 files changed, 1029 insertions(+), 519 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java
index 16a7513..1ae4aaf 100644
--- a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java
+++ b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/window/WindowProcessor.java
@@ -89,7 +89,7 @@ public class WindowProcessor extends WindowBaseListener {
if(checkForException(ctx)) {
return;
}
- stack.push(new Token<>(ctx.getText().substring(1), String.class));
+ stack.push(new Token<>(ctx.getText().substring(1), String.class ));
}
/**
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index 9e1de7a..95f59ba 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -26,6 +26,7 @@ The query language supports the following:
* The literal `"\"foo\""` would represent `"foo"`
* The literal `'foo \\ bar'` would represent `foo \ bar`
* Simple boolean operations: `and`, `not`, `or`
+ * Boolean expressions are short-circuited (e.g. `true or FUNC()` would never execute `FUNC`)
* Simple arithmetic operations: `*`, `/`, `+`, `-` on real numbers or integers
* Simple comparison operations `<`, `>`, `<=`, `>=`
* Simple equality comparison operations `==`, `!=`
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4 b/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
index 8602fde..3577377 100644
--- a/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
+++ b/metron-platform/metron-common/src/main/antlr4/org/apache/metron/common/stellar/generated/Stellar.g4
@@ -139,9 +139,21 @@ transformation_expr:
| in_expr #InExpression
;
+if_expr:
+ logical_expr
+ ;
+
+then_expr:
+ transformation_expr
+ ;
+
+else_expr:
+ transformation_expr
+ ;
+
conditional_expr :
- logical_expr QUESTION transformation_expr COLON transformation_expr #TernaryFuncWithoutIf
- | IF logical_expr THEN transformation_expr ELSE transformation_expr #TernaryFuncWithIf
+ if_expr QUESTION then_expr COLON else_expr #TernaryFuncWithoutIf
+ | IF if_expr THEN then_expr ELSE else_expr #TernaryFuncWithIf
;
logical_expr:
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/Token.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/Token.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/Token.java
index 4c94be0..cf168a4 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/Token.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/Token.java
@@ -18,13 +18,27 @@
package org.apache.metron.common.dsl;
+import org.apache.metron.common.stellar.FrameContext;
+
public class Token<T> {
T value;
Class<T> underlyingType;
+ FrameContext.Context multiArgContext;
+
public Token(T value, Class<T> clazz) {
+ this(value, clazz, null);
+ }
+
+ public Token(T value, Class<T> clazz, FrameContext.Context multiArgContext) {
this.value = value;
this.underlyingType = clazz;
+ this.multiArgContext = multiArgContext;
}
+
+ public FrameContext.Context getMultiArgContext() {
+ return multiArgContext;
+ }
+
public T getValue() {
return value;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
index 8f4af93..5bf4a64 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/BaseStellarProcessor.java
@@ -24,6 +24,7 @@ import com.google.common.cache.CacheLoader;
import com.google.common.util.concurrent.UncheckedExecutionException;
import org.antlr.v4.runtime.ANTLRInputStream;
import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.ParserRuleContext;
import org.antlr.v4.runtime.TokenStream;
import java.util.HashSet;
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/FrameContext.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/FrameContext.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/FrameContext.java
new file mode 100644
index 0000000..16bc07a
--- /dev/null
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/FrameContext.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.stellar;
+
+public enum FrameContext {
+ BOOLEAN_AND,
+ BOOLEAN_OR;
+
+ public static class Context {
+ private FrameContext variety;
+ public Context(FrameContext variety) {
+ this.variety = variety;
+ }
+
+ public FrameContext getVariety() {
+ return variety;
+ }
+
+ @Override
+ public String toString() {
+ return "Context{" +
+ "variety=" + variety +
+ '}';
+ }
+ }
+
+ public Context create() {
+ return new Context(this);
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
index b289f48..136728d 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
@@ -48,6 +48,15 @@ public class StellarCompiler extends StellarBaseListener {
private final NumberLiteralEvaluator numberLiteralEvaluator;
private final ComparisonExpressionWithOperatorEvaluator comparisonExpressionWithOperatorEvaluator;
+ public interface ShortCircuitOp {}
+
+ public static class ShortCircuitFrame {}
+ public static class BooleanArg implements ShortCircuitOp {}
+ public static class IfExpr implements ShortCircuitOp {}
+ public static class ThenExpr implements ShortCircuitOp {}
+ public static class ElseExpr implements ShortCircuitOp {}
+ public static class EndConditional implements ShortCircuitOp {}
+
public static class ExpressionState {
Context context;
FunctionResolver functionResolver;
@@ -64,10 +73,18 @@ public class StellarCompiler extends StellarBaseListener {
public static class Expression implements Serializable {
final Deque<Token<?>> tokenDeque;
+ final Deque<FrameContext.Context> multiArgumentState;
final Set<String> variablesUsed;
public Expression(Deque<Token<?>> tokenDeque) {
this.tokenDeque = tokenDeque;
this.variablesUsed = new HashSet<>();
+ this.multiArgumentState = new ArrayDeque<>();
+ }
+
+ public void clear() {
+ tokenDeque.clear();
+ variablesUsed.clear();
+ multiArgumentState.clear();
}
public Deque<Token<?>> getTokenDeque() {
@@ -76,14 +93,77 @@ public class StellarCompiler extends StellarBaseListener {
public Object apply(ExpressionState state) {
Deque<Token<?>> instanceDeque = new ArrayDeque<>();
- for(Iterator<Token<?>> it = getTokenDeque().descendingIterator();it.hasNext();) {
- Token<?> token = it.next();
- if(token.getUnderlyingType() == DeferredFunction.class) {
- DeferredFunction func = (DeferredFunction) token.getValue();
- func.apply(instanceDeque, state);
- }
- else {
- instanceDeque.push(token);
+ {
+ boolean skipElse = false;
+ Token<?> token = null;
+ for (Iterator<Token<?>> it = getTokenDeque().descendingIterator(); it.hasNext(); ) {
+ token = it.next();
+ //if we've skipped an else previously, then we need to skip the deferred tokens associated with the else.
+ if(skipElse && token.getUnderlyingType() == ElseExpr.class) {
+ while(it.hasNext()) {
+ token = it.next();
+ if(token.getUnderlyingType() == EndConditional.class) {
+ break;
+ }
+ }
+ skipElse = false;
+ }
+ /*
+ curr is the current value on the stack. This is the non-deferred actual evaluation for this expression
+ and with the current context.
+ */
+ Token<?> curr = instanceDeque.peek();
+ if( curr != null
+ && curr.getValue() != null && curr.getValue() instanceof Boolean
+ && ShortCircuitOp.class.isAssignableFrom(token.getUnderlyingType())
+ ) {
+ //if we have a boolean as the current value and the next non-contextual token is a short circuit op
+ //then we need to short circuit possibly
+ if(token.getUnderlyingType() == BooleanArg.class) {
+ if (curr.getMultiArgContext() != null
+ && curr.getMultiArgContext().getVariety() == FrameContext.BOOLEAN_OR
+ && (Boolean) (curr.getValue())
+ ) {
+ //short circuit the or
+ FrameContext.Context context = curr.getMultiArgContext();
+ shortCircuit(it, context);
+ } else if (curr.getMultiArgContext() != null
+ && curr.getMultiArgContext().getVariety() == FrameContext.BOOLEAN_AND
+ && !(Boolean) (curr.getValue())
+ ) {
+ //short circuit the and
+ FrameContext.Context context = curr.getMultiArgContext();
+ shortCircuit(it, context);
+ }
+ }
+ else if(token.getUnderlyingType() == IfExpr.class) {
+ //short circuit the if/then/else
+ instanceDeque.pop();
+ if((Boolean)curr.getValue()) {
+ //choose then
+ skipElse = true;
+ }
+ else {
+ //choose else
+ while(it.hasNext()) {
+ Token<?> t = it.next();
+ if(t.getUnderlyingType() == ElseExpr.class) {
+ break;
+ }
+ }
+ }
+ }
+ }
+ if (token.getUnderlyingType() == DeferredFunction.class) {
+ DeferredFunction func = (DeferredFunction) token.getValue();
+ func.apply(instanceDeque, state);
+ }
+ else if(token.getUnderlyingType() != ShortCircuitFrame.class
+ && !ShortCircuitOp.class.isAssignableFrom(token.getUnderlyingType())
+ ) {
+ instanceDeque.push(token);
+ }
+
}
}
@@ -100,6 +180,15 @@ public class StellarCompiler extends StellarBaseListener {
throw new ParseException("Invalid parse, found " + token);
}
}
+
+ public void shortCircuit(Iterator<Token<?>> it, FrameContext.Context context) {
+ while (it.hasNext()) {
+ Token<?> token = it.next();
+ if (token.getUnderlyingType() == ShortCircuitFrame.class && token.getMultiArgContext() == context) {
+ break;
+ }
+ }
+ }
}
interface DeferredFunction {
@@ -130,7 +219,7 @@ public class StellarCompiler extends StellarBaseListener {
@Override
public void enterTransformation(StellarParser.TransformationContext ctx) {
- expression.tokenDeque.clear();
+ expression.clear();
}
private boolean handleIn(final Token<?> left, final Token<?> right) {
@@ -162,39 +251,43 @@ public class StellarCompiler extends StellarBaseListener {
@Override
public void exitNullConst(StellarParser.NullConstContext ctx) {
- expression.tokenDeque.push(new Token<>(null, Object.class));
+ expression.tokenDeque.push(new Token<>(null, Object.class, getArgContext()));
}
@Override
public void exitArithExpr_plus(StellarParser.ArithExpr_plusContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>((tokenDeque, state) -> {
Pair<Token<? extends Number>, Token<? extends Number>> p = getArithExpressionPair(tokenDeque);
- tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p));
- }, DeferredFunction.class));
+ tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(context), p));
+ }, DeferredFunction.class, context));
}
@Override
public void exitArithExpr_minus(StellarParser.ArithExpr_minusContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Pair<Token<? extends Number>, Token<? extends Number>> p = getArithExpressionPair(tokenDeque);
- tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), p));
- }, DeferredFunction.class));
+ tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(context), p));
+ }, DeferredFunction.class, context));
}
@Override
public void exitArithExpr_div(StellarParser.ArithExpr_divContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Pair<Token<? extends Number>, Token<? extends Number>> p = getArithExpressionPair(tokenDeque);
- tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p));
- }, DeferredFunction.class));
+ tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(context), p));
+ }, DeferredFunction.class, context));
}
@Override
public void exitArithExpr_mul(StellarParser.ArithExpr_mulContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Pair<Token<? extends Number>, Token<? extends Number>> p = getArithExpressionPair(tokenDeque);
- tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), p));
- }, DeferredFunction.class));
+ tokenDeque.push(arithmeticEvaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(context), p));
+ }, DeferredFunction.class, context));
}
@SuppressWarnings("unchecked")
@@ -204,61 +297,62 @@ public class StellarCompiler extends StellarBaseListener {
return Pair.of(left, right);
}
- private void handleConditional() {
- expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
- Token<?> elseExpr = popDeque(tokenDeque);
- Token<?> thenExpr = popDeque(tokenDeque);
- Token<?> ifExpr = popDeque(tokenDeque);
- @SuppressWarnings("unchecked") boolean b = ((Token<Boolean>) ifExpr).getValue();
- if (b) {
- tokenDeque.push(thenExpr);
- } else {
- tokenDeque.push(elseExpr);
- }
- }, DeferredFunction.class));
+ @Override
+ public void exitIf_expr(StellarParser.If_exprContext ctx) {
+ expression.tokenDeque.push(new Token<>(new IfExpr(), IfExpr.class, getArgContext()));
}
@Override
- public void exitTernaryFuncWithoutIf(StellarParser.TernaryFuncWithoutIfContext ctx) {
- handleConditional();
+ public void enterThen_expr(StellarParser.Then_exprContext ctx) {
+ expression.tokenDeque.push(new Token<>(new ThenExpr(), ThenExpr.class, getArgContext()));
}
@Override
- public void exitTernaryFuncWithIf(StellarParser.TernaryFuncWithIfContext ctx) {
- handleConditional();
+ public void enterElse_expr(StellarParser.Else_exprContext ctx) {
+ expression.tokenDeque.push(new Token<>(new ElseExpr(), ElseExpr.class, getArgContext()));
+ }
+
+ @Override
+ public void exitElse_expr(StellarParser.Else_exprContext ctx) {
+ expression.tokenDeque.push(new Token<>(new EndConditional(), EndConditional.class, getArgContext()));
}
@Override
public void exitInExpressionStatement(StellarParser.InExpressionStatementContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Token<?> left = popDeque(tokenDeque);
Token<?> right = popDeque(tokenDeque);
- tokenDeque.push(new Token<>(handleIn(left, right), Boolean.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(handleIn(left, right), Boolean.class, context));
+ }, DeferredFunction.class, context));
}
+
@Override
public void exitNInExpressionStatement(StellarParser.NInExpressionStatementContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Token<?> left = popDeque(tokenDeque);
Token<?> right = popDeque(tokenDeque);
- tokenDeque.push(new Token<>(!handleIn(left, right), Boolean.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(!handleIn(left, right), Boolean.class, context));
+ }, DeferredFunction.class, context));
}
@Override
public void exitNotFunc(StellarParser.NotFuncContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Token<Boolean> arg = (Token<Boolean>) popDeque(tokenDeque);
- tokenDeque.push(new Token<>(!arg.getValue(), Boolean.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(!arg.getValue(), Boolean.class, context));
+ }, DeferredFunction.class, context));
}
@Override
public void exitVariable(StellarParser.VariableContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
- tokenDeque.push(new Token<>(state.variableResolver.resolve(ctx.getText()), Object.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(state.variableResolver.resolve(ctx.getText()), Object.class, context));
+ }, DeferredFunction.class, context));
expression.variablesUsed.add(ctx.getText());
}
@@ -266,46 +360,77 @@ public class StellarCompiler extends StellarBaseListener {
public void exitStringLiteral(StellarParser.StringLiteralContext ctx) {
String rawToken = ctx.getText();
String literal = StringEscapeUtils.UNESCAPE_JSON.translate(rawToken);
- expression.tokenDeque.push(new Token<>(literal.substring(1, literal.length()-1), String.class));
+ expression.tokenDeque.push(new Token<>(literal.substring(1, literal.length()-1), String.class, getArgContext()));
}
@Override
public void exitIntLiteral(StellarParser.IntLiteralContext ctx) {
- expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx));
+ expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx, getArgContext()));
}
@Override
public void exitDoubleLiteral(StellarParser.DoubleLiteralContext ctx) {
- expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx));
+ expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx, getArgContext()));
}
@Override
public void exitFloatLiteral(StellarParser.FloatLiteralContext ctx) {
- expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx));
+ expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx, getArgContext()));
}
@Override
public void exitLongLiteral(StellarParser.LongLiteralContext ctx) {
- expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx));
+ expression.tokenDeque.push(numberLiteralEvaluator.evaluate(ctx, getArgContext()));
+ }
+
+ @Override
+ public void enterB_expr(StellarParser.B_exprContext ctx) {
+ //Enter is not guaranteed to be called by Antlr for logical labels, so we need to
+ //emulate it like this. See https://github.com/antlr/antlr4/issues/802
+ if(ctx.getParent() instanceof StellarParser.LogicalExpressionOrContext) {
+ expression.multiArgumentState.push(FrameContext.BOOLEAN_OR.create());
+ }
+ else if(ctx.getParent() instanceof StellarParser.LogicalExpressionAndContext) {
+ expression.multiArgumentState.push(FrameContext.BOOLEAN_AND.create());
+ }
+ }
+
+ @Override
+ public void exitB_expr(StellarParser.B_exprContext ctx) {
+ if(ctx.getParent() instanceof StellarParser.LogicalExpressionOrContext
+ || ctx.getParent() instanceof StellarParser.LogicalExpressionAndContext
+ )
+ {
+ //we want to know when the argument to the boolean expression is complete
+ expression.tokenDeque.push(new Token<>(new BooleanArg(), BooleanArg.class, getArgContext()));
+ }
}
@Override
public void exitLogicalExpressionAnd(StellarParser.LogicalExpressionAndContext ctx) {
+ final FrameContext.Context context = getArgContext();
+ popArgContext();
+ final FrameContext.Context parentContext = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Token<?> left = popDeque(tokenDeque);
Token<?> right = popDeque(tokenDeque);
- tokenDeque.push(new Token<>(booleanOp(left, right, (l, r) -> l && r, "&&"), Boolean.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(booleanOp(left, right, (l, r) -> l && r, "&&"), Boolean.class, parentContext));
+ }, DeferredFunction.class, context));
+ expression.tokenDeque.push(new Token<>(new ShortCircuitFrame(), ShortCircuitFrame.class, context));
}
@Override
public void exitLogicalExpressionOr(StellarParser.LogicalExpressionOrContext ctx) {
+ final FrameContext.Context context = getArgContext();
+ popArgContext();
+ final FrameContext.Context parentContext = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
Token<?> left = popDeque(tokenDeque);
Token<?> right = popDeque(tokenDeque);
- tokenDeque.push(new Token<>(booleanOp(left, right, (l, r) -> l || r, "||"), Boolean.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(booleanOp(left, right, (l, r) -> l || r, "||"), Boolean.class, parentContext));
+ }, DeferredFunction.class, context));
+ expression.tokenDeque.push(new Token<>(new ShortCircuitFrame(), ShortCircuitFrame.class, context));
}
@Override
@@ -321,7 +446,7 @@ public class StellarCompiler extends StellarBaseListener {
default:
throw new ParseException("Unable to process " + ctx.getText() + " as a boolean constant");
}
- expression.tokenDeque.push(new Token<>(b, Boolean.class));
+ expression.tokenDeque.push(new Token<>(b, Boolean.class, getArgContext()));
}
private boolean booleanOp(final Token<?> left, final Token<?> right, final BooleanOp op, final String opName) {
@@ -356,7 +481,7 @@ public class StellarCompiler extends StellarBaseListener {
@Override
public void exitLambda_variable(StellarParser.Lambda_variableContext ctx) {
- expression.tokenDeque.push(new Token<>(ctx.getText(), String.class));
+ expression.tokenDeque.push(new Token<>(ctx.getText(), String.class, getArgContext()));
}
private void enterLambdaVariables() {
@@ -369,7 +494,7 @@ public class StellarCompiler extends StellarBaseListener {
for(; !expression.tokenDeque.isEmpty() && t != LAMBDA_VARIABLES; t = expression.tokenDeque.pop()) {
variables.addFirst(t.getValue().toString());
}
- expression.tokenDeque.push(new Token<>(variables, List.class));
+ expression.tokenDeque.push(new Token<>(variables, List.class, getArgContext()));
}
private void enterLambda() {
@@ -377,6 +502,7 @@ public class StellarCompiler extends StellarBaseListener {
}
private void exitLambda(boolean hasArgs) {
+ final FrameContext.Context context = getArgContext();
Token<?> t = expression.tokenDeque.pop();
final Deque<Token<?>> instanceDeque = new ArrayDeque<>();
for(; !expression.tokenDeque.isEmpty() && t != EXPRESSION_REFERENCE; t = expression.tokenDeque.pop()) {
@@ -385,8 +511,8 @@ public class StellarCompiler extends StellarBaseListener {
final List<String> variables = hasArgs? (List<String>) instanceDeque.removeLast().getValue() :new ArrayList<>();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
LambdaExpression expr = new LambdaExpression(variables, instanceDeque, state);
- tokenDeque.push(new Token<>(expr, Object.class));
- }, DeferredFunction.class) );
+ tokenDeque.push(new Token<>(expr, Object.class, context));
+ }, DeferredFunction.class, context) );
}
@Override
@@ -411,7 +537,7 @@ public class StellarCompiler extends StellarBaseListener {
@Override
public void exitTransformationFunc(StellarParser.TransformationFuncContext ctx) {
-
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
// resolve and initialize the function
String functionName = ctx.getChild(0).getText();
@@ -421,8 +547,8 @@ public class StellarCompiler extends StellarBaseListener {
// fetch the args, execute, and push result onto the stack
List<Object> args = getFunctionArguments(popDeque(tokenDeque));
Object result = function.apply(args, state.context);
- tokenDeque.push(new Token<>(result, Object.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(result, Object.class, context));
+ }, DeferredFunction.class, context));
}
/**
@@ -474,22 +600,24 @@ public class StellarCompiler extends StellarBaseListener {
@Override
public void exitExistsFunc(StellarParser.ExistsFuncContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
String variable = ctx.getChild(2).getText();
boolean exists = state.variableResolver.resolve(variable) != null;
- tokenDeque.push(new Token<>(exists, Boolean.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(exists, Boolean.class, context));
+ }, DeferredFunction.class, context));
String variable = ctx.getChild(2).getText();
expression.variablesUsed.add(variable);
}
@Override
public void enterFunc_args(StellarParser.Func_argsContext ctx) {
- expression.tokenDeque.push(new Token<>(new FunctionMarker(), FunctionMarker.class));
+ expression.tokenDeque.push(new Token<>(new FunctionMarker(), FunctionMarker.class, getArgContext()));
}
@Override
public void exitFunc_args(StellarParser.Func_argsContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>((tokenDeque, state) -> {
LinkedList<Object> args = new LinkedList<>();
while (true) {
@@ -500,17 +628,18 @@ public class StellarCompiler extends StellarBaseListener {
args.addFirst(token.getValue());
}
}
- tokenDeque.push(new Token<>(args, List.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(args, List.class, context));
+ }, DeferredFunction.class, context));
}
@Override
public void enterMap_entity(StellarParser.Map_entityContext ctx) {
- expression.tokenDeque.push(new Token<>(new FunctionMarker(), FunctionMarker.class));
+ expression.tokenDeque.push(new Token<>(new FunctionMarker(), FunctionMarker.class, getArgContext()));
}
@Override
public void exitMap_entity(StellarParser.Map_entityContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
HashMap<String, Object> args = new HashMap<>();
Object value = null;
@@ -526,12 +655,13 @@ public class StellarCompiler extends StellarBaseListener {
}
}
}
- tokenDeque.push(new Token<>(args, Map.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(args, Map.class, context));
+ }, DeferredFunction.class, context));
}
@Override
public void exitList_entity(StellarParser.List_entityContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
LinkedList<Object> args = new LinkedList<>();
while (true) {
@@ -542,24 +672,37 @@ public class StellarCompiler extends StellarBaseListener {
args.addFirst(token.getValue());
}
}
- tokenDeque.push(new Token<>(args, List.class));
- }, DeferredFunction.class));
+ tokenDeque.push(new Token<>(args, List.class, context));
+ }, DeferredFunction.class, context));
}
+
+
@Override
public void exitComparisonExpressionWithOperator(StellarParser.ComparisonExpressionWithOperatorContext ctx) {
+ final FrameContext.Context context = getArgContext();
expression.tokenDeque.push(new Token<>( (tokenDeque, state) -> {
StellarParser.Comp_operatorContext op = ctx.comp_operator();
Token<?> right = popDeque(tokenDeque);
Token<?> left = popDeque(tokenDeque);
- tokenDeque.push(comparisonExpressionWithOperatorEvaluator.evaluate(left, right, (StellarParser.ComparisonOpContext) op));
- }, DeferredFunction.class));
+ tokenDeque.push(comparisonExpressionWithOperatorEvaluator.evaluate(left, right, (StellarParser.ComparisonOpContext) op, context));
+ }, DeferredFunction.class, context));
}
@Override
public void enterList_entity(StellarParser.List_entityContext ctx) {
- expression.tokenDeque.push(new Token<>(new FunctionMarker(), FunctionMarker.class));
+ expression.tokenDeque.push(new Token<>(new FunctionMarker(), FunctionMarker.class, getArgContext()));
+ }
+
+ private void popArgContext() {
+ if(!expression.multiArgumentState.isEmpty()) {
+ expression.multiArgumentState.pop();
+ }
+ }
+
+ private FrameContext.Context getArgContext() {
+ return expression.multiArgumentState.isEmpty()?null:expression.multiArgumentState.peek();
}
private Token<?> popDeque(Deque<Token<?>> tokenDeque) {
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluator.java
index f1264e2..b458f9c 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluator.java
@@ -19,8 +19,8 @@
package org.apache.metron.common.stellar.evaluators;
import org.apache.commons.lang3.tuple.Pair;
-import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import java.util.function.BiFunction;
@@ -44,58 +44,58 @@ public enum ArithmeticEvaluator {
* types is taken for the Java spec: http://docs.oracle.com/javase/specs/jls/se8/html/jls-5.html#jls-5.6.2
*/
public static final class ArithmeticEvaluatorFunctions {
- public static BiFunction<Number, Number, Token<? extends Number>> addition() {
+ public static BiFunction<Number, Number, Token<? extends Number>> addition(final FrameContext.Context context) {
return (Number l, Number r) -> {
if (l instanceof Double || r instanceof Double) {
- return new Token<>(l.doubleValue() + r.doubleValue(), Double.class);
+ return new Token<>(l.doubleValue() + r.doubleValue(), Double.class, context);
} else if (l instanceof Float || r instanceof Float) {
- return new Token<>(l.floatValue() + r.floatValue(), Float.class);
+ return new Token<>(l.floatValue() + r.floatValue(), Float.class, context);
} else if (l instanceof Long || r instanceof Long) {
- return new Token<>(l.longValue() + r.longValue(), Long.class);
+ return new Token<>(l.longValue() + r.longValue(), Long.class, context);
} else {
- return new Token<>(l.intValue() + r.intValue(), Integer.class);
+ return new Token<>(l.intValue() + r.intValue(), Integer.class, context);
}
};
}
- public static BiFunction<Number, Number, Token<? extends Number>> multiplication() {
+ public static BiFunction<Number, Number, Token<? extends Number>> multiplication(final FrameContext.Context context) {
return (Number l, Number r) -> {
if (l instanceof Double || r instanceof Double) {
- return new Token<>(l.doubleValue() * r.doubleValue(), Double.class);
+ return new Token<>(l.doubleValue() * r.doubleValue(), Double.class, context);
} else if (l instanceof Float || r instanceof Float) {
- return new Token<>(l.floatValue() * r.floatValue(), Float.class);
+ return new Token<>(l.floatValue() * r.floatValue(), Float.class, context);
} else if (l instanceof Long || r instanceof Long) {
- return new Token<>(l.longValue() * r.longValue(), Long.class);
+ return new Token<>(l.longValue() * r.longValue(), Long.class, context);
} else {
- return new Token<>(l.intValue() * r.intValue(), Integer.class);
+ return new Token<>(l.intValue() * r.intValue(), Integer.class, context);
}
};
}
- public static BiFunction<Number, Number, Token<? extends Number>> subtraction() {
+ public static BiFunction<Number, Number, Token<? extends Number>> subtraction(final FrameContext.Context context) {
return (Number l, Number r) -> {
if (l instanceof Double || r instanceof Double) {
- return new Token<>(l.doubleValue() - r.doubleValue(), Double.class);
+ return new Token<>(l.doubleValue() - r.doubleValue(), Double.class, context);
} else if (l instanceof Float || r instanceof Float) {
- return new Token<>(l.floatValue() - r.floatValue(), Float.class);
+ return new Token<>(l.floatValue() - r.floatValue(), Float.class, context);
} else if (l instanceof Long || r instanceof Long) {
- return new Token<>(l.longValue() - r.longValue(), Long.class);
+ return new Token<>(l.longValue() - r.longValue(), Long.class, context);
} else {
- return new Token<>(l.intValue() - r.intValue(), Integer.class);
+ return new Token<>(l.intValue() - r.intValue(), Integer.class, context);
}
};
}
- public static BiFunction<Number, Number, Token<? extends Number>> division() {
+ public static BiFunction<Number, Number, Token<? extends Number>> division(FrameContext.Context context) {
return (Number l, Number r) -> {
if (l instanceof Double || r instanceof Double) {
- return new Token<>(l.doubleValue() / r.doubleValue(), Double.class);
+ return new Token<>(l.doubleValue() / r.doubleValue(), Double.class, context);
} else if (l instanceof Float || r instanceof Float) {
- return new Token<>(l.floatValue() / r.floatValue(), Float.class);
+ return new Token<>(l.floatValue() / r.floatValue(), Float.class, context);
} else if (l instanceof Long || r instanceof Long) {
- return new Token<>(l.longValue() / r.longValue(), Long.class);
+ return new Token<>(l.longValue() / r.longValue(), Long.class, context);
} else {
- return new Token<>(l.intValue() / r.intValue(), Integer.class);
+ return new Token<>(l.intValue() / r.intValue(), Integer.class, context);
}
};
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluator.java
index 4862684..e213eb5 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluator.java
@@ -20,6 +20,7 @@ package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
/**
@@ -81,13 +82,13 @@ public enum ComparisonExpressionWithOperatorEvaluator {
* @param op The operator in the Stellar expression.
* @return A token with type boolean. This is based on the comparison of the {@code right} and {@code left} values.
*/
- public Token<Boolean> evaluate(final Token<?> left, final Token<?> right, final StellarParser.ComparisonOpContext op) {
+ public Token<Boolean> evaluate(final Token<?> left, final Token<?> right, final StellarParser.ComparisonOpContext op, FrameContext.Context context) {
if (op.EQ() != null) {
- return new Token<>(Strategy.EQUALITY_OPERATORS.evaluator().evaluate(left, right, op), Boolean.class);
+ return new Token<>(Strategy.EQUALITY_OPERATORS.evaluator().evaluate(left, right, op), Boolean.class, context);
} else if (op.NEQ() != null) {
- return new Token<>(!Strategy.EQUALITY_OPERATORS.evaluator().evaluate(left, right, op), Boolean.class);
+ return new Token<>(!Strategy.EQUALITY_OPERATORS.evaluator().evaluate(left, right, op), Boolean.class, context);
} else if (op.LT() != null || op.GT() != null || op.LTE() != null || op.GTE() != null) {
- return new Token<>(Strategy.COMPARISON_OPERATORS.evaluator().evaluate(left, right, op), Boolean.class);
+ return new Token<>(Strategy.COMPARISON_OPERATORS.evaluator().evaluate(left, right, op), Boolean.class, context);
}
throw new ParseException("Unsupported operations. The following expression is invalid: " + left.getValue() + op.getText() + right.getValue());
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluator.java
index 1bd1a49..653b56c 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluator.java
@@ -19,15 +19,16 @@
package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
public class DoubleLiteralEvaluator implements NumberEvaluator<StellarParser.DoubleLiteralContext> {
@Override
- public Token<Double> evaluate(StellarParser.DoubleLiteralContext context) {
+ public Token<Double> evaluate(StellarParser.DoubleLiteralContext context, FrameContext.Context contextVariety) {
if (context == null) {
throw new IllegalArgumentException("Cannot evaluate a context that is null.");
}
- return new Token<>(Double.parseDouble(context.getText()), Double.class);
+ return new Token<>(Double.parseDouble(context.getText()), Double.class, contextVariety);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluator.java
index 60b312b..4489d30 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluator.java
@@ -19,15 +19,16 @@
package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
public class FloatLiteralEvaluator implements NumberEvaluator<StellarParser.FloatLiteralContext> {
@Override
- public Token<Float> evaluate(StellarParser.FloatLiteralContext context) {
+ public Token<Float> evaluate(StellarParser.FloatLiteralContext context, FrameContext.Context contextVariety) {
if (context == null) {
throw new IllegalArgumentException("Cannot evaluate a context that is null.");
}
- return new Token<>(Float.parseFloat(context.getText()), Float.class);
+ return new Token<>(Float.parseFloat(context.getText()), Float.class, contextVariety);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluator.java
index c928a9a..00e62e6 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluator.java
@@ -19,15 +19,16 @@
package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
public class IntLiteralEvaluator implements NumberEvaluator<StellarParser.IntLiteralContext> {
@Override
- public Token<Integer> evaluate(StellarParser.IntLiteralContext context) {
+ public Token<Integer> evaluate(StellarParser.IntLiteralContext context, FrameContext.Context contextVariety) {
if (context == null) {
throw new IllegalArgumentException("Cannot evaluate a context that is null.");
}
- return new Token<>(Integer.parseInt(context.getText()), Integer.class);
+ return new Token<>(Integer.parseInt(context.getText()), Integer.class, contextVariety);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluator.java
index 05ee162..338adc4 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluator.java
@@ -20,11 +20,12 @@ package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
public class LongLiteralEvaluator implements NumberEvaluator<StellarParser.LongLiteralContext> {
@Override
- public Token<Long> evaluate(StellarParser.LongLiteralContext context) {
+ public Token<Long> evaluate(StellarParser.LongLiteralContext context, FrameContext.Context contextVariety) {
if (context == null) {
throw new IllegalArgumentException("Cannot evaluate a context that is null.");
}
@@ -32,7 +33,7 @@ public class LongLiteralEvaluator implements NumberEvaluator<StellarParser.LongL
String value = context.getText();
if (value.endsWith("l") || value.endsWith("L")) {
value = value.substring(0, value.length() - 1); // Drop the 'L' or 'l'. Long.parseLong does not accept a string with either of these.
- return new Token<>(Long.parseLong(value), Long.class);
+ return new Token<>(Long.parseLong(value), Long.class, contextVariety);
} else {
// Technically this should never happen, but just being safe.
throw new ParseException("Invalid format for long. Failed trying to parse a long with the following value: " + value);
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberEvaluator.java
index 0007b72..2c18f9d 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberEvaluator.java
@@ -19,8 +19,9 @@
package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
public interface NumberEvaluator<T extends StellarParser.Arithmetic_operandsContext> {
- Token<? extends Number> evaluate(T context);
+ Token<? extends Number> evaluate(T context, FrameContext.Context contextVariety);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluator.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluator.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluator.java
index 83d795f..0ec3859 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluator.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluator.java
@@ -20,6 +20,7 @@ package org.apache.metron.common.stellar.evaluators;
import org.apache.metron.common.dsl.ParseException;
import org.apache.metron.common.dsl.Token;
+import org.apache.metron.common.stellar.FrameContext;
import org.apache.metron.common.stellar.generated.StellarParser;
import java.util.HashMap;
@@ -53,17 +54,18 @@ public enum NumberLiteralEvaluator {
Token<? extends Number> evaluate(StellarParser.Arithmetic_operandsContext context
, Map<Class<? extends StellarParser.Arithmetic_operandsContext>, NumberEvaluator> instanceMap
+ , FrameContext.Context contextVariety
)
{
NumberEvaluator evaluator = instanceMap.get(context.getClass());
if(evaluator == null) {
throw new ParseException("Does not support evaluation for type " + context.getClass());
}
- return evaluator.evaluate(context);
+ return evaluator.evaluate(context, contextVariety);
}
- public Token<? extends Number> evaluate(StellarParser.Arithmetic_operandsContext context) {
- return evaluate(context, Strategy.strategyMap);
+ public Token<? extends Number> evaluate(StellarParser.Arithmetic_operandsContext context, FrameContext.Context contextVariety) {
+ return evaluate(context, Strategy.strategyMap, contextVariety);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarBaseListener.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarBaseListener.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarBaseListener.java
index 2be710b..3ca1daa 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarBaseListener.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarBaseListener.java
@@ -133,6 +133,42 @@ public class StellarBaseListener implements StellarListener {
*
* <p>The default implementation does nothing.</p>
*/
+ @Override public void enterIf_expr(StellarParser.If_exprContext ctx) { }
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The default implementation does nothing.</p>
+ */
+ @Override public void exitIf_expr(StellarParser.If_exprContext ctx) { }
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The default implementation does nothing.</p>
+ */
+ @Override public void enterThen_expr(StellarParser.Then_exprContext ctx) { }
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The default implementation does nothing.</p>
+ */
+ @Override public void exitThen_expr(StellarParser.Then_exprContext ctx) { }
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The default implementation does nothing.</p>
+ */
+ @Override public void enterElse_expr(StellarParser.Else_exprContext ctx) { }
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The default implementation does nothing.</p>
+ */
+ @Override public void exitElse_expr(StellarParser.Else_exprContext ctx) { }
+ /**
+ * {@inheritDoc}
+ *
+ * <p>The default implementation does nothing.</p>
+ */
@Override public void enterTernaryFuncWithoutIf(StellarParser.TernaryFuncWithoutIfContext ctx) { }
/**
* {@inheritDoc}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarListener.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarListener.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarListener.java
index 66493f7..6c00691 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarListener.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarListener.java
@@ -123,6 +123,36 @@ public interface StellarListener extends ParseTreeListener {
*/
void exitInExpression(StellarParser.InExpressionContext ctx);
/**
+ * Enter a parse tree produced by {@link StellarParser#if_expr}.
+ * @param ctx the parse tree
+ */
+ void enterIf_expr(StellarParser.If_exprContext ctx);
+ /**
+ * Exit a parse tree produced by {@link StellarParser#if_expr}.
+ * @param ctx the parse tree
+ */
+ void exitIf_expr(StellarParser.If_exprContext ctx);
+ /**
+ * Enter a parse tree produced by {@link StellarParser#then_expr}.
+ * @param ctx the parse tree
+ */
+ void enterThen_expr(StellarParser.Then_exprContext ctx);
+ /**
+ * Exit a parse tree produced by {@link StellarParser#then_expr}.
+ * @param ctx the parse tree
+ */
+ void exitThen_expr(StellarParser.Then_exprContext ctx);
+ /**
+ * Enter a parse tree produced by {@link StellarParser#else_expr}.
+ * @param ctx the parse tree
+ */
+ void enterElse_expr(StellarParser.Else_exprContext ctx);
+ /**
+ * Exit a parse tree produced by {@link StellarParser#else_expr}.
+ * @param ctx the parse tree
+ */
+ void exitElse_expr(StellarParser.Else_exprContext ctx);
+ /**
* Enter a parse tree produced by the {@code TernaryFuncWithoutIf}
* labeled alternative in {@link StellarParser#conditional_expr}.
* @param ctx the parse tree
[16/18] metron git commit: METRON-906 Rest service storm
configuration does not allow for proper URLs (justinleet via ottobackwards)
closes apache/metron#602
Posted by ma...@apache.org.
METRON-906 Rest service storm configuration does not allow for proper URLs (justinleet via ottobackwards) closes apache/metron#602
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/f523c179
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/f523c179
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/f523c179
Branch: refs/heads/Metron_0.4.0
Commit: f523c179585bea3a61f99c694d2752fc6ceb7a3b
Parents: 30d0e2a
Author: justinleet <ju...@gmail.com>
Authored: Fri Jun 16 10:16:27 2017 -0400
Committer: otto <ot...@apache.org>
Committed: Fri Jun 16 10:16:27 2017 -0400
----------------------------------------------------------------------
.../METRON/CURRENT/configuration/metron-env.xml | 2 +-
.../METRON/CURRENT/service_advisor.py | 6 +-
.../roles/ambari_config/vars/single_node_vm.yml | 2 +-
.../roles/ambari_config/vars/small_cluster.yml | 2 +-
.../service/impl/StormStatusServiceImpl.java | 19 +++--
.../src/main/resources/application-vagrant.yml | 2 +-
.../metron/rest/mock/MockStormRestTemplate.java | 13 +++-
.../impl/StormStatusServiceImplTest.java | 73 +++++++++++++-------
8 files changed, 83 insertions(+), 36 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
index 3e5a405..00c48a6 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-env.xml
@@ -224,7 +224,7 @@
<property require-input = "true">
<name>storm_rest_addr</name>
<display-name>Storm Rest Server Address</display-name>
- <description>URL of Storm UI (storm.ui.hostname:8744)</description>
+ <description>URL of Storm UI (http://storm.ui.hostname:8744). If no protocol is provided, http is assumed.</description>
<value></value>
</property>
<property require-input = "true">
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
----------------------------------------------------------------------
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index 7da06f5..4a95e63 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -108,7 +108,11 @@ class METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
if "storm-site" in services["configurations"]:
stormUIServerHost = self.getComponentHostNames(services, "STORM", "STORM_UI_SERVER")[0]
stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"]
- stormUIServerURL = stormUIServerHost + ":" + stormUIServerPort
+ stormUIProtocol = "http://"
+ if "ui.https.port" in services["configurations"]["storm-site"]["properties"]:
+ stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.https.port"]
+ stormUIProtocol = "https://"
+ stormUIServerURL = stormUIProtocol + stormUIServerHost + ":" + stormUIServerPort
putMetronEnvProperty = self.putProperty(configurations, "metron-env", services)
putMetronEnvProperty("storm_rest_addr",stormUIServerURL)
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
index 7e54db3..6a36fac 100644
--- a/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
+++ b/metron-deployment/roles/ambari_config/vars/single_node_vm.yml
@@ -103,7 +103,7 @@ configurations:
required_configurations:
- metron-env:
- storm_rest_addr: "{{ groups.ambari_slave[0] }}:8744"
+ storm_rest_addr: "http://{{ groups.ambari_slave[0] }}:8744"
es_hosts: "{{ groups.search | join(',') }}"
zeppelin_server_url: "{{ groups.zeppelin[0] }}:9995"
metron_jdbc_driver: "org.h2.Driver"
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-deployment/roles/ambari_config/vars/small_cluster.yml
----------------------------------------------------------------------
diff --git a/metron-deployment/roles/ambari_config/vars/small_cluster.yml b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
index dbcfb60..af22cb6 100644
--- a/metron-deployment/roles/ambari_config/vars/small_cluster.yml
+++ b/metron-deployment/roles/ambari_config/vars/small_cluster.yml
@@ -96,7 +96,7 @@ configurations:
required_configurations:
- metron-env:
- storm_rest_addr: "{{ groups.ambari_slave[1] }}:8744"
+ storm_rest_addr: "http://{{ groups.ambari_slave[1] }}:8744"
es_hosts: "{{ groups.web[0] }},{{ groups.search | join(',') }}"
zeppelin_server_url: "{{ groups.zeppelin[0] }}"
metron_jdbc_driver: "org.h2.Driver"
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java
index a02ee75..9f89a3c 100644
--- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java
+++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormStatusServiceImpl.java
@@ -50,7 +50,7 @@ public class StormStatusServiceImpl implements StormStatusService {
@Override
public TopologySummary getTopologySummary() {
- return restTemplate.getForObject("http://" + environment.getProperty(STORM_UI_SPRING_PROPERTY) + TOPOLOGY_SUMMARY_URL, TopologySummary.class);
+ return restTemplate.getForObject(getStormUiProperty() + TOPOLOGY_SUMMARY_URL, TopologySummary.class);
}
@Override
@@ -64,7 +64,7 @@ public class StormStatusServiceImpl implements StormStatusService {
}
}
if (id != null) {
- topologyResponse = restTemplate.getForObject("http://" + environment.getProperty(STORM_UI_SPRING_PROPERTY) + TOPOLOGY_URL + "/" + id, TopologyStatus.class);
+ topologyResponse = restTemplate.getForObject(getStormUiProperty() + TOPOLOGY_URL + "/" + id, TopologyStatus.class);
}
return topologyResponse;
}
@@ -73,7 +73,7 @@ public class StormStatusServiceImpl implements StormStatusService {
public List<TopologyStatus> getAllTopologyStatus() {
List<TopologyStatus> topologyStatus = new ArrayList<>();
for (TopologyStatus topology : getTopologySummary().getTopologies()) {
- topologyStatus.add(restTemplate.getForObject("http://" + environment.getProperty(STORM_UI_SPRING_PROPERTY) + TOPOLOGY_URL + "/" + topology.getId(), TopologyStatus.class));
+ topologyStatus.add(restTemplate.getForObject(getStormUiProperty() + TOPOLOGY_URL + "/" + topology.getId(), TopologyStatus.class));
}
return topologyStatus;
}
@@ -89,7 +89,7 @@ public class StormStatusServiceImpl implements StormStatusService {
}
}
if (id != null) {
- Map result = restTemplate.postForObject("http://" + environment.getProperty(STORM_UI_SPRING_PROPERTY) + TOPOLOGY_URL + "/" + id + "/activate", null, Map.class);
+ Map result = restTemplate.postForObject(getStormUiProperty() + TOPOLOGY_URL + "/" + id + "/activate", null, Map.class);
if("success".equals(result.get("status"))) {
topologyResponse.setSuccessMessage(TopologyStatusCode.ACTIVE.toString());
} else {
@@ -112,7 +112,7 @@ public class StormStatusServiceImpl implements StormStatusService {
}
}
if (id != null) {
- Map result = restTemplate.postForObject("http://" + environment.getProperty(STORM_UI_SPRING_PROPERTY) + TOPOLOGY_URL + "/" + id + "/deactivate", null, Map.class);
+ Map result = restTemplate.postForObject(getStormUiProperty() + TOPOLOGY_URL + "/" + id + "/deactivate", null, Map.class);
if("success".equals(result.get("status"))) {
topologyResponse.setSuccessMessage(TopologyStatusCode.INACTIVE.toString());
} else {
@@ -123,4 +123,13 @@ public class StormStatusServiceImpl implements StormStatusService {
}
return topologyResponse;
}
+
+ // If we don't have a protocol, choose http
+ protected String getStormUiProperty() {
+ String baseValue = environment.getProperty(STORM_UI_SPRING_PROPERTY);
+ if(!(baseValue.contains("://"))) {
+ return "http://" + baseValue;
+ }
+ return baseValue;
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
index c4964c0..1884ca3 100644
--- a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
+++ b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml
@@ -42,7 +42,7 @@ grok:
storm:
ui:
- url: node1:8744
+ url: http://node1:8744
parser:
script.path: /usr/metron/${metron.version}/bin/start_parser_topology.sh
enrichment:
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
index ca5c1c9..ebdefcd 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/mock/MockStormRestTemplate.java
@@ -45,7 +45,7 @@ public class MockStormRestTemplate extends RestTemplate {
@Override
public Object getForObject(String url, Class responseType, Object... urlVariables) throws RestClientException {
Object response = null;
- if (url.equals("http://" + environment.getProperty(MetronRestConstants.STORM_UI_SPRING_PROPERTY) + MetronRestConstants.TOPOLOGY_SUMMARY_URL)) {
+ if (url.equals(getStormUiProperty() + MetronRestConstants.TOPOLOGY_SUMMARY_URL)) {
TopologySummary topologySummary = new TopologySummary();
List<TopologyStatus> topologyStatusList = new ArrayList<>();
for(String name: mockStormCLIClientWrapper.getParserTopologyNames()) {
@@ -61,7 +61,7 @@ public class MockStormRestTemplate extends RestTemplate {
}
topologySummary.setTopologies(topologyStatusList.toArray(new TopologyStatus[topologyStatusList.size()]));
response = topologySummary;
- } else if (url.startsWith("http://" + environment.getProperty(MetronRestConstants.STORM_UI_SPRING_PROPERTY) + MetronRestConstants.TOPOLOGY_URL + "/")){
+ } else if (url.startsWith(getStormUiProperty() + MetronRestConstants.TOPOLOGY_URL + "/")){
String name = url.substring(url.lastIndexOf('/') + 1, url.length()).replaceFirst("-id", "");
response = getTopologyStatus(name);
}
@@ -113,4 +113,13 @@ public class MockStormRestTemplate extends RestTemplate {
}
return result;
}
+
+ // If we don't have a protocol, prepend one
+ protected String getStormUiProperty() {
+ String baseValue = environment.getProperty(MetronRestConstants.STORM_UI_SPRING_PROPERTY);
+ if(!(baseValue.contains("://"))) {
+ return "http://" + baseValue;
+ }
+ return baseValue;
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/f523c179/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormStatusServiceImplTest.java
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormStatusServiceImplTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormStatusServiceImplTest.java
index db2cb7f..606fed4 100644
--- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormStatusServiceImplTest.java
+++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormStatusServiceImplTest.java
@@ -42,6 +42,10 @@ import static org.mockito.Mockito.when;
@SuppressWarnings("ALL")
public class StormStatusServiceImplTest {
+
+ private static final String HTTP_STORM_UI = "http://storm_ui";
+ private static final String HTTPS_STORM_UI = "https://storm_ui";
+ private static final String NO_PROTOCOL_STORM_UI = "storm_ui";
@Rule
public final ExpectedException exception = ExpectedException.none();
@@ -57,6 +61,27 @@ public class StormStatusServiceImplTest {
}
@Test
+ public void testgetStormUiPropertyHttp() {
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ StormStatusServiceImpl serviceImpl = (StormStatusServiceImpl) stormStatusService;
+ assertEquals(HTTP_STORM_UI, serviceImpl.getStormUiProperty());
+ }
+
+ @Test
+ public void testgetStormUiPropertyHttps() {
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTPS_STORM_UI);
+ StormStatusServiceImpl serviceImpl = (StormStatusServiceImpl) stormStatusService;
+ assertEquals(HTTPS_STORM_UI, serviceImpl.getStormUiProperty());
+ }
+
+ @Test
+ public void testgetStormUiPropertyNoProtocol() {
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(NO_PROTOCOL_STORM_UI);
+ StormStatusServiceImpl serviceImpl = (StormStatusServiceImpl) stormStatusService;
+ assertEquals(HTTP_STORM_UI, serviceImpl.getStormUiProperty());
+ }
+
+ @Test
public void getTopologySummaryShouldReturnTopologySummary() throws Exception {
final TopologyStatus topologyStatus = new TopologyStatus();
topologyStatus.setStatus(TopologyStatusCode.STARTED);
@@ -65,8 +90,8 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
TopologyStatus expectedStatus = new TopologyStatus();
expectedStatus.setStatus(TopologyStatusCode.STARTED);
@@ -89,9 +114,9 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_URL + "/bro_id", TopologyStatus.class)).thenReturn(topologyStatus);
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_URL + "/bro_id", TopologyStatus.class)).thenReturn(topologyStatus);
TopologyStatus expected = new TopologyStatus();
expected.setStatus(TopologyStatusCode.STARTED);
@@ -112,9 +137,9 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_URL + "/bro_id", TopologyStatus.class)).thenReturn(topologyStatus);
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_URL + "/bro_id", TopologyStatus.class)).thenReturn(topologyStatus);
TopologyStatus expected = new TopologyStatus();
expected.setStatus(TopologyStatusCode.STARTED);
@@ -133,9 +158,9 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
- when(restTemplate.postForObject("http://storm_ui" + TOPOLOGY_URL + "/bro_id/activate", null, Map.class))
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(restTemplate.postForObject(HTTP_STORM_UI + TOPOLOGY_URL + "/bro_id/activate", null, Map.class))
.thenReturn(new HashMap() {{ put("status", "success"); }});
TopologyResponse expected = new TopologyResponse();
@@ -151,9 +176,9 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
- when(restTemplate.postForObject("http://storm_ui" + TOPOLOGY_URL + "/bro_id/activate", null, Map.class))
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(restTemplate.postForObject(HTTP_STORM_UI + TOPOLOGY_URL + "/bro_id/activate", null, Map.class))
.thenReturn(new HashMap() {{ put("status", "error message"); }});
TopologyResponse expected = new TopologyResponse();
@@ -163,8 +188,8 @@ public class StormStatusServiceImplTest {
@Test
public void activateTopologyShouldReturnTopologyNotFoundTopologyResponse() {
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(new TopologySummary());
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(new TopologySummary());
TopologyResponse expected = new TopologyResponse();
expected.setErrorMessage(TopologyStatusCode.TOPOLOGY_NOT_FOUND.toString());
@@ -179,9 +204,9 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
- when(restTemplate.postForObject("http://storm_ui" + TOPOLOGY_URL + "/bro_id/deactivate", null, Map.class))
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(restTemplate.postForObject(HTTP_STORM_UI + TOPOLOGY_URL + "/bro_id/deactivate", null, Map.class))
.thenReturn(new HashMap() {{ put("status", "success"); }});
TopologyResponse expected = new TopologyResponse();
@@ -197,9 +222,9 @@ public class StormStatusServiceImplTest {
final TopologySummary topologySummary = new TopologySummary();
topologySummary.setTopologies(new TopologyStatus[]{topologyStatus});
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
- when(restTemplate.postForObject("http://storm_ui" + TOPOLOGY_URL + "/bro_id/deactivate", null, Map.class))
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(topologySummary);
+ when(restTemplate.postForObject(HTTP_STORM_UI + TOPOLOGY_URL + "/bro_id/deactivate", null, Map.class))
.thenReturn(new HashMap() {{ put("status", "error message"); }});
TopologyResponse expected = new TopologyResponse();
@@ -209,8 +234,8 @@ public class StormStatusServiceImplTest {
@Test
public void deactivateTopologyShouldReturnTopologyNotFoundTopologyResponse() {
- when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn("storm_ui");
- when(restTemplate.getForObject("http://storm_ui" + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(new TopologySummary());
+ when(environment.getProperty(STORM_UI_SPRING_PROPERTY)).thenReturn(HTTP_STORM_UI);
+ when(restTemplate.getForObject(HTTP_STORM_UI + TOPOLOGY_SUMMARY_URL, TopologySummary.class)).thenReturn(new TopologySummary());
TopologyResponse expected = new TopologyResponse();
expected.setErrorMessage(TopologyStatusCode.TOPOLOGY_NOT_FOUND.toString());
[09/18] metron git commit: METRON-962 Configuration Based Unit Tests
and Add integration tests (justinleet via leet) closes apache/metron#612
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/lancope/BasicLancopeParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/lancope/BasicLancopeParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/lancope/BasicLancopeParserTest.java
index bbb4e88..50fdcd0 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/lancope/BasicLancopeParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/lancope/BasicLancopeParserTest.java
@@ -17,145 +17,41 @@
*/
package org.apache.metron.parsers.lancope;
+import com.github.fge.jsonschema.core.exceptions.ProcessingException;
import java.io.IOException;
import java.net.URL;
import java.util.Map;
-
+import org.apache.metron.parsers.AbstractParserConfigTest;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractSchemaTest;
import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
- /**
- * <ul>
- * <li>Title: Junit for LancopeParserTest</li>
- * <li>Description: </li>
- * <li>Created: Aug 25, 2014</li>
- * </ul>
- * @version $Revision: 1.1 $
- */
-public class BasicLancopeParserTest extends AbstractSchemaTest {
-
- /**
- * The inputStrings.
- */
- private static String[] inputStrings;
+public class BasicLancopeParserTest extends AbstractParserConfigTest {
+ @Before
+ public void setUp() throws Exception {
+ inputStrings = super.readTestDataFromFile("src/test/resources/logData/LancopeParserTest.txt");
+ parser = new BasicLancopeParser();
- /**
- * The parser.
- */
- private static BasicLancopeParser parser=null;
+ URL schema_url = getClass().getClassLoader().getResource(
+ "TestSchemas/LancopeSchema.json");
+ super.setSchemaJsonString(super.readSchemaFromFile(schema_url));
+ }
- /**
- * Constructs a new <code>BasicLancopeParserTest</code> instance.
- * @param name
- */
+ @Test
+ public void testParse() throws ParseException, IOException, ProcessingException {
+ for (String inputString : inputStrings) {
+ JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
+ Assert.assertNotNull(parsed);
- public BasicLancopeParserTest(String name) {
- super(name);
- }
+ JSONParser parser = new JSONParser();
- /**
-
- * @throws java.lang.Exception
- */
- protected static void setUpBeforeClass() throws Exception {
+ Map<?, ?> json = (Map<?, ?>) parser.parse(parsed.toJSONString());
+ Assert.assertTrue(validateJsonData(getSchemaJsonString(), json.toString()));
}
-
- /**
-
- * @throws java.lang.Exception
- */
- protected static void tearDownAfterClass() throws Exception {
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#setUp()
- */
- @Override
- protected void setUp() throws Exception {
- super.setUp("org.apache.metron.parsers.lancope.BasicLancopeParserTest");
- setInputStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- BasicLancopeParserTest.setParser(new BasicLancopeParser());
-
- URL schema_url = getClass().getClassLoader().getResource(
- "TestSchemas/LancopeSchema.json");
- super.setSchemaJsonString(super.readSchemaFromFile(schema_url));
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#tearDown()
- */
- @Override
- protected void tearDown() throws Exception {
- super.tearDown();
- }
-
- /**
- * Test method for {@link BasicLancopeParser#parse(byte[])}.
- * @throws Exception
- * @throws IOException
- */
- public void testParse() throws IOException, Exception {
-
- for (String inputString : getInputStrings()) {
- JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
- assertNotNull(parsed);
-
- System.out.println(parsed);
- JSONParser parser = new JSONParser();
-
- Map<?, ?> json=null;
- try {
- json = (Map<?, ?>) parser.parse(parsed.toJSONString());
- Assert.assertEquals(true, validateJsonData(super.getSchemaJsonString(), json.toString()));
- } catch (ParseException e) {
- e.printStackTrace();
- }
- }
- }
-
- /**
- * Returns the parser.
- * @return the parser.
- */
-
- public static BasicLancopeParser getParser() {
- return parser;
- }
-
- /**
- * Sets the parser.
- * @param parser the parser.
- */
-
- public static void setParser(BasicLancopeParser parser) {
-
- BasicLancopeParserTest.parser = parser;
- }
-
- /**
- * Returns the inputStrings.
- * @return the inputStrings.
- */
-
- public static String[] getInputStrings() {
- return inputStrings;
- }
-
- /**
- * Sets the inputStrings.
- * @param inputStrings the inputStrings.
- */
-
- public static void setInputStrings(String[] inputStrings) {
-
- BasicLancopeParserTest.inputStrings = inputStrings;
- }
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
index 6edd546..cf93c92 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
@@ -17,141 +17,41 @@
*/
package org.apache.metron.parsers.paloalto;
-import java.util.Iterator;
import java.util.Map;
-
-import org.apache.metron.parsers.sourcefire.BasicSourcefireParser;
+import java.util.Map.Entry;
+import org.apache.metron.parsers.AbstractParserConfigTest;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractConfigTest;
import org.junit.Assert;
-
-public class BasicPaloAltoFirewallParserTest extends AbstractConfigTest {
- /**
- * The inputStrings.
- */
- private static String[] inputStrings;
-
- /**
- * Constructs a new <code>BasicPaloAltoFirewallParserTest</code> instance.
- * @throws Exception
- */
- public BasicPaloAltoFirewallParserTest() throws Exception {
- super();
+import org.junit.Before;
+import org.junit.Test;
+
+public class BasicPaloAltoFirewallParserTest extends AbstractParserConfigTest {
+
+ @Before
+ public void setUp() throws Exception {
+ inputStrings = readTestDataFromFile(
+ "src/test/resources/logData/PaloAltoFirewallParserTest.txt");
+ parser = new BasicPaloAltoFirewallParser();
+ }
+
+ @SuppressWarnings({"rawtypes"})
+ @Test
+ public void testParse() throws ParseException {
+ for (String inputString : inputStrings) {
+ JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
+ Assert.assertNotNull(parsed);
+
+ JSONParser parser = new JSONParser();
+ Map json = (Map) parser.parse(parsed.toJSONString());
+
+ for (Object o : json.entrySet()) {
+ Entry entry = (Entry) o;
+ String key = (String) entry.getKey();
+ String value = json.get(key).toString();
+ Assert.assertNotNull(value);
+ }
}
-
- /**
- * Sets the inputStrings.
- * @param inputStrings the inputStrings.
- */
-
- public static void setInputStrings(String[] inputStrings) {
-
- BasicPaloAltoFirewallParserTest.inputStrings = inputStrings;
- }
-
- /**
- * The paParser.
- */
- private BasicPaloAltoFirewallParser paParser=null;
-
- /**
- * @throws java.lang.Exception
- */
- public static void setUpBeforeClass() throws Exception {
- }
-
- /**
- * @throws java.lang.Exception
- */
- public static void tearDownAfterClass() throws Exception {
- setPAStrings(null);
- }
-
- /**
- * @throws java.lang.Exception
- */
- @Override
- public void setUp() throws Exception {
- super.setUp("org.apache.metron.parsers.paloalto.BasicPaloAltoFirewallParserTest");
- setPAStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- paParser = new BasicPaloAltoFirewallParser();
- }
-
- /**
- *
- *
- * @throws java.lang.Exception
- */
- @Override
- public void tearDown() throws Exception {
- paParser = null;
- }
-
- /**
- * Test method for
- * {@link BasicSourcefireParser#parse(byte[])}.
- */
- @SuppressWarnings({ "rawtypes" })
- public void testParse() {
- for (String inputString : getInputStrings()) {
- JSONObject parsed = paParser.parse(inputString.getBytes()).get(0);
- Assert.assertNotNull(parsed);
-
- System.out.println(parsed);
- JSONParser parser = new JSONParser();
-
- Map json=null;
- try {
- json = (Map) parser.parse(parsed.toJSONString());
- } catch (ParseException e) {
- e.printStackTrace();
- }
- Iterator iter = json.entrySet().iterator();
-
-
- while (iter.hasNext()) {
- Map.Entry entry = (Map.Entry) iter.next();
- String key = (String) entry.getKey();
- String value = (String) json.get(key).toString();
- Assert.assertNotNull(value);
- }
- }
- }
-
- /**
- * Returns Input String
- */
- public static String[] getInputStrings() {
- return inputStrings;
- }
-
-
- /**
- * Sets Input String
- */
- public static void setPAStrings(String[] strings) {
- BasicPaloAltoFirewallParserTest.inputStrings = strings;
- }
-
- /**
- * Returns the paParser.
- * @return the paParser.
- */
- public BasicPaloAltoFirewallParser getPaParser() {
- return paParser;
- }
-
- /**
- * Sets the paParser.
- * @param paParser the paParser.
- */
-
- public void setPaParser(BasicPaloAltoFirewallParser paParser) {
-
- this.paParser = paParser;
- }
-
- }
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/sourcefire/BasicSourcefireParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/sourcefire/BasicSourcefireParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/sourcefire/BasicSourcefireParserTest.java
index f5056ba..dedd9db 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/sourcefire/BasicSourcefireParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/sourcefire/BasicSourcefireParserTest.java
@@ -17,142 +17,42 @@
*/
package org.apache.metron.parsers.sourcefire;
-
-
-import java.util.Iterator;
import java.util.Map;
-
+import java.util.Map.Entry;
+import org.apache.metron.parsers.AbstractParserConfigTest;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractConfigTest;
import org.junit.Assert;
-
-/**
- * <ul>
- * <li>Title: Test For SourceFireParser</li>
- * <li>Description: </li>
- * <li>Created: July 8, 2014</li>
- * </ul>
- * @version $Revision: 1.0 $
- */
-public class BasicSourcefireParserTest extends AbstractConfigTest
-{
- /**
- * The sourceFireStrings.
- */
- private static String[] sourceFireStrings;
-
- /**
- * The sourceFireParser.
- */
- private BasicSourcefireParser sourceFireParser=null;
-
-
- /**
- * Constructs a new <code>BasicSourcefireParserTest</code> instance.
- * @throws Exception
- */
-
- public BasicSourcefireParserTest() throws Exception {
- super();
+import org.junit.Before;
+import org.junit.Test;
+
+public class BasicSourcefireParserTest extends AbstractParserConfigTest {
+
+ @Before
+ public void setUp() throws Exception {
+ inputStrings = super
+ .readTestDataFromFile("src/test/resources/logData/SourcefireParserTest.txt");
+ parser = new BasicSourcefireParser();
+ }
+
+ @SuppressWarnings({"rawtypes", "unused"})
+ @Test
+ public void testParse() throws ParseException {
+ for (String inputString : inputStrings) {
+ byte[] srcBytes = inputString.getBytes();
+ JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
+ Assert.assertNotNull(parsed);
+
+ JSONParser parser = new JSONParser();
+ Map json = (Map) parser.parse(parsed.toJSONString());
+
+ for (Object o : json.entrySet()) {
+ Entry entry = (Entry) o;
+ String key = (String) entry.getKey();
+ String value = json.get("original_string").toString();
+ Assert.assertNotNull(value);
+ }
}
-
- /**
- * @throws java.lang.Exception
- */
- public static void setUpBeforeClass() throws Exception {
- }
-
- /**
- * @throws java.lang.Exception
- */
- public static void tearDownAfterClass() throws Exception {
- setSourceFireStrings(null);
- }
-
- /**
- * @throws java.lang.Exception
- */
- @Override
- public void setUp() throws Exception {
- super.setUp("org.apache.metron.parsing.test.BasicSoureceFireParserTest");
- setSourceFireStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- sourceFireParser = new BasicSourcefireParser();
- }
-
- /**
- *
- *
- * @throws java.lang.Exception
- */
- @Override
- public void tearDown() throws Exception {
- sourceFireParser = null;
- }
-
- /**
- * Test method for {@link BasicSourcefireParser#parse(byte[])}.
- */
- @SuppressWarnings({ "rawtypes", "unused" })
- public void testParse() {
- for (String sourceFireString : getSourceFireStrings()) {
- byte[] srcBytes = sourceFireString.getBytes();
- JSONObject parsed = sourceFireParser.parse(sourceFireString.getBytes()).get(0);
- Assert.assertNotNull(parsed);
-
- System.out.println(parsed);
- JSONParser parser = new JSONParser();
-
- Map json=null;
- try {
- json = (Map) parser.parse(parsed.toJSONString());
- } catch (ParseException e) {
- e.printStackTrace();
- }
- Iterator iter = json.entrySet().iterator();
-
-
- while (iter.hasNext()) {
- Map.Entry entry = (Map.Entry) iter.next();
- String key = (String) entry.getKey();
- String value = (String) json.get("original_string").toString();
- Assert.assertNotNull(value);
- }
- }
- }
-
- /**
- * Returns SourceFire Input String
- */
- public static String[] getSourceFireStrings() {
- return sourceFireStrings;
- }
-
-
- /**
- * Sets SourceFire Input String
- */
- public static void setSourceFireStrings(String[] strings) {
- BasicSourcefireParserTest.sourceFireStrings = strings;
- }
- /**
- * Returns the sourceFireParser.
- * @return the sourceFireParser.
- */
-
- public BasicSourcefireParser getSourceFireParser() {
- return sourceFireParser;
- }
-
- /**
- * Sets the sourceFireParser.
- * @param sourceFireParser the sourceFireParser.
- */
-
- public void setSourceFireParser(BasicSourcefireParser sourceFireParser) {
-
- this.sourceFireParser = sourceFireParser;
- }
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/resources/config/GrokAsaParserTest.config
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/config/GrokAsaParserTest.config b/metron-platform/metron-parsers/src/test/resources/config/GrokAsaParserTest.config
deleted file mode 100644
index 9dbc3b6..0000000
--- a/metron-platform/metron-parsers/src/test/resources/config/GrokAsaParserTest.config
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#GrokParserTestConfig
-logFile=src/test/resources/GrokParserTest.log
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/resources/logData/FireEyeParserTest.txt
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/logData/FireEyeParserTest.txt b/metron-platform/metron-parsers/src/test/resources/logData/FireEyeParserTest.txt
new file mode 100644
index 0000000..f3be97a
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/logData/FireEyeParserTest.txt
@@ -0,0 +1,8 @@
+<164>Mar 19 05:24:39 10.220.15.15 fenotify-851983.alert: CEF:0|FireEye|CMS|7.2.1.244420|DM|domain-match|1|rt=Feb 09 2015 12:28:26 UTC dvc=10.201.78.57 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 shost=dev001srv02.example.com proto=udp cs5Label=cncHost cs5=mfdclk001.org dvchost=DEVFEYE1 spt=54527 dvc=10.100.25.16 smac=00:00:0c:07:ac:00 cn1Label=vlan cn1=0 externalId=851983 cs4Label=link cs4=https://DEVCMS01.example.com/event_stream/events_for_bot?ev_id\\=851983 dmac=00:1d:a2:af:32:a1 cs1Label=sname cs1=Trojan.Generic.DNS
+<164>Mar 19 05:24:39 10.220.15.15 fenotify-851987.alert: CEF:0|FireEye|CMS|7.2.1.244420|DM|domain-match|1|rt=Feb 09 2015 12:33:41 UTC dvc=10.201.78.113 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 shost=dev001srv02.example.com proto=udp cs5Label=cncHost cs5=mfdclk001.org dvchost=DEVFEYE1 spt=51218 dvc=10.100.25.16 smac=00:00:0c:07:ac:00 cn1Label=vlan cn1=0 externalId=851987 cs4Label=link cs4=https://DEVCMS01.example.com/event_stream/events_for_bot?ev_id\\=851987 dmac=00:1d:a2:af:32:a1 cs1Label=sname cs1=Trojan.Generic.DNS
+<164>Mar 19 05:24:39 10.220.15.15 fenotify-3483808.2.alert: 1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/microads/update/InjectScript.js HTTP/1.1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/microads/update/InjectScript.js HTTP/1.1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/microads/update/InjectScript.js HTTP/1.1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/microads/update/InjectScript.js HTTP/1.1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/microads/update/InjectScript.js HTTP/1.1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/microads/update/InjectScript.js HTTP/1.1::~~User-Agent: WinHttpClient::~~Host: www.microads.me::~~Connection: Keep-Alive::~~::~~GET /files/mic
roads/update/InjectScript.js HTTP
+<164>Mar 19 05:24:39 10.220.15.15 fenotify-793972.2.alert: Control: no-cache::~~::~~ dmac=00:1d:a2:af:32:a1 cs1Label=sname cs1=Exploit.Kit.Magnitude
+<161>Apr 1 05:24:39 10.220.15.15 fenotify-864461.alert: CEF:0|FireEye|CMS|7.5.1.318703|DM|domain-match|1|rt=Mar 19 2015 12:23:47 UTC src=10.191.193.20 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 shost=abc123.example.com proto=udp spt=60903 cs5Label=cncHost cs5=mfdclk001.org dvchost=ABC123 dvc=10.190.1.16 smac=00:00:0c:07:ac:c8 cn1Label=vlan cn1=0 externalId=864461 cs4Label=link cs4=https:\/\/ABC123.example.com\/event_stream\/events_for_bot?ev_id\\=864461 act=notified dmac=88:43:e1:95:13:29 cs1Label=sname cs1=Trojan.Generic.DNS
+fireeye[-]: <161>Mar 19 05:24:39 10.220.15.15 fenotify-864461.alert: CEF:0|FireEye|CMS|7.5.1.318703|DM|domain-match|1|rt=Mar 19 2015 12:23:47 UTC src=10.191.193.20 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 shost=abc123.example.com proto=udp spt=60903 cs5Label=cncHost cs5=mfdclk001.org dvchost=ABC123 dvc=10.190.1.16 smac=00:00:0c:07:ac:c8 cn1Label=vlan cn1=0 externalId=864461 cs4Label=link cs4=https:\/\/ABC123.example.com\/event_stream\/events_for_bot?ev_id\\=864461 act=notified dmac=88:43:e1:95:13:29 cs1Label=sname cs1=Trojan.Generic.DNS
+fireeye[-]: <161>Apr 1 02:49:49 10.220.15.15 fenotify-900702.alert: CEF:0|FireEye|CMS|7.5.1.318703|DM|domain-match|1|rt=Apr 01 2015 09:49:14 UTC src=10.1.97.20 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 shost=abcd0060xzy03.example.com proto=udp spt=63100 cs5Label=cncHost cs5=mfdclk001.org dvchost=DEV1FEYE1 dvc=10.220.15.16 smac=00:00:0c:07:ac:00 cn1Label=vlan cn1=0 externalId=900702 cs4Label=link cs4=https://ABCD0040CMS01.example.com/event_stream/events_for_bot?ev_id\=900702 act=notified dmac=00:1d:a2:af:32:a1 cs1Label=sname cs1=Trojan.Generic.DNS
+<161>Apr 11 05:24:39 10.220.15.15 fenotify-864461.alert: CEF:0|FireEye|CMS|7.5.1.318703|DM|domain-match|1|rt=Mar 19 2015 12:23:47 UTC src=10.191.193.20 cn3Label=cncPort cn3=53 cn2Label=sid cn2=80494706 shost=abc123.example.com proto=udp spt=60903 cs5Label=cncHost cs5=mfdclk001.org dvchost=ABC123 dvc=10.190.1.16 smac=00:00:0c:07:ac:c8 cn1Label=vlan cn1=0 externalId=864461 cs4Label=link cs4=https:\/\/ABC123.example.com\/event_stream\/events_for_bot?ev_id\\=864461 act=notified dmac=88:43:e1:95:13:29 cs1Label=sname cs1=Trojan.Generic.DNS
\ No newline at end of file
[06/18] metron git commit: METRON-979 markdown errors in site-book,
Part 3 (mattf-horton) closes apache/metron#611
Posted by ma...@apache.org.
METRON-979 markdown errors in site-book, Part 3 (mattf-horton) closes apache/metron#611
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/605a3cc6
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/605a3cc6
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/605a3cc6
Branch: refs/heads/Metron_0.4.0
Commit: 605a3cc6d29d0ac1377270b1455819645f4a95bc
Parents: d0e1ba5
Author: mattf-horton <mf...@hortonworks.com>
Authored: Tue Jun 6 21:18:55 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Tue Jun 6 21:18:55 2017 -0700
----------------------------------------------------------------------
metron-deployment/Kerberos-manual-setup.md | 4 ----
1 file changed, 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/605a3cc6/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index cdfbdfb..4efebc6 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -381,11 +381,7 @@ Push Data
```
cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
-<<<<<<< HEAD
--broker-list ${BROKERLIST} \
-=======
- --broker-list ${BROKERLIST}
->>>>>>> d550b9eff34d931aeae7151713c0e07f28719d4c
--security-protocol SASL_PLAINTEXT \
--topic bro
```
[14/18] metron git commit: METRON-980: Short circuit operations for
Stellar closes apache/metron#606
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarParser.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarParser.java
index 51c99e4..adf7e99 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarParser.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/generated/StellarParser.java
@@ -45,21 +45,21 @@ public class StellarParser extends Parser {
DOUBLE_LITERAL=38, FLOAT_LITERAL=39, LONG_LITERAL=40, IDENTIFIER=41, STRING_LITERAL=42,
COMMENT=43, WS=44;
public static final int
- RULE_transformation = 0, RULE_transformation_expr = 1, RULE_conditional_expr = 2,
- RULE_logical_expr = 3, RULE_b_expr = 4, RULE_in_expr = 5, RULE_comparison_expr = 6,
- RULE_transformation_entity = 7, RULE_comp_operator = 8, RULE_func_args = 9,
- RULE_op_list = 10, RULE_list_entity = 11, RULE_kv_list = 12, RULE_map_entity = 13,
- RULE_arithmetic_expr = 14, RULE_arithmetic_expr_mul = 15, RULE_functions = 16,
- RULE_arithmetic_operands = 17, RULE_identifier_operand = 18, RULE_lambda_without_args = 19,
- RULE_lambda_with_args = 20, RULE_lambda_variables = 21, RULE_single_lambda_variable = 22,
- RULE_lambda_variable = 23;
+ RULE_transformation = 0, RULE_transformation_expr = 1, RULE_if_expr = 2,
+ RULE_then_expr = 3, RULE_else_expr = 4, RULE_conditional_expr = 5, RULE_logical_expr = 6,
+ RULE_b_expr = 7, RULE_in_expr = 8, RULE_comparison_expr = 9, RULE_transformation_entity = 10,
+ RULE_comp_operator = 11, RULE_func_args = 12, RULE_op_list = 13, RULE_list_entity = 14,
+ RULE_kv_list = 15, RULE_map_entity = 16, RULE_arithmetic_expr = 17, RULE_arithmetic_expr_mul = 18,
+ RULE_functions = 19, RULE_arithmetic_operands = 20, RULE_identifier_operand = 21,
+ RULE_lambda_without_args = 22, RULE_lambda_with_args = 23, RULE_lambda_variables = 24,
+ RULE_single_lambda_variable = 25, RULE_lambda_variable = 26;
public static final String[] ruleNames = {
- "transformation", "transformation_expr", "conditional_expr", "logical_expr",
- "b_expr", "in_expr", "comparison_expr", "transformation_entity", "comp_operator",
- "func_args", "op_list", "list_entity", "kv_list", "map_entity", "arithmetic_expr",
- "arithmetic_expr_mul", "functions", "arithmetic_operands", "identifier_operand",
- "lambda_without_args", "lambda_with_args", "lambda_variables", "single_lambda_variable",
- "lambda_variable"
+ "transformation", "transformation_expr", "if_expr", "then_expr", "else_expr",
+ "conditional_expr", "logical_expr", "b_expr", "in_expr", "comparison_expr",
+ "transformation_entity", "comp_operator", "func_args", "op_list", "list_entity",
+ "kv_list", "map_entity", "arithmetic_expr", "arithmetic_expr_mul", "functions",
+ "arithmetic_operands", "identifier_operand", "lambda_without_args", "lambda_with_args",
+ "lambda_variables", "single_lambda_variable", "lambda_variable"
};
private static final String[] _LITERAL_NAMES = {
@@ -150,9 +150,9 @@ public class StellarParser extends Parser {
try {
enterOuterAlt(_localctx, 1);
{
- setState(48);
+ setState(54);
transformation_expr();
- setState(49);
+ setState(55);
match(EOF);
}
}
@@ -283,13 +283,13 @@ public class StellarParser extends Parser {
Transformation_exprContext _localctx = new Transformation_exprContext(_ctx, getState());
enterRule(_localctx, 2, RULE_transformation_expr);
try {
- setState(61);
+ setState(67);
switch ( getInterpreter().adaptivePredict(_input,0,_ctx) ) {
case 1:
_localctx = new ConditionalExprContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(51);
+ setState(57);
conditional_expr();
}
break;
@@ -297,11 +297,11 @@ public class StellarParser extends Parser {
_localctx = new TransformationExprContext(_localctx);
enterOuterAlt(_localctx, 2);
{
- setState(52);
+ setState(58);
match(LPAREN);
- setState(53);
+ setState(59);
transformation_expr();
- setState(54);
+ setState(60);
match(RPAREN);
}
break;
@@ -309,7 +309,7 @@ public class StellarParser extends Parser {
_localctx = new ArithExpressionContext(_localctx);
enterOuterAlt(_localctx, 3);
{
- setState(56);
+ setState(62);
arithmetic_expr(0);
}
break;
@@ -317,7 +317,7 @@ public class StellarParser extends Parser {
_localctx = new TransformationEntityContext(_localctx);
enterOuterAlt(_localctx, 4);
{
- setState(57);
+ setState(63);
transformation_entity();
}
break;
@@ -325,7 +325,7 @@ public class StellarParser extends Parser {
_localctx = new ComparisonExpressionContext(_localctx);
enterOuterAlt(_localctx, 5);
{
- setState(58);
+ setState(64);
comparison_expr(0);
}
break;
@@ -333,7 +333,7 @@ public class StellarParser extends Parser {
_localctx = new LogicalExpressionContext(_localctx);
enterOuterAlt(_localctx, 6);
{
- setState(59);
+ setState(65);
logical_expr();
}
break;
@@ -341,7 +341,7 @@ public class StellarParser extends Parser {
_localctx = new InExpressionContext(_localctx);
enterOuterAlt(_localctx, 7);
{
- setState(60);
+ setState(66);
in_expr();
}
break;
@@ -358,6 +358,123 @@ public class StellarParser extends Parser {
return _localctx;
}
+ public static class If_exprContext extends ParserRuleContext {
+ public Logical_exprContext logical_expr() {
+ return getRuleContext(Logical_exprContext.class,0);
+ }
+ public If_exprContext(ParserRuleContext parent, int invokingState) {
+ super(parent, invokingState);
+ }
+ @Override public int getRuleIndex() { return RULE_if_expr; }
+ @Override
+ public void enterRule(ParseTreeListener listener) {
+ if ( listener instanceof StellarListener ) ((StellarListener)listener).enterIf_expr(this);
+ }
+ @Override
+ public void exitRule(ParseTreeListener listener) {
+ if ( listener instanceof StellarListener ) ((StellarListener)listener).exitIf_expr(this);
+ }
+ }
+
+ public final If_exprContext if_expr() throws RecognitionException {
+ If_exprContext _localctx = new If_exprContext(_ctx, getState());
+ enterRule(_localctx, 4, RULE_if_expr);
+ try {
+ enterOuterAlt(_localctx, 1);
+ {
+ setState(69);
+ logical_expr();
+ }
+ }
+ catch (RecognitionException re) {
+ _localctx.exception = re;
+ _errHandler.reportError(this, re);
+ _errHandler.recover(this, re);
+ }
+ finally {
+ exitRule();
+ }
+ return _localctx;
+ }
+
+ public static class Then_exprContext extends ParserRuleContext {
+ public Transformation_exprContext transformation_expr() {
+ return getRuleContext(Transformation_exprContext.class,0);
+ }
+ public Then_exprContext(ParserRuleContext parent, int invokingState) {
+ super(parent, invokingState);
+ }
+ @Override public int getRuleIndex() { return RULE_then_expr; }
+ @Override
+ public void enterRule(ParseTreeListener listener) {
+ if ( listener instanceof StellarListener ) ((StellarListener)listener).enterThen_expr(this);
+ }
+ @Override
+ public void exitRule(ParseTreeListener listener) {
+ if ( listener instanceof StellarListener ) ((StellarListener)listener).exitThen_expr(this);
+ }
+ }
+
+ public final Then_exprContext then_expr() throws RecognitionException {
+ Then_exprContext _localctx = new Then_exprContext(_ctx, getState());
+ enterRule(_localctx, 6, RULE_then_expr);
+ try {
+ enterOuterAlt(_localctx, 1);
+ {
+ setState(71);
+ transformation_expr();
+ }
+ }
+ catch (RecognitionException re) {
+ _localctx.exception = re;
+ _errHandler.reportError(this, re);
+ _errHandler.recover(this, re);
+ }
+ finally {
+ exitRule();
+ }
+ return _localctx;
+ }
+
+ public static class Else_exprContext extends ParserRuleContext {
+ public Transformation_exprContext transformation_expr() {
+ return getRuleContext(Transformation_exprContext.class,0);
+ }
+ public Else_exprContext(ParserRuleContext parent, int invokingState) {
+ super(parent, invokingState);
+ }
+ @Override public int getRuleIndex() { return RULE_else_expr; }
+ @Override
+ public void enterRule(ParseTreeListener listener) {
+ if ( listener instanceof StellarListener ) ((StellarListener)listener).enterElse_expr(this);
+ }
+ @Override
+ public void exitRule(ParseTreeListener listener) {
+ if ( listener instanceof StellarListener ) ((StellarListener)listener).exitElse_expr(this);
+ }
+ }
+
+ public final Else_exprContext else_expr() throws RecognitionException {
+ Else_exprContext _localctx = new Else_exprContext(_ctx, getState());
+ enterRule(_localctx, 8, RULE_else_expr);
+ try {
+ enterOuterAlt(_localctx, 1);
+ {
+ setState(73);
+ transformation_expr();
+ }
+ }
+ catch (RecognitionException re) {
+ _localctx.exception = re;
+ _errHandler.reportError(this, re);
+ _errHandler.recover(this, re);
+ }
+ finally {
+ exitRule();
+ }
+ return _localctx;
+ }
+
public static class Conditional_exprContext extends ParserRuleContext {
public Conditional_exprContext(ParserRuleContext parent, int invokingState) {
super(parent, invokingState);
@@ -370,17 +487,17 @@ public class StellarParser extends Parser {
}
}
public static class TernaryFuncWithoutIfContext extends Conditional_exprContext {
- public Logical_exprContext logical_expr() {
- return getRuleContext(Logical_exprContext.class,0);
+ public If_exprContext if_expr() {
+ return getRuleContext(If_exprContext.class,0);
}
public TerminalNode QUESTION() { return getToken(StellarParser.QUESTION, 0); }
- public List<Transformation_exprContext> transformation_expr() {
- return getRuleContexts(Transformation_exprContext.class);
- }
- public Transformation_exprContext transformation_expr(int i) {
- return getRuleContext(Transformation_exprContext.class,i);
+ public Then_exprContext then_expr() {
+ return getRuleContext(Then_exprContext.class,0);
}
public TerminalNode COLON() { return getToken(StellarParser.COLON, 0); }
+ public Else_exprContext else_expr() {
+ return getRuleContext(Else_exprContext.class,0);
+ }
public TernaryFuncWithoutIfContext(Conditional_exprContext ctx) { copyFrom(ctx); }
@Override
public void enterRule(ParseTreeListener listener) {
@@ -393,17 +510,17 @@ public class StellarParser extends Parser {
}
public static class TernaryFuncWithIfContext extends Conditional_exprContext {
public TerminalNode IF() { return getToken(StellarParser.IF, 0); }
- public Logical_exprContext logical_expr() {
- return getRuleContext(Logical_exprContext.class,0);
+ public If_exprContext if_expr() {
+ return getRuleContext(If_exprContext.class,0);
}
public TerminalNode THEN() { return getToken(StellarParser.THEN, 0); }
- public List<Transformation_exprContext> transformation_expr() {
- return getRuleContexts(Transformation_exprContext.class);
- }
- public Transformation_exprContext transformation_expr(int i) {
- return getRuleContext(Transformation_exprContext.class,i);
+ public Then_exprContext then_expr() {
+ return getRuleContext(Then_exprContext.class,0);
}
public TerminalNode ELSE() { return getToken(StellarParser.ELSE, 0); }
+ public Else_exprContext else_expr() {
+ return getRuleContext(Else_exprContext.class,0);
+ }
public TernaryFuncWithIfContext(Conditional_exprContext ctx) { copyFrom(ctx); }
@Override
public void enterRule(ParseTreeListener listener) {
@@ -417,9 +534,9 @@ public class StellarParser extends Parser {
public final Conditional_exprContext conditional_expr() throws RecognitionException {
Conditional_exprContext _localctx = new Conditional_exprContext(_ctx, getState());
- enterRule(_localctx, 4, RULE_conditional_expr);
+ enterRule(_localctx, 10, RULE_conditional_expr);
try {
- setState(76);
+ setState(88);
switch (_input.LA(1)) {
case NOT:
case TRUE:
@@ -438,34 +555,34 @@ public class StellarParser extends Parser {
_localctx = new TernaryFuncWithoutIfContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(63);
- logical_expr();
- setState(64);
+ setState(75);
+ if_expr();
+ setState(76);
match(QUESTION);
- setState(65);
- transformation_expr();
- setState(66);
+ setState(77);
+ then_expr();
+ setState(78);
match(COLON);
- setState(67);
- transformation_expr();
+ setState(79);
+ else_expr();
}
break;
case IF:
_localctx = new TernaryFuncWithIfContext(_localctx);
enterOuterAlt(_localctx, 2);
{
- setState(69);
+ setState(81);
match(IF);
- setState(70);
- logical_expr();
- setState(71);
+ setState(82);
+ if_expr();
+ setState(83);
match(THEN);
- setState(72);
- transformation_expr();
- setState(73);
+ setState(84);
+ then_expr();
+ setState(85);
match(ELSE);
- setState(74);
- transformation_expr();
+ setState(86);
+ else_expr();
}
break;
default:
@@ -547,19 +664,19 @@ public class StellarParser extends Parser {
public final Logical_exprContext logical_expr() throws RecognitionException {
Logical_exprContext _localctx = new Logical_exprContext(_ctx, getState());
- enterRule(_localctx, 6, RULE_logical_expr);
+ enterRule(_localctx, 12, RULE_logical_expr);
try {
- setState(87);
+ setState(99);
switch ( getInterpreter().adaptivePredict(_input,2,_ctx) ) {
case 1:
_localctx = new LogicalExpressionAndContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(78);
+ setState(90);
b_expr();
- setState(79);
+ setState(91);
match(AND);
- setState(80);
+ setState(92);
logical_expr();
}
break;
@@ -567,11 +684,11 @@ public class StellarParser extends Parser {
_localctx = new LogicalExpressionOrContext(_localctx);
enterOuterAlt(_localctx, 2);
{
- setState(82);
+ setState(94);
b_expr();
- setState(83);
+ setState(95);
match(OR);
- setState(84);
+ setState(96);
logical_expr();
}
break;
@@ -579,7 +696,7 @@ public class StellarParser extends Parser {
_localctx = new BoleanExpressionContext(_localctx);
enterOuterAlt(_localctx, 3);
{
- setState(86);
+ setState(98);
b_expr();
}
break;
@@ -619,21 +736,21 @@ public class StellarParser extends Parser {
public final B_exprContext b_expr() throws RecognitionException {
B_exprContext _localctx = new B_exprContext(_ctx, getState());
- enterRule(_localctx, 8, RULE_b_expr);
+ enterRule(_localctx, 14, RULE_b_expr);
try {
- setState(91);
+ setState(103);
switch ( getInterpreter().adaptivePredict(_input,3,_ctx) ) {
case 1:
enterOuterAlt(_localctx, 1);
{
- setState(89);
+ setState(101);
comparison_expr(0);
}
break;
case 2:
enterOuterAlt(_localctx, 2);
{
- setState(90);
+ setState(102);
in_expr();
}
break;
@@ -700,19 +817,19 @@ public class StellarParser extends Parser {
public final In_exprContext in_expr() throws RecognitionException {
In_exprContext _localctx = new In_exprContext(_ctx, getState());
- enterRule(_localctx, 10, RULE_in_expr);
+ enterRule(_localctx, 16, RULE_in_expr);
try {
- setState(101);
+ setState(113);
switch ( getInterpreter().adaptivePredict(_input,4,_ctx) ) {
case 1:
_localctx = new InExpressionStatementContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(93);
+ setState(105);
identifier_operand();
- setState(94);
+ setState(106);
match(IN);
- setState(95);
+ setState(107);
b_expr();
}
break;
@@ -720,11 +837,11 @@ public class StellarParser extends Parser {
_localctx = new NInExpressionStatementContext(_localctx);
enterOuterAlt(_localctx, 2);
{
- setState(97);
+ setState(109);
identifier_operand();
- setState(98);
+ setState(110);
match(NIN);
- setState(99);
+ setState(111);
b_expr();
}
break;
@@ -829,13 +946,13 @@ public class StellarParser extends Parser {
int _parentState = getState();
Comparison_exprContext _localctx = new Comparison_exprContext(_ctx, _parentState);
Comparison_exprContext _prevctx = _localctx;
- int _startState = 12;
- enterRecursionRule(_localctx, 12, RULE_comparison_expr, _p);
+ int _startState = 18;
+ enterRecursionRule(_localctx, 18, RULE_comparison_expr, _p);
try {
int _alt;
enterOuterAlt(_localctx, 1);
{
- setState(114);
+ setState(126);
switch ( getInterpreter().adaptivePredict(_input,5,_ctx) ) {
case 1:
{
@@ -843,13 +960,13 @@ public class StellarParser extends Parser {
_ctx = _localctx;
_prevctx = _localctx;
- setState(104);
+ setState(116);
match(NOT);
- setState(105);
+ setState(117);
match(LPAREN);
- setState(106);
+ setState(118);
logical_expr();
- setState(107);
+ setState(119);
match(RPAREN);
}
break;
@@ -858,11 +975,11 @@ public class StellarParser extends Parser {
_localctx = new ComparisonExpressionParensContext(_localctx);
_ctx = _localctx;
_prevctx = _localctx;
- setState(109);
+ setState(121);
match(LPAREN);
- setState(110);
+ setState(122);
logical_expr();
- setState(111);
+ setState(123);
match(RPAREN);
}
break;
@@ -871,13 +988,13 @@ public class StellarParser extends Parser {
_localctx = new OperandContext(_localctx);
_ctx = _localctx;
_prevctx = _localctx;
- setState(113);
+ setState(125);
identifier_operand();
}
break;
}
_ctx.stop = _input.LT(-1);
- setState(122);
+ setState(134);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,6,_ctx);
while ( _alt!=2 && _alt!=org.antlr.v4.runtime.atn.ATN.INVALID_ALT_NUMBER ) {
@@ -888,16 +1005,16 @@ public class StellarParser extends Parser {
{
_localctx = new ComparisonExpressionWithOperatorContext(new Comparison_exprContext(_parentctx, _parentState));
pushNewRecursionContext(_localctx, _startState, RULE_comparison_expr);
- setState(116);
+ setState(128);
if (!(precpred(_ctx, 4))) throw new FailedPredicateException(this, "precpred(_ctx, 4)");
- setState(117);
+ setState(129);
comp_operator();
- setState(118);
+ setState(130);
comparison_expr(5);
}
}
}
- setState(124);
+ setState(136);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,6,_ctx);
}
@@ -934,11 +1051,11 @@ public class StellarParser extends Parser {
public final Transformation_entityContext transformation_entity() throws RecognitionException {
Transformation_entityContext _localctx = new Transformation_entityContext(_ctx, getState());
- enterRule(_localctx, 14, RULE_transformation_entity);
+ enterRule(_localctx, 20, RULE_transformation_entity);
try {
enterOuterAlt(_localctx, 1);
{
- setState(125);
+ setState(137);
identifier_operand();
}
}
@@ -984,13 +1101,13 @@ public class StellarParser extends Parser {
public final Comp_operatorContext comp_operator() throws RecognitionException {
Comp_operatorContext _localctx = new Comp_operatorContext(_ctx, getState());
- enterRule(_localctx, 16, RULE_comp_operator);
+ enterRule(_localctx, 22, RULE_comp_operator);
int _la;
try {
_localctx = new ComparisonOpContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(127);
+ setState(139);
_la = _input.LA(1);
if ( !((((_la) & ~0x3f) == 0 && ((1L << _la) & ((1L << EQ) | (1L << NEQ) | (1L << LT) | (1L << LTE) | (1L << GT) | (1L << GTE))) != 0)) ) {
_errHandler.recoverInline(this);
@@ -1032,27 +1149,27 @@ public class StellarParser extends Parser {
public final Func_argsContext func_args() throws RecognitionException {
Func_argsContext _localctx = new Func_argsContext(_ctx, getState());
- enterRule(_localctx, 18, RULE_func_args);
+ enterRule(_localctx, 24, RULE_func_args);
try {
- setState(135);
+ setState(147);
switch ( getInterpreter().adaptivePredict(_input,7,_ctx) ) {
case 1:
enterOuterAlt(_localctx, 1);
{
- setState(129);
+ setState(141);
match(LPAREN);
- setState(130);
+ setState(142);
op_list(0);
- setState(131);
+ setState(143);
match(RPAREN);
}
break;
case 2:
enterOuterAlt(_localctx, 2);
{
- setState(133);
+ setState(145);
match(LPAREN);
- setState(134);
+ setState(146);
match(RPAREN);
}
break;
@@ -1103,29 +1220,29 @@ public class StellarParser extends Parser {
int _parentState = getState();
Op_listContext _localctx = new Op_listContext(_ctx, _parentState);
Op_listContext _prevctx = _localctx;
- int _startState = 20;
- enterRecursionRule(_localctx, 20, RULE_op_list, _p);
+ int _startState = 26;
+ enterRecursionRule(_localctx, 26, RULE_op_list, _p);
try {
int _alt;
enterOuterAlt(_localctx, 1);
{
- setState(140);
+ setState(152);
switch ( getInterpreter().adaptivePredict(_input,8,_ctx) ) {
case 1:
{
- setState(138);
+ setState(150);
identifier_operand();
}
break;
case 2:
{
- setState(139);
+ setState(151);
conditional_expr();
}
break;
}
_ctx.stop = _input.LT(-1);
- setState(150);
+ setState(162);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,10,_ctx);
while ( _alt!=2 && _alt!=org.antlr.v4.runtime.atn.ATN.INVALID_ALT_NUMBER ) {
@@ -1133,17 +1250,17 @@ public class StellarParser extends Parser {
if ( _parseListeners!=null ) triggerExitRuleEvent();
_prevctx = _localctx;
{
- setState(148);
+ setState(160);
switch ( getInterpreter().adaptivePredict(_input,9,_ctx) ) {
case 1:
{
_localctx = new Op_listContext(_parentctx, _parentState);
pushNewRecursionContext(_localctx, _startState, RULE_op_list);
- setState(142);
+ setState(154);
if (!(precpred(_ctx, 3))) throw new FailedPredicateException(this, "precpred(_ctx, 3)");
- setState(143);
+ setState(155);
match(COMMA);
- setState(144);
+ setState(156);
identifier_operand();
}
break;
@@ -1151,18 +1268,18 @@ public class StellarParser extends Parser {
{
_localctx = new Op_listContext(_parentctx, _parentState);
pushNewRecursionContext(_localctx, _startState, RULE_op_list);
- setState(145);
+ setState(157);
if (!(precpred(_ctx, 1))) throw new FailedPredicateException(this, "precpred(_ctx, 1)");
- setState(146);
+ setState(158);
match(COMMA);
- setState(147);
+ setState(159);
conditional_expr();
}
break;
}
}
}
- setState(152);
+ setState(164);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,10,_ctx);
}
@@ -1201,27 +1318,27 @@ public class StellarParser extends Parser {
public final List_entityContext list_entity() throws RecognitionException {
List_entityContext _localctx = new List_entityContext(_ctx, getState());
- enterRule(_localctx, 22, RULE_list_entity);
+ enterRule(_localctx, 28, RULE_list_entity);
try {
- setState(159);
+ setState(171);
switch ( getInterpreter().adaptivePredict(_input,11,_ctx) ) {
case 1:
enterOuterAlt(_localctx, 1);
{
- setState(153);
+ setState(165);
match(LBRACKET);
- setState(154);
+ setState(166);
match(RBRACKET);
}
break;
case 2:
enterOuterAlt(_localctx, 2);
{
- setState(155);
+ setState(167);
match(LBRACKET);
- setState(156);
+ setState(168);
op_list(0);
- setState(157);
+ setState(169);
match(RBRACKET);
}
break;
@@ -1273,22 +1390,22 @@ public class StellarParser extends Parser {
int _parentState = getState();
Kv_listContext _localctx = new Kv_listContext(_ctx, _parentState);
Kv_listContext _prevctx = _localctx;
- int _startState = 24;
- enterRecursionRule(_localctx, 24, RULE_kv_list, _p);
+ int _startState = 30;
+ enterRecursionRule(_localctx, 30, RULE_kv_list, _p);
try {
int _alt;
enterOuterAlt(_localctx, 1);
{
{
- setState(162);
+ setState(174);
identifier_operand();
- setState(163);
+ setState(175);
match(COLON);
- setState(164);
+ setState(176);
transformation_expr();
}
_ctx.stop = _input.LT(-1);
- setState(174);
+ setState(186);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,12,_ctx);
while ( _alt!=2 && _alt!=org.antlr.v4.runtime.atn.ATN.INVALID_ALT_NUMBER ) {
@@ -1299,20 +1416,20 @@ public class StellarParser extends Parser {
{
_localctx = new Kv_listContext(_parentctx, _parentState);
pushNewRecursionContext(_localctx, _startState, RULE_kv_list);
- setState(166);
+ setState(178);
if (!(precpred(_ctx, 1))) throw new FailedPredicateException(this, "precpred(_ctx, 1)");
- setState(167);
+ setState(179);
match(COMMA);
- setState(168);
+ setState(180);
identifier_operand();
- setState(169);
+ setState(181);
match(COLON);
- setState(170);
+ setState(182);
transformation_expr();
}
}
}
- setState(176);
+ setState(188);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,12,_ctx);
}
@@ -1351,27 +1468,27 @@ public class StellarParser extends Parser {
public final Map_entityContext map_entity() throws RecognitionException {
Map_entityContext _localctx = new Map_entityContext(_ctx, getState());
- enterRule(_localctx, 26, RULE_map_entity);
+ enterRule(_localctx, 32, RULE_map_entity);
try {
- setState(183);
+ setState(195);
switch ( getInterpreter().adaptivePredict(_input,13,_ctx) ) {
case 1:
enterOuterAlt(_localctx, 1);
{
- setState(177);
+ setState(189);
match(LBRACE);
- setState(178);
+ setState(190);
kv_list(0);
- setState(179);
+ setState(191);
match(RBRACE);
}
break;
case 2:
enterOuterAlt(_localctx, 2);
{
- setState(181);
+ setState(193);
match(LBRACE);
- setState(182);
+ setState(194);
match(RBRACE);
}
break;
@@ -1459,8 +1576,8 @@ public class StellarParser extends Parser {
int _parentState = getState();
Arithmetic_exprContext _localctx = new Arithmetic_exprContext(_ctx, _parentState);
Arithmetic_exprContext _prevctx = _localctx;
- int _startState = 28;
- enterRecursionRule(_localctx, 28, RULE_arithmetic_expr, _p);
+ int _startState = 34;
+ enterRecursionRule(_localctx, 34, RULE_arithmetic_expr, _p);
try {
int _alt;
enterOuterAlt(_localctx, 1);
@@ -1470,11 +1587,11 @@ public class StellarParser extends Parser {
_ctx = _localctx;
_prevctx = _localctx;
- setState(186);
+ setState(198);
arithmetic_expr_mul(0);
}
_ctx.stop = _input.LT(-1);
- setState(196);
+ setState(208);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,15,_ctx);
while ( _alt!=2 && _alt!=org.antlr.v4.runtime.atn.ATN.INVALID_ALT_NUMBER ) {
@@ -1482,17 +1599,17 @@ public class StellarParser extends Parser {
if ( _parseListeners!=null ) triggerExitRuleEvent();
_prevctx = _localctx;
{
- setState(194);
+ setState(206);
switch ( getInterpreter().adaptivePredict(_input,14,_ctx) ) {
case 1:
{
_localctx = new ArithExpr_plusContext(new Arithmetic_exprContext(_parentctx, _parentState));
pushNewRecursionContext(_localctx, _startState, RULE_arithmetic_expr);
- setState(188);
+ setState(200);
if (!(precpred(_ctx, 2))) throw new FailedPredicateException(this, "precpred(_ctx, 2)");
- setState(189);
+ setState(201);
match(PLUS);
- setState(190);
+ setState(202);
arithmetic_expr_mul(0);
}
break;
@@ -1500,18 +1617,18 @@ public class StellarParser extends Parser {
{
_localctx = new ArithExpr_minusContext(new Arithmetic_exprContext(_parentctx, _parentState));
pushNewRecursionContext(_localctx, _startState, RULE_arithmetic_expr);
- setState(191);
+ setState(203);
if (!(precpred(_ctx, 1))) throw new FailedPredicateException(this, "precpred(_ctx, 1)");
- setState(192);
+ setState(204);
match(MINUS);
- setState(193);
+ setState(205);
arithmetic_expr_mul(0);
}
break;
}
}
}
- setState(198);
+ setState(210);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,15,_ctx);
}
@@ -1599,8 +1716,8 @@ public class StellarParser extends Parser {
int _parentState = getState();
Arithmetic_expr_mulContext _localctx = new Arithmetic_expr_mulContext(_ctx, _parentState);
Arithmetic_expr_mulContext _prevctx = _localctx;
- int _startState = 30;
- enterRecursionRule(_localctx, 30, RULE_arithmetic_expr_mul, _p);
+ int _startState = 36;
+ enterRecursionRule(_localctx, 36, RULE_arithmetic_expr_mul, _p);
try {
int _alt;
enterOuterAlt(_localctx, 1);
@@ -1610,11 +1727,11 @@ public class StellarParser extends Parser {
_ctx = _localctx;
_prevctx = _localctx;
- setState(200);
+ setState(212);
arithmetic_operands();
}
_ctx.stop = _input.LT(-1);
- setState(210);
+ setState(222);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,17,_ctx);
while ( _alt!=2 && _alt!=org.antlr.v4.runtime.atn.ATN.INVALID_ALT_NUMBER ) {
@@ -1622,17 +1739,17 @@ public class StellarParser extends Parser {
if ( _parseListeners!=null ) triggerExitRuleEvent();
_prevctx = _localctx;
{
- setState(208);
+ setState(220);
switch ( getInterpreter().adaptivePredict(_input,16,_ctx) ) {
case 1:
{
_localctx = new ArithExpr_mulContext(new Arithmetic_expr_mulContext(_parentctx, _parentState));
pushNewRecursionContext(_localctx, _startState, RULE_arithmetic_expr_mul);
- setState(202);
+ setState(214);
if (!(precpred(_ctx, 2))) throw new FailedPredicateException(this, "precpred(_ctx, 2)");
- setState(203);
+ setState(215);
match(MUL);
- setState(204);
+ setState(216);
arithmetic_expr_mul(3);
}
break;
@@ -1640,18 +1757,18 @@ public class StellarParser extends Parser {
{
_localctx = new ArithExpr_divContext(new Arithmetic_expr_mulContext(_parentctx, _parentState));
pushNewRecursionContext(_localctx, _startState, RULE_arithmetic_expr_mul);
- setState(205);
+ setState(217);
if (!(precpred(_ctx, 1))) throw new FailedPredicateException(this, "precpred(_ctx, 1)");
- setState(206);
+ setState(218);
match(DIV);
- setState(207);
+ setState(219);
arithmetic_expr_mul(2);
}
break;
}
}
}
- setState(212);
+ setState(224);
_errHandler.sync(this);
_alt = getInterpreter().adaptivePredict(_input,17,_ctx);
}
@@ -1697,14 +1814,14 @@ public class StellarParser extends Parser {
public final FunctionsContext functions() throws RecognitionException {
FunctionsContext _localctx = new FunctionsContext(_ctx, getState());
- enterRule(_localctx, 32, RULE_functions);
+ enterRule(_localctx, 38, RULE_functions);
try {
_localctx = new TransformationFuncContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(213);
+ setState(225);
match(IDENTIFIER);
- setState(214);
+ setState(226);
func_args();
}
}
@@ -1839,15 +1956,15 @@ public class StellarParser extends Parser {
public final Arithmetic_operandsContext arithmetic_operands() throws RecognitionException {
Arithmetic_operandsContext _localctx = new Arithmetic_operandsContext(_ctx, getState());
- enterRule(_localctx, 34, RULE_arithmetic_operands);
+ enterRule(_localctx, 40, RULE_arithmetic_operands);
try {
- setState(230);
+ setState(242);
switch ( getInterpreter().adaptivePredict(_input,18,_ctx) ) {
case 1:
_localctx = new NumericFunctionsContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(216);
+ setState(228);
functions();
}
break;
@@ -1855,7 +1972,7 @@ public class StellarParser extends Parser {
_localctx = new DoubleLiteralContext(_localctx);
enterOuterAlt(_localctx, 2);
{
- setState(217);
+ setState(229);
match(DOUBLE_LITERAL);
}
break;
@@ -1863,7 +1980,7 @@ public class StellarParser extends Parser {
_localctx = new IntLiteralContext(_localctx);
enterOuterAlt(_localctx, 3);
{
- setState(218);
+ setState(230);
match(INT_LITERAL);
}
break;
@@ -1871,7 +1988,7 @@ public class StellarParser extends Parser {
_localctx = new LongLiteralContext(_localctx);
enterOuterAlt(_localctx, 4);
{
- setState(219);
+ setState(231);
match(LONG_LITERAL);
}
break;
@@ -1879,7 +1996,7 @@ public class StellarParser extends Parser {
_localctx = new FloatLiteralContext(_localctx);
enterOuterAlt(_localctx, 5);
{
- setState(220);
+ setState(232);
match(FLOAT_LITERAL);
}
break;
@@ -1887,7 +2004,7 @@ public class StellarParser extends Parser {
_localctx = new VariableContext(_localctx);
enterOuterAlt(_localctx, 6);
{
- setState(221);
+ setState(233);
match(IDENTIFIER);
}
break;
@@ -1895,11 +2012,11 @@ public class StellarParser extends Parser {
_localctx = new ParenArithContext(_localctx);
enterOuterAlt(_localctx, 7);
{
- setState(222);
+ setState(234);
match(LPAREN);
- setState(223);
+ setState(235);
arithmetic_expr(0);
- setState(224);
+ setState(236);
match(RPAREN);
}
break;
@@ -1907,11 +2024,11 @@ public class StellarParser extends Parser {
_localctx = new CondExprContext(_localctx);
enterOuterAlt(_localctx, 8);
{
- setState(226);
+ setState(238);
match(LPAREN);
- setState(227);
+ setState(239);
conditional_expr();
- setState(228);
+ setState(240);
match(RPAREN);
}
break;
@@ -2080,16 +2197,16 @@ public class StellarParser extends Parser {
public final Identifier_operandContext identifier_operand() throws RecognitionException {
Identifier_operandContext _localctx = new Identifier_operandContext(_ctx, getState());
- enterRule(_localctx, 36, RULE_identifier_operand);
+ enterRule(_localctx, 42, RULE_identifier_operand);
int _la;
try {
- setState(248);
+ setState(260);
switch ( getInterpreter().adaptivePredict(_input,19,_ctx) ) {
case 1:
_localctx = new LogicalConstContext(_localctx);
enterOuterAlt(_localctx, 1);
{
- setState(232);
+ setState(244);
_la = _input.LA(1);
if ( !(_la==TRUE || _la==FALSE) ) {
_errHandler.recoverInline(this);
@@ -2102,7 +2219,7 @@ public class StellarParser extends Parser {
_localctx = new LambdaWithArgsExprContext(_localctx);
enterOuterAlt(_localctx, 2);
{
- setState(233);
+ setState(245);
lambda_with_args();
}
break;
@@ -2110,7 +2227,7 @@ public class StellarParser extends Parser {
_localctx = new LambdaWithoutArgsExprContext(_localctx);
enterOuterAlt(_localctx, 3);
{
- setState(234);
+ setState(246);
lambda_without_args();
}
break;
@@ -2118,7 +2235,7 @@ public class StellarParser extends Parser {
_localctx = new ArithmeticOperandsContext(_localctx);
enterOuterAlt(_localctx, 4);
{
- setState(235);
+ setState(247);
arithmetic_expr(0);
}
break;
@@ -2126,7 +2243,7 @@ public class StellarParser extends Parser {
_localctx = new StringLiteralContext(_localctx);
enterOuterAlt(_localctx, 5);
{
- setState(236);
+ setState(248);
match(STRING_LITERAL);
}
break;
@@ -2134,7 +2251,7 @@ public class StellarParser extends Parser {
_localctx = new ListContext(_localctx);
enterOuterAlt(_localctx, 6);
{
- setState(237);
+ setState(249);
list_entity();
}
break;
@@ -2142,7 +2259,7 @@ public class StellarParser extends Parser {
_localctx = new MapConstContext(_localctx);
enterOuterAlt(_localctx, 7);
{
- setState(238);
+ setState(250);
map_entity();
}
break;
@@ -2150,7 +2267,7 @@ public class StellarParser extends Parser {
_localctx = new NullConstContext(_localctx);
enterOuterAlt(_localctx, 8);
{
- setState(239);
+ setState(251);
match(NULL);
}
break;
@@ -2158,13 +2275,13 @@ public class StellarParser extends Parser {
_localctx = new ExistsFuncContext(_localctx);
enterOuterAlt(_localctx, 9);
{
- setState(240);
+ setState(252);
match(EXISTS);
- setState(241);
+ setState(253);
match(LPAREN);
- setState(242);
+ setState(254);
match(IDENTIFIER);
- setState(243);
+ setState(255);
match(RPAREN);
}
break;
@@ -2172,11 +2289,11 @@ public class StellarParser extends Parser {
_localctx = new CondExpr_parenContext(_localctx);
enterOuterAlt(_localctx, 10);
{
- setState(244);
+ setState(256);
match(LPAREN);
- setState(245);
+ setState(257);
conditional_expr();
- setState(246);
+ setState(258);
match(RPAREN);
}
break;
@@ -2216,17 +2333,17 @@ public class StellarParser extends Parser {
public final Lambda_without_argsContext lambda_without_args() throws RecognitionException {
Lambda_without_argsContext _localctx = new Lambda_without_argsContext(_ctx, getState());
- enterRule(_localctx, 38, RULE_lambda_without_args);
+ enterRule(_localctx, 44, RULE_lambda_without_args);
try {
enterOuterAlt(_localctx, 1);
{
- setState(250);
+ setState(262);
match(LPAREN);
- setState(251);
+ setState(263);
match(RPAREN);
- setState(252);
+ setState(264);
match(LAMBDA_OP);
- setState(253);
+ setState(265);
transformation_expr();
}
}
@@ -2270,33 +2387,33 @@ public class StellarParser extends Parser {
public final Lambda_with_argsContext lambda_with_args() throws RecognitionException {
Lambda_with_argsContext _localctx = new Lambda_with_argsContext(_ctx, getState());
- enterRule(_localctx, 40, RULE_lambda_with_args);
+ enterRule(_localctx, 46, RULE_lambda_with_args);
try {
- setState(265);
+ setState(277);
switch (_input.LA(1)) {
case LPAREN:
enterOuterAlt(_localctx, 1);
{
- setState(255);
+ setState(267);
match(LPAREN);
- setState(256);
+ setState(268);
lambda_variables();
- setState(257);
+ setState(269);
match(RPAREN);
- setState(258);
+ setState(270);
match(LAMBDA_OP);
- setState(259);
+ setState(271);
transformation_expr();
}
break;
case IDENTIFIER:
enterOuterAlt(_localctx, 2);
{
- setState(261);
+ setState(273);
single_lambda_variable();
- setState(262);
+ setState(274);
match(LAMBDA_OP);
- setState(263);
+ setState(275);
transformation_expr();
}
break;
@@ -2342,26 +2459,26 @@ public class StellarParser extends Parser {
public final Lambda_variablesContext lambda_variables() throws RecognitionException {
Lambda_variablesContext _localctx = new Lambda_variablesContext(_ctx, getState());
- enterRule(_localctx, 42, RULE_lambda_variables);
+ enterRule(_localctx, 48, RULE_lambda_variables);
int _la;
try {
enterOuterAlt(_localctx, 1);
{
- setState(267);
+ setState(279);
lambda_variable();
- setState(272);
+ setState(284);
_errHandler.sync(this);
_la = _input.LA(1);
while (_la==COMMA) {
{
{
- setState(268);
+ setState(280);
match(COMMA);
- setState(269);
+ setState(281);
lambda_variable();
}
}
- setState(274);
+ setState(286);
_errHandler.sync(this);
_la = _input.LA(1);
}
@@ -2398,11 +2515,11 @@ public class StellarParser extends Parser {
public final Single_lambda_variableContext single_lambda_variable() throws RecognitionException {
Single_lambda_variableContext _localctx = new Single_lambda_variableContext(_ctx, getState());
- enterRule(_localctx, 44, RULE_single_lambda_variable);
+ enterRule(_localctx, 50, RULE_single_lambda_variable);
try {
enterOuterAlt(_localctx, 1);
{
- setState(275);
+ setState(287);
lambda_variable();
}
}
@@ -2435,11 +2552,11 @@ public class StellarParser extends Parser {
public final Lambda_variableContext lambda_variable() throws RecognitionException {
Lambda_variableContext _localctx = new Lambda_variableContext(_ctx, getState());
- enterRule(_localctx, 46, RULE_lambda_variable);
+ enterRule(_localctx, 52, RULE_lambda_variable);
try {
enterOuterAlt(_localctx, 1);
{
- setState(277);
+ setState(289);
match(IDENTIFIER);
}
}
@@ -2456,15 +2573,15 @@ public class StellarParser extends Parser {
public boolean sempred(RuleContext _localctx, int ruleIndex, int predIndex) {
switch (ruleIndex) {
- case 6:
+ case 9:
return comparison_expr_sempred((Comparison_exprContext)_localctx, predIndex);
- case 10:
+ case 13:
return op_list_sempred((Op_listContext)_localctx, predIndex);
- case 12:
+ case 15:
return kv_list_sempred((Kv_listContext)_localctx, predIndex);
- case 14:
+ case 17:
return arithmetic_expr_sempred((Arithmetic_exprContext)_localctx, predIndex);
- case 15:
+ case 18:
return arithmetic_expr_mul_sempred((Arithmetic_expr_mulContext)_localctx, predIndex);
}
return true;
@@ -2512,100 +2629,105 @@ public class StellarParser extends Parser {
}
public static final String _serializedATN =
- "\3\u0430\ud6d1\u8206\uad2d\u4417\uaef1\u8d80\uaadd\3.\u011a\4\2\t\2\4"+
+ "\3\u0430\ud6d1\u8206\uad2d\u4417\uaef1\u8d80\uaadd\3.\u0126\4\2\t\2\4"+
"\3\t\3\4\4\t\4\4\5\t\5\4\6\t\6\4\7\t\7\4\b\t\b\4\t\t\t\4\n\t\n\4\13\t"+
"\13\4\f\t\f\4\r\t\r\4\16\t\16\4\17\t\17\4\20\t\20\4\21\t\21\4\22\t\22"+
"\4\23\t\23\4\24\t\24\4\25\t\25\4\26\t\26\4\27\t\27\4\30\t\30\4\31\t\31"+
- "\3\2\3\2\3\2\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\3\5\3@\n\3\3\4\3\4"+
- "\3\4\3\4\3\4\3\4\3\4\3\4\3\4\3\4\3\4\3\4\3\4\5\4O\n\4\3\5\3\5\3\5\3\5"+
- "\3\5\3\5\3\5\3\5\3\5\5\5Z\n\5\3\6\3\6\5\6^\n\6\3\7\3\7\3\7\3\7\3\7\3\7"+
- "\3\7\3\7\5\7h\n\7\3\b\3\b\3\b\3\b\3\b\3\b\3\b\3\b\3\b\3\b\3\b\5\bu\n\b"+
- "\3\b\3\b\3\b\3\b\7\b{\n\b\f\b\16\b~\13\b\3\t\3\t\3\n\3\n\3\13\3\13\3\13"+
- "\3\13\3\13\3\13\5\13\u008a\n\13\3\f\3\f\3\f\5\f\u008f\n\f\3\f\3\f\3\f"+
- "\3\f\3\f\3\f\7\f\u0097\n\f\f\f\16\f\u009a\13\f\3\r\3\r\3\r\3\r\3\r\3\r"+
- "\5\r\u00a2\n\r\3\16\3\16\3\16\3\16\3\16\3\16\3\16\3\16\3\16\3\16\3\16"+
- "\7\16\u00af\n\16\f\16\16\16\u00b2\13\16\3\17\3\17\3\17\3\17\3\17\3\17"+
- "\5\17\u00ba\n\17\3\20\3\20\3\20\3\20\3\20\3\20\3\20\3\20\3\20\7\20\u00c5"+
- "\n\20\f\20\16\20\u00c8\13\20\3\21\3\21\3\21\3\21\3\21\3\21\3\21\3\21\3"+
- "\21\7\21\u00d3\n\21\f\21\16\21\u00d6\13\21\3\22\3\22\3\22\3\23\3\23\3"+
- "\23\3\23\3\23\3\23\3\23\3\23\3\23\3\23\3\23\3\23\3\23\3\23\5\23\u00e9"+
- "\n\23\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24"+
- "\3\24\3\24\3\24\5\24\u00fb\n\24\3\25\3\25\3\25\3\25\3\25\3\26\3\26\3\26"+
- "\3\26\3\26\3\26\3\26\3\26\3\26\3\26\5\26\u010c\n\26\3\27\3\27\3\27\7\27"+
- "\u0111\n\27\f\27\16\27\u0114\13\27\3\30\3\30\3\31\3\31\3\31\2\7\16\26"+
- "\32\36 \32\2\4\6\b\n\f\16\20\22\24\26\30\32\34\36 \"$&(*,.\60\2\4\3\2"+
- "\16\23\3\2\f\r\u012c\2\62\3\2\2\2\4?\3\2\2\2\6N\3\2\2\2\bY\3\2\2\2\n]"+
- "\3\2\2\2\fg\3\2\2\2\16t\3\2\2\2\20\177\3\2\2\2\22\u0081\3\2\2\2\24\u0089"+
- "\3\2\2\2\26\u008e\3\2\2\2\30\u00a1\3\2\2\2\32\u00a3\3\2\2\2\34\u00b9\3"+
- "\2\2\2\36\u00bb\3\2\2\2 \u00c9\3\2\2\2\"\u00d7\3\2\2\2$\u00e8\3\2\2\2"+
- "&\u00fa\3\2\2\2(\u00fc\3\2\2\2*\u010b\3\2\2\2,\u010d\3\2\2\2.\u0115\3"+
- "\2\2\2\60\u0117\3\2\2\2\62\63\5\4\3\2\63\64\7\2\2\3\64\3\3\2\2\2\65@\5"+
- "\6\4\2\66\67\7\"\2\2\678\5\4\3\289\7#\2\29@\3\2\2\2:@\5\36\20\2;@\5\20"+
- "\t\2<@\5\16\b\2=@\5\b\5\2>@\5\f\7\2?\65\3\2\2\2?\66\3\2\2\2?:\3\2\2\2"+
- "?;\3\2\2\2?<\3\2\2\2?=\3\2\2\2?>\3\2\2\2@\5\3\2\2\2AB\5\b\5\2BC\7\24\2"+
- "\2CD\5\4\3\2DE\7\25\2\2EF\5\4\3\2FO\3\2\2\2GH\7\26\2\2HI\5\b\5\2IJ\7\27"+
- "\2\2JK\5\4\3\2KL\7\30\2\2LM\5\4\3\2MO\3\2\2\2NA\3\2\2\2NG\3\2\2\2O\7\3"+
- "\2\2\2PQ\5\n\6\2QR\7\t\2\2RS\5\b\5\2SZ\3\2\2\2TU\5\n\6\2UV\7\n\2\2VW\5"+
- "\b\5\2WZ\3\2\2\2XZ\5\n\6\2YP\3\2\2\2YT\3\2\2\2YX\3\2\2\2Z\t\3\2\2\2[^"+
- "\5\16\b\2\\^\5\f\7\2][\3\2\2\2]\\\3\2\2\2^\13\3\2\2\2_`\5&\24\2`a\7\3"+
- "\2\2ab\5\n\6\2bh\3\2\2\2cd\5&\24\2de\7$\2\2ef\5\n\6\2fh\3\2\2\2g_\3\2"+
- "\2\2gc\3\2\2\2h\r\3\2\2\2ij\b\b\1\2jk\7\13\2\2kl\7\"\2\2lm\5\b\5\2mn\7"+
- "#\2\2nu\3\2\2\2op\7\"\2\2pq\5\b\5\2qr\7#\2\2ru\3\2\2\2su\5&\24\2ti\3\2"+
- "\2\2to\3\2\2\2ts\3\2\2\2u|\3\2\2\2vw\f\6\2\2wx\5\22\n\2xy\5\16\b\7y{\3"+
- "\2\2\2zv\3\2\2\2{~\3\2\2\2|z\3\2\2\2|}\3\2\2\2}\17\3\2\2\2~|\3\2\2\2\177"+
- "\u0080\5&\24\2\u0080\21\3\2\2\2\u0081\u0082\t\2\2\2\u0082\23\3\2\2\2\u0083"+
- "\u0084\7\"\2\2\u0084\u0085\5\26\f\2\u0085\u0086\7#\2\2\u0086\u008a\3\2"+
- "\2\2\u0087\u0088\7\"\2\2\u0088\u008a\7#\2\2\u0089\u0083\3\2\2\2\u0089"+
- "\u0087\3\2\2\2\u008a\25\3\2\2\2\u008b\u008c\b\f\1\2\u008c\u008f\5&\24"+
- "\2\u008d\u008f\5\6\4\2\u008e\u008b\3\2\2\2\u008e\u008d\3\2\2\2\u008f\u0098"+
- "\3\2\2\2\u0090\u0091\f\5\2\2\u0091\u0092\7\7\2\2\u0092\u0097\5&\24\2\u0093"+
- "\u0094\f\3\2\2\u0094\u0095\7\7\2\2\u0095\u0097\5\6\4\2\u0096\u0090\3\2"+
- "\2\2\u0096\u0093\3\2\2\2\u0097\u009a\3\2\2\2\u0098\u0096\3\2\2\2\u0098"+
- "\u0099\3\2\2\2\u0099\27\3\2\2\2\u009a\u0098\3\2\2\2\u009b\u009c\7 \2\2"+
- "\u009c\u00a2\7!\2\2\u009d\u009e\7 \2\2\u009e\u009f\5\26\f\2\u009f\u00a0"+
- "\7!\2\2\u00a0\u00a2\3\2\2\2\u00a1\u009b\3\2\2\2\u00a1\u009d\3\2\2\2\u00a2"+
- "\31\3\2\2\2\u00a3\u00a4\b\16\1\2\u00a4\u00a5\5&\24\2\u00a5\u00a6\7\25"+
- "\2\2\u00a6\u00a7\5\4\3\2\u00a7\u00b0\3\2\2\2\u00a8\u00a9\f\3\2\2\u00a9"+
- "\u00aa\7\7\2\2\u00aa\u00ab\5&\24\2\u00ab\u00ac\7\25\2\2\u00ac\u00ad\5"+
- "\4\3\2\u00ad\u00af\3\2\2\2\u00ae\u00a8\3\2\2\2\u00af\u00b2\3\2\2\2\u00b0"+
- "\u00ae\3\2\2\2\u00b0\u00b1\3\2\2\2\u00b1\33\3\2\2\2\u00b2\u00b0\3\2\2"+
- "\2\u00b3\u00b4\7\36\2\2\u00b4\u00b5\5\32\16\2\u00b5\u00b6\7\37\2\2\u00b6"+
- "\u00ba\3\2\2\2\u00b7\u00b8\7\36\2\2\u00b8\u00ba\7\37\2\2\u00b9\u00b3\3"+
- "\2\2\2\u00b9\u00b7\3\2\2\2\u00ba\35\3\2\2\2\u00bb\u00bc\b\20\1\2\u00bc"+
- "\u00bd\5 \21\2\u00bd\u00c6\3\2\2\2\u00be\u00bf\f\4\2\2\u00bf\u00c0\7\33"+
- "\2\2\u00c0\u00c5\5 \21\2\u00c1\u00c2\f\3\2\2\u00c2\u00c3\7\32\2\2\u00c3"+
- "\u00c5\5 \21\2\u00c4\u00be\3\2\2\2\u00c4\u00c1\3\2\2\2\u00c5\u00c8\3\2"+
- "\2\2\u00c6\u00c4\3\2\2\2\u00c6\u00c7\3\2\2\2\u00c7\37\3\2\2\2\u00c8\u00c6"+
- "\3\2\2\2\u00c9\u00ca\b\21\1\2\u00ca\u00cb\5$\23\2\u00cb\u00d4\3\2\2\2"+
- "\u00cc\u00cd\f\4\2\2\u00cd\u00ce\7\35\2\2\u00ce\u00d3\5 \21\5\u00cf\u00d0"+
- "\f\3\2\2\u00d0\u00d1\7\34\2\2\u00d1\u00d3\5 \21\4\u00d2\u00cc\3\2\2\2"+
- "\u00d2\u00cf\3\2\2\2\u00d3\u00d6\3\2\2\2\u00d4\u00d2\3\2\2\2\u00d4\u00d5"+
- "\3\2\2\2\u00d5!\3\2\2\2\u00d6\u00d4\3\2\2\2\u00d7\u00d8\7+\2\2\u00d8\u00d9"+
- "\5\24\13\2\u00d9#\3\2\2\2\u00da\u00e9\5\"\22\2\u00db\u00e9\7(\2\2\u00dc"+
- "\u00e9\7\'\2\2\u00dd\u00e9\7*\2\2\u00de\u00e9\7)\2\2\u00df\u00e9\7+\2"+
- "\2\u00e0\u00e1\7\"\2\2\u00e1\u00e2\5\36\20\2\u00e2\u00e3\7#\2\2\u00e3"+
- "\u00e9\3\2\2\2\u00e4\u00e5\7\"\2\2\u00e5\u00e6\5\6\4\2\u00e6\u00e7\7#"+
- "\2\2\u00e7\u00e9\3\2\2\2\u00e8\u00da\3\2\2\2\u00e8\u00db\3\2\2\2\u00e8"+
- "\u00dc\3\2\2\2\u00e8\u00dd\3\2\2\2\u00e8\u00de\3\2\2\2\u00e8\u00df\3\2"+
- "\2\2\u00e8\u00e0\3\2\2\2\u00e8\u00e4\3\2\2\2\u00e9%\3\2\2\2\u00ea\u00fb"+
- "\t\3\2\2\u00eb\u00fb\5*\26\2\u00ec\u00fb\5(\25\2\u00ed\u00fb\5\36\20\2"+
- "\u00ee\u00fb\7,\2\2\u00ef\u00fb\5\30\r\2\u00f0\u00fb\5\34\17\2\u00f1\u00fb"+
- "\7\31\2\2\u00f2\u00f3\7%\2\2\u00f3\u00f4\7\"\2\2\u00f4\u00f5\7+\2\2\u00f5"+
- "\u00fb\7#\2\2\u00f6\u00f7\7\"\2\2\u00f7\u00f8\5\6\4\2\u00f8\u00f9\7#\2"+
- "\2\u00f9\u00fb\3\2\2\2\u00fa\u00ea\3\2\2\2\u00fa\u00eb\3\2\2\2\u00fa\u00ec"+
- "\3\2\2\2\u00fa\u00ed\3\2\2\2\u00fa\u00ee\3\2\2\2\u00fa\u00ef\3\2\2\2\u00fa"+
- "\u00f0\3\2\2\2\u00fa\u00f1\3\2\2\2\u00fa\u00f2\3\2\2\2\u00fa\u00f6\3\2"+
- "\2\2\u00fb\'\3\2\2\2\u00fc\u00fd\7\"\2\2\u00fd\u00fe\7#\2\2\u00fe\u00ff"+
- "\7\4\2\2\u00ff\u0100\5\4\3\2\u0100)\3\2\2\2\u0101\u0102\7\"\2\2\u0102"+
- "\u0103\5,\27\2\u0103\u0104\7#\2\2\u0104\u0105\7\4\2\2\u0105\u0106\5\4"+
- "\3\2\u0106\u010c\3\2\2\2\u0107\u0108\5.\30\2\u0108\u0109\7\4\2\2\u0109"+
- "\u010a\5\4\3\2\u010a\u010c\3\2\2\2\u010b\u0101\3\2\2\2\u010b\u0107\3\2"+
- "\2\2\u010c+\3\2\2\2\u010d\u0112\5\60\31\2\u010e\u010f\7\7\2\2\u010f\u0111"+
- "\5\60\31\2\u0110\u010e\3\2\2\2\u0111\u0114\3\2\2\2\u0112\u0110\3\2\2\2"+
- "\u0112\u0113\3\2\2\2\u0113-\3\2\2\2\u0114\u0112\3\2\2\2\u0115\u0116\5"+
- "\60\31\2\u0116/\3\2\2\2\u0117\u0118\7+\2\2\u0118\61\3\2\2\2\30?NY]gt|"+
- "\u0089\u008e\u0096\u0098\u00a1\u00b0\u00b9\u00c4\u00c6\u00d2\u00d4\u00e8"+
- "\u00fa\u010b\u0112";
+ "\4\32\t\32\4\33\t\33\4\34\t\34\3\2\3\2\3\2\3\3\3\3\3\3\3\3\3\3\3\3\3\3"+
+ "\3\3\3\3\3\3\5\3F\n\3\3\4\3\4\3\5\3\5\3\6\3\6\3\7\3\7\3\7\3\7\3\7\3\7"+
+ "\3\7\3\7\3\7\3\7\3\7\3\7\3\7\5\7[\n\7\3\b\3\b\3\b\3\b\3\b\3\b\3\b\3\b"+
+ "\3\b\5\bf\n\b\3\t\3\t\5\tj\n\t\3\n\3\n\3\n\3\n\3\n\3\n\3\n\3\n\5\nt\n"+
+ "\n\3\13\3\13\3\13\3\13\3\13\3\13\3\13\3\13\3\13\3\13\3\13\5\13\u0081\n"+
+ "\13\3\13\3\13\3\13\3\13\7\13\u0087\n\13\f\13\16\13\u008a\13\13\3\f\3\f"+
+ "\3\r\3\r\3\16\3\16\3\16\3\16\3\16\3\16\5\16\u0096\n\16\3\17\3\17\3\17"+
+ "\5\17\u009b\n\17\3\17\3\17\3\17\3\17\3\17\3\17\7\17\u00a3\n\17\f\17\16"+
+ "\17\u00a6\13\17\3\20\3\20\3\20\3\20\3\20\3\20\5\20\u00ae\n\20\3\21\3\21"+
+ "\3\21\3\21\3\21\3\21\3\21\3\21\3\21\3\21\3\21\7\21\u00bb\n\21\f\21\16"+
+ "\21\u00be\13\21\3\22\3\22\3\22\3\22\3\22\3\22\5\22\u00c6\n\22\3\23\3\23"+
+ "\3\23\3\23\3\23\3\23\3\23\3\23\3\23\7\23\u00d1\n\23\f\23\16\23\u00d4\13"+
+ "\23\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\3\24\7\24\u00df\n\24\f\24"+
+ "\16\24\u00e2\13\24\3\25\3\25\3\25\3\26\3\26\3\26\3\26\3\26\3\26\3\26\3"+
+ "\26\3\26\3\26\3\26\3\26\3\26\3\26\5\26\u00f5\n\26\3\27\3\27\3\27\3\27"+
+ "\3\27\3\27\3\27\3\27\3\27\3\27\3\27\3\27\3\27\3\27\3\27\3\27\5\27\u0107"+
+ "\n\27\3\30\3\30\3\30\3\30\3\30\3\31\3\31\3\31\3\31\3\31\3\31\3\31\3\31"+
+ "\3\31\3\31\5\31\u0118\n\31\3\32\3\32\3\32\7\32\u011d\n\32\f\32\16\32\u0120"+
+ "\13\32\3\33\3\33\3\34\3\34\3\34\2\7\24\34 $&\35\2\4\6\b\n\f\16\20\22\24"+
+ "\26\30\32\34\36 \"$&(*,.\60\62\64\66\2\4\3\2\16\23\3\2\f\r\u0135\28\3"+
+ "\2\2\2\4E\3\2\2\2\6G\3\2\2\2\bI\3\2\2\2\nK\3\2\2\2\fZ\3\2\2\2\16e\3\2"+
+ "\2\2\20i\3\2\2\2\22s\3\2\2\2\24\u0080\3\2\2\2\26\u008b\3\2\2\2\30\u008d"+
+ "\3\2\2\2\32\u0095\3\2\2\2\34\u009a\3\2\2\2\36\u00ad\3\2\2\2 \u00af\3\2"+
+ "\2\2\"\u00c5\3\2\2\2$\u00c7\3\2\2\2&\u00d5\3\2\2\2(\u00e3\3\2\2\2*\u00f4"+
+ "\3\2\2\2,\u0106\3\2\2\2.\u0108\3\2\2\2\60\u0117\3\2\2\2\62\u0119\3\2\2"+
+ "\2\64\u0121\3\2\2\2\66\u0123\3\2\2\289\5\4\3\29:\7\2\2\3:\3\3\2\2\2;F"+
+ "\5\f\7\2<=\7\"\2\2=>\5\4\3\2>?\7#\2\2?F\3\2\2\2@F\5$\23\2AF\5\26\f\2B"+
+ "F\5\24\13\2CF\5\16\b\2DF\5\22\n\2E;\3\2\2\2E<\3\2\2\2E@\3\2\2\2EA\3\2"+
+ "\2\2EB\3\2\2\2EC\3\2\2\2ED\3\2\2\2F\5\3\2\2\2GH\5\16\b\2H\7\3\2\2\2IJ"+
+ "\5\4\3\2J\t\3\2\2\2KL\5\4\3\2L\13\3\2\2\2MN\5\6\4\2NO\7\24\2\2OP\5\b\5"+
+ "\2PQ\7\25\2\2QR\5\n\6\2R[\3\2\2\2ST\7\26\2\2TU\5\6\4\2UV\7\27\2\2VW\5"+
+ "\b\5\2WX\7\30\2\2XY\5\n\6\2Y[\3\2\2\2ZM\3\2\2\2ZS\3\2\2\2[\r\3\2\2\2\\"+
+ "]\5\20\t\2]^\7\t\2\2^_\5\16\b\2_f\3\2\2\2`a\5\20\t\2ab\7\n\2\2bc\5\16"+
+ "\b\2cf\3\2\2\2df\5\20\t\2e\\\3\2\2\2e`\3\2\2\2ed\3\2\2\2f\17\3\2\2\2g"+
+ "j\5\24\13\2hj\5\22\n\2ig\3\2\2\2ih\3\2\2\2j\21\3\2\2\2kl\5,\27\2lm\7\3"+
+ "\2\2mn\5\20\t\2nt\3\2\2\2op\5,\27\2pq\7$\2\2qr\5\20\t\2rt\3\2\2\2sk\3"+
+ "\2\2\2so\3\2\2\2t\23\3\2\2\2uv\b\13\1\2vw\7\13\2\2wx\7\"\2\2xy\5\16\b"+
+ "\2yz\7#\2\2z\u0081\3\2\2\2{|\7\"\2\2|}\5\16\b\2}~\7#\2\2~\u0081\3\2\2"+
+ "\2\177\u0081\5,\27\2\u0080u\3\2\2\2\u0080{\3\2\2\2\u0080\177\3\2\2\2\u0081"+
+ "\u0088\3\2\2\2\u0082\u0083\f\6\2\2\u0083\u0084\5\30\r\2\u0084\u0085\5"+
+ "\24\13\7\u0085\u0087\3\2\2\2\u0086\u0082\3\2\2\2\u0087\u008a\3\2\2\2\u0088"+
+ "\u0086\3\2\2\2\u0088\u0089\3\2\2\2\u0089\25\3\2\2\2\u008a\u0088\3\2\2"+
+ "\2\u008b\u008c\5,\27\2\u008c\27\3\2\2\2\u008d\u008e\t\2\2\2\u008e\31\3"+
+ "\2\2\2\u008f\u0090\7\"\2\2\u0090\u0091\5\34\17\2\u0091\u0092\7#\2\2\u0092"+
+ "\u0096\3\2\2\2\u0093\u0094\7\"\2\2\u0094\u0096\7#\2\2\u0095\u008f\3\2"+
+ "\2\2\u0095\u0093\3\2\2\2\u0096\33\3\2\2\2\u0097\u0098\b\17\1\2\u0098\u009b"+
+ "\5,\27\2\u0099\u009b\5\f\7\2\u009a\u0097\3\2\2\2\u009a\u0099\3\2\2\2\u009b"+
+ "\u00a4\3\2\2\2\u009c\u009d\f\5\2\2\u009d\u009e\7\7\2\2\u009e\u00a3\5,"+
+ "\27\2\u009f\u00a0\f\3\2\2\u00a0\u00a1\7\7\2\2\u00a1\u00a3\5\f\7\2\u00a2"+
+ "\u009c\3\2\2\2\u00a2\u009f\3\2\2\2\u00a3\u00a6\3\2\2\2\u00a4\u00a2\3\2"+
+ "\2\2\u00a4\u00a5\3\2\2\2\u00a5\35\3\2\2\2\u00a6\u00a4\3\2\2\2\u00a7\u00a8"+
+ "\7 \2\2\u00a8\u00ae\7!\2\2\u00a9\u00aa\7 \2\2\u00aa\u00ab\5\34\17\2\u00ab"+
+ "\u00ac\7!\2\2\u00ac\u00ae\3\2\2\2\u00ad\u00a7\3\2\2\2\u00ad\u00a9\3\2"+
+ "\2\2\u00ae\37\3\2\2\2\u00af\u00b0\b\21\1\2\u00b0\u00b1\5,\27\2\u00b1\u00b2"+
+ "\7\25\2\2\u00b2\u00b3\5\4\3\2\u00b3\u00bc\3\2\2\2\u00b4\u00b5\f\3\2\2"+
+ "\u00b5\u00b6\7\7\2\2\u00b6\u00b7\5,\27\2\u00b7\u00b8\7\25\2\2\u00b8\u00b9"+
+ "\5\4\3\2\u00b9\u00bb\3\2\2\2\u00ba\u00b4\3\2\2\2\u00bb\u00be\3\2\2\2\u00bc"+
+ "\u00ba\3\2\2\2\u00bc\u00bd\3\2\2\2\u00bd!\3\2\2\2\u00be\u00bc\3\2\2\2"+
+ "\u00bf\u00c0\7\36\2\2\u00c0\u00c1\5 \21\2\u00c1\u00c2\7\37\2\2\u00c2\u00c6"+
+ "\3\2\2\2\u00c3\u00c4\7\36\2\2\u00c4\u00c6\7\37\2\2\u00c5\u00bf\3\2\2\2"+
+ "\u00c5\u00c3\3\2\2\2\u00c6#\3\2\2\2\u00c7\u00c8\b\23\1\2\u00c8\u00c9\5"+
+ "&\24\2\u00c9\u00d2\3\2\2\2\u00ca\u00cb\f\4\2\2\u00cb\u00cc\7\33\2\2\u00cc"+
+ "\u00d1\5&\24\2\u00cd\u00ce\f\3\2\2\u00ce\u00cf\7\32\2\2\u00cf\u00d1\5"+
+ "&\24\2\u00d0\u00ca\3\2\2\2\u00d0\u00cd\3\2\2\2\u00d1\u00d4\3\2\2\2\u00d2"+
+ "\u00d0\3\2\2\2\u00d2\u00d3\3\2\2\2\u00d3%\3\2\2\2\u00d4\u00d2\3\2\2\2"+
+ "\u00d5\u00d6\b\24\1\2\u00d6\u00d7\5*\26\2\u00d7\u00e0\3\2\2\2\u00d8\u00d9"+
+ "\f\4\2\2\u00d9\u00da\7\35\2\2\u00da\u00df\5&\24\5\u00db\u00dc\f\3\2\2"+
+ "\u00dc\u00dd\7\34\2\2\u00dd\u00df\5&\24\4\u00de\u00d8\3\2\2\2\u00de\u00db"+
+ "\3\2\2\2\u00df\u00e2\3\2\2\2\u00e0\u00de\3\2\2\2\u00e0\u00e1\3\2\2\2\u00e1"+
+ "\'\3\2\2\2\u00e2\u00e0\3\2\2\2\u00e3\u00e4\7+\2\2\u00e4\u00e5\5\32\16"+
+ "\2\u00e5)\3\2\2\2\u00e6\u00f5\5(\25\2\u00e7\u00f5\7(\2\2\u00e8\u00f5\7"+
+ "\'\2\2\u00e9\u00f5\7*\2\2\u00ea\u00f5\7)\2\2\u00eb\u00f5\7+\2\2\u00ec"+
+ "\u00ed\7\"\2\2\u00ed\u00ee\5$\23\2\u00ee\u00ef\7#\2\2\u00ef\u00f5\3\2"+
+ "\2\2\u00f0\u00f1\7\"\2\2\u00f1\u00f2\5\f\7\2\u00f2\u00f3\7#\2\2\u00f3"+
+ "\u00f5\3\2\2\2\u00f4\u00e6\3\2\2\2\u00f4\u00e7\3\2\2\2\u00f4\u00e8\3\2"+
+ "\2\2\u00f4\u00e9\3\2\2\2\u00f4\u00ea\3\2\2\2\u00f4\u00eb\3\2\2\2\u00f4"+
+ "\u00ec\3\2\2\2\u00f4\u00f0\3\2\2\2\u00f5+\3\2\2\2\u00f6\u0107\t\3\2\2"+
+ "\u00f7\u0107\5\60\31\2\u00f8\u0107\5.\30\2\u00f9\u0107\5$\23\2\u00fa\u0107"+
+ "\7,\2\2\u00fb\u0107\5\36\20\2\u00fc\u0107\5\"\22\2\u00fd\u0107\7\31\2"+
+ "\2\u00fe\u00ff\7%\2\2\u00ff\u0100\7\"\2\2\u0100\u0101\7+\2\2\u0101\u0107"+
+ "\7#\2\2\u0102\u0103\7\"\2\2\u0103\u0104\5\f\7\2\u0104\u0105\7#\2\2\u0105"+
+ "\u0107\3\2\2\2\u0106\u00f6\3\2\2\2\u0106\u00f7\3\2\2\2\u0106\u00f8\3\2"+
+ "\2\2\u0106\u00f9\3\2\2\2\u0106\u00fa\3\2\2\2\u0106\u00fb\3\2\2\2\u0106"+
+ "\u00fc\3\2\2\2\u0106\u00fd\3\2\2\2\u0106\u00fe\3\2\2\2\u0106\u0102\3\2"+
+ "\2\2\u0107-\3\2\2\2\u0108\u0109\7\"\2\2\u0109\u010a\7#\2\2\u010a\u010b"+
+ "\7\4\2\2\u010b\u010c\5\4\3\2\u010c/\3\2\2\2\u010d\u010e\7\"\2\2\u010e"+
+ "\u010f\5\62\32\2\u010f\u0110\7#\2\2\u0110\u0111\7\4\2\2\u0111\u0112\5"+
+ "\4\3\2\u0112\u0118\3\2\2\2\u0113\u0114\5\64\33\2\u0114\u0115\7\4\2\2\u0115"+
+ "\u0116\5\4\3\2\u0116\u0118\3\2\2\2\u0117\u010d\3\2\2\2\u0117\u0113\3\2"+
+ "\2\2\u0118\61\3\2\2\2\u0119\u011e\5\66\34\2\u011a\u011b\7\7\2\2\u011b"+
+ "\u011d\5\66\34\2\u011c\u011a\3\2\2\2\u011d\u0120\3\2\2\2\u011e\u011c\3"+
+ "\2\2\2\u011e\u011f\3\2\2\2\u011f\63\3\2\2\2\u0120\u011e\3\2\2\2\u0121"+
+ "\u0122\5\66\34\2\u0122\65\3\2\2\2\u0123\u0124\7+\2\2\u0124\67\3\2\2\2"+
+ "\30EZeis\u0080\u0088\u0095\u009a\u00a2\u00a4\u00ad\u00bc\u00c5\u00d0\u00d2"+
+ "\u00de\u00e0\u00f4\u0106\u0117\u011e";
public static final ATN _ATN =
new ATNDeserializer().deserialize(_serializedATN.toCharArray());
static {
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
index bd70a33..81a9adc 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/dsl/functions/FunctionalFunctionsTest.java
@@ -71,6 +71,8 @@ public class FunctionalFunctionsTest {
, "MAP([ foo, 'bar'], (x) -> TO_UPPER(x) )"
, "MAP([ foo, bar], (x) -> TO_UPPER(x) )"
, "MAP([ foo, bar], x -> TO_UPPER(x) )"
+ , "MAP([ foo, bar], x -> true?TO_UPPER(x):THROW('error') )"
+ , "MAP([ foo, bar], x -> false?THROW('error'):TO_UPPER(x) )"
)
)
{
@@ -123,6 +125,21 @@ public class FunctionalFunctionsTest {
@Test
+ public void testFilter_shortcircuit() {
+ for (String expr : ImmutableList.of("FILTER([ 'foo'], item -> item == 'foo' or THROW('exception') )"
+ ,"FILTER([ 'foo'], (item) -> item == 'foo' or THROW('exception') )"
+ )
+ )
+ {
+ Object o = run(expr, ImmutableMap.of("foo", "foo", "bar", "bar"));
+ Assert.assertTrue(o instanceof List);
+ List<String> result = (List<String>) o;
+ Assert.assertEquals(1, result.size());
+ Assert.assertEquals("foo", result.get(0));
+ }
+ }
+
+ @Test
public void testFilter_null() {
for (String expr : ImmutableList.of("FILTER([ 'foo', null], item -> item == null )"
,"FILTER([ 'foo', baz], (item) -> item == null )"
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarInterpreterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarInterpreterTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarInterpreterTest.java
index b25f0a7..39563f7 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarInterpreterTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarInterpreterTest.java
@@ -77,9 +77,9 @@ public class StellarInterpreterTest {
StellarParser.IntLiteralContext ctx = mock(StellarParser.IntLiteralContext.class);
Token result = mock(Token.class);
when(ctx.getText()).thenReturn("1000");
- when(numberLiteralEvaluator.evaluate(ctx)).thenReturn(result);
+ when(numberLiteralEvaluator.evaluate(ctx, null)).thenReturn(result);
compiler.exitIntLiteral(ctx);
- verify(numberLiteralEvaluator).evaluate(ctx);
+ verify(numberLiteralEvaluator).evaluate(ctx, null);
Assert.assertEquals(1, tokenStack.size());
Assert.assertEquals(tokenStack.getFirst(), result);
verifyZeroInteractions(variableResolver);
@@ -93,12 +93,12 @@ public class StellarInterpreterTest {
public void exitDoubleLiteralShouldProperlyParseStringsAsDoubles() throws Exception {
StellarParser.DoubleLiteralContext ctx = mock(StellarParser.DoubleLiteralContext.class);
Token result = mock(Token.class);
- when(numberLiteralEvaluator.evaluate(ctx)).thenReturn(result);
+ when(numberLiteralEvaluator.evaluate(ctx, null)).thenReturn(result);
when(ctx.getText()).thenReturn("1000D");
compiler.exitDoubleLiteral(ctx);
- verify(numberLiteralEvaluator).evaluate(ctx);
+ verify(numberLiteralEvaluator).evaluate(ctx, null);
Assert.assertEquals(1, tokenStack.size());
Assert.assertEquals(tokenStack.getFirst(), result);
verifyZeroInteractions(variableResolver);
@@ -113,11 +113,11 @@ public class StellarInterpreterTest {
StellarParser.FloatLiteralContext ctx = mock(StellarParser.FloatLiteralContext.class);
when(ctx.getText()).thenReturn("1000f");
Token result = mock(Token.class);
- when(numberLiteralEvaluator.evaluate(ctx)).thenReturn(result);
+ when(numberLiteralEvaluator.evaluate(ctx, null)).thenReturn(result);
compiler.exitFloatLiteral(ctx);
- verify(numberLiteralEvaluator).evaluate(ctx);
+ verify(numberLiteralEvaluator).evaluate(ctx, null);
Assert.assertEquals(1, tokenStack.size());
Assert.assertEquals(tokenStack.getFirst(), result);
verifyZeroInteractions(variableResolver);
@@ -132,11 +132,11 @@ public class StellarInterpreterTest {
StellarParser.LongLiteralContext ctx = mock(StellarParser.LongLiteralContext.class);
when(ctx.getText()).thenReturn("1000l");
Token result = mock(Token.class);
- when(numberLiteralEvaluator.evaluate(ctx)).thenReturn(result);
+ when(numberLiteralEvaluator.evaluate(ctx, null)).thenReturn(result);
compiler.exitLongLiteral(ctx);
- verify(numberLiteralEvaluator).evaluate(ctx);
+ verify(numberLiteralEvaluator).evaluate(ctx, null);
Assert.assertEquals(1, tokenStack.size());
Assert.assertEquals(tokenStack.getFirst(), result);
verifyZeroInteractions(variableResolver);
@@ -152,17 +152,17 @@ public class StellarInterpreterTest {
StellarParser.ComparisonOpContext mockOp = mock(StellarParser.ComparisonOpContext.class);
when(ctx.comp_operator()).thenReturn(mockOp);
Token result = mock(Token.class);
- when(comparisonExpressionWithOperatorEvaluator.evaluate(any(Token.class), any(Token.class), any(StellarParser.ComparisonOpContext.class))).thenReturn(result);
+ when(comparisonExpressionWithOperatorEvaluator.evaluate(any(Token.class), any(Token.class), any(StellarParser.ComparisonOpContext.class), any())).thenReturn(result);
compiler.exitComparisonExpressionWithOperator(ctx);
Assert.assertEquals(1, tokenStack.size());
StellarCompiler.DeferredFunction func = (StellarCompiler.DeferredFunction) tokenStack.pop().getValue();
- tokenStack.push(new Token<>(1000, Integer.class));
- tokenStack.push(new Token<>(1500f, Float.class));
+ tokenStack.push(new Token<>(1000, Integer.class, null));
+ tokenStack.push(new Token<>(1500f, Float.class, null));
func.apply(tokenStack, new StellarCompiler.ExpressionState(context, functionResolver, variableResolver));
Assert.assertEquals(1, tokenStack.size());
Assert.assertEquals(tokenStack.getFirst(), result);
- verify(comparisonExpressionWithOperatorEvaluator).evaluate(any(Token.class), any(Token.class), eq(mockOp));
+ verify(comparisonExpressionWithOperatorEvaluator).evaluate(any(Token.class), any(Token.class), eq(mockOp), any());
verifyZeroInteractions(numberLiteralEvaluator);
verifyZeroInteractions(variableResolver);
verifyZeroInteractions(functionResolver);
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
index d7a33d4..98abd57 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
@@ -18,13 +18,11 @@
package org.apache.metron.common.stellar;
+import com.google.common.base.Joiner;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import org.apache.commons.lang3.StringUtils;
-import org.apache.metron.common.dsl.ParseException;
-import org.apache.metron.common.dsl.Stellar;
-import org.apache.metron.common.dsl.StellarFunction;
-import org.apache.metron.common.dsl.StellarFunctions;
+import org.apache.metron.common.dsl.*;
import org.apache.metron.common.dsl.functions.resolver.ClasspathFunctionResolver;
import org.junit.Assert;
import org.junit.Rule;
@@ -33,12 +31,7 @@ import org.junit.rules.ExpectedException;
import org.reflections.Reflections;
import org.reflections.util.ConfigurationBuilder;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.Map;
+import java.util.*;
import static org.apache.metron.common.utils.StellarProcessorUtils.run;
import static org.apache.metron.common.utils.StellarProcessorUtils.runPredicate;
@@ -46,6 +39,58 @@ import static org.apache.metron.common.utils.StellarProcessorUtils.runPredicate;
@SuppressWarnings("ALL")
public class StellarTest {
+ @Stellar(
+ description="throw exception",
+ name="THROW",
+ params = {
+ "message - exception message"
+ },
+ returns="nothing"
+ )
+ public static class Throw implements StellarFunction {
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ throw new IllegalStateException(Joiner.on(" ").join(args));
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
+ @Stellar(
+ description="Always returns true",
+ name="RET_TRUE",
+ params = {
+ "arg* - Any set of args you wish to give it (including the empty set), they're ignored."
+ },
+ returns="true"
+ )
+ public static class TrueFunc implements StellarFunction {
+
+ @Override
+ public Object apply(List<Object> args, Context context) throws ParseException {
+ return true;
+ }
+
+ @Override
+ public void initialize(Context context) {
+
+ }
+
+ @Override
+ public boolean isInitialized() {
+ return true;
+ }
+ }
+
@Test
public void ensureDocumentation() {
ClassLoader classLoader = getClass().getClassLoader();
@@ -634,6 +679,42 @@ public class StellarTest {
Assert.assertTrue(runPredicate("not(IN_SUBNET(ip_dst_addr, '192.168.0.0/24'))", v-> variableMap.get(v)));
}
+ @Test
+ public void testShortCircuit_conditional() throws Exception {
+ Assert.assertEquals("foo", run("if true then 'foo' else (if false then 'bar' else 'grok')", new HashMap<>()));
+ Assert.assertEquals("foo", run("if true_var != null && true_var then 'foo' else (if false then 'bar' else 'grok')", ImmutableMap.of("true_var", true)));
+ Assert.assertEquals("foo", run("if true then 'foo' else THROW('expression')", new HashMap<>()));
+ Assert.assertEquals("foo", run("true ? 'foo' : THROW('expression')", new HashMap<>()));
+ Assert.assertEquals("foo", run("if false then THROW('exception') else 'foo'", new HashMap<>()));
+ Assert.assertEquals("foo", run("false ? THROW('exception') : 'foo'", new HashMap<>()));
+ Assert.assertEquals(true, run("RET_TRUE(if true then 'foo' else THROW('expression'))", new HashMap<>()));
+ Assert.assertEquals("foo", run("if true or (true or THROW('if exception')) then 'foo' else THROW('expression')", new HashMap<>()));
+ Assert.assertEquals("foo", run("if true or (false or THROW('if exception')) then 'foo' else THROW('expression')", new HashMap<>()));
+ Assert.assertEquals("foo", run("if NOT(true or (false or THROW('if exception'))) then THROW('expression') else 'foo'", new HashMap<>()));
+ Assert.assertEquals("foo", run("if NOT('metron' in [ 'metron', 'metronicus'] ) then THROW('expression') else 'foo'", new HashMap<>()));
+ }
+
+ @Test
+ public void testShortCircuit_boolean() throws Exception {
+ Assert.assertTrue(runPredicate("'metron' in ['metron', 'metronicus', 'mortron'] or (true or THROW('exception'))", x -> null));
+ Assert.assertTrue(runPredicate("true or (true or THROW('exception'))", x -> null));
+ Assert.assertTrue(runPredicate("true or (false or THROW('exception'))", x -> null));
+ Assert.assertTrue(runPredicate("TO_UPPER('foo') == 'FOO' or (true or THROW('exception'))", x -> null));
+ Assert.assertFalse(runPredicate("false and (true or THROW('exception'))", x -> null));
+ Assert.assertTrue(runPredicate("true or false or false or true", x -> null));
+ Assert.assertFalse(runPredicate("false or (false and THROW('exception'))", x -> null));
+ Assert.assertTrue(runPredicate("'casey' == 'casey' or THROW('exception')", x -> null));
+ Assert.assertTrue(runPredicate("TO_UPPER('casey') == 'CASEY' or THROW('exception')", x -> null));
+ Assert.assertTrue(runPredicate("NOT(TO_UPPER('casey') != 'CASEY') or THROW('exception')", x -> null));
+ Assert.assertTrue(runPredicate("(TO_UPPER('casey') == 'CASEY') or THROW('exception')", x -> null));
+ Assert.assertFalse(runPredicate("NOT(NOT(TO_UPPER('casey') != 'CASEY') or THROW('exception'))", x -> null));
+ Assert.assertFalse(runPredicate("NOT(NOT(TO_UPPER('casey') != 'CASEY')) and THROW('exception')", x -> null));
+ Assert.assertTrue(runPredicate("RET_TRUE('foo') or THROW('exception')", x -> null));
+ Assert.assertFalse(runPredicate("NOT(foo == null or THROW('exception')) and THROW('and exception')", x -> null));
+ Assert.assertTrue(runPredicate("(foo == null or THROW('exception') ) or THROW('and exception')", x -> null));
+ Assert.assertTrue(runPredicate("( RET_TRUE('foo', true, false) or ( foo == null or THROW('exception') ) or THROW('and exception')) or THROW('or exception')", x -> null));
+ }
+
@Rule
public ExpectedException thrown = ExpectedException.none();
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluatorTest.java
index f914c3c..de9f722 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ArithmeticEvaluatorTest.java
@@ -44,7 +44,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), p);
assertTrue(evaluated.getValue() instanceof Double);
assertEquals(3.0D, evaluated.getValue());
@@ -60,7 +60,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), p);
assertTrue(evaluated.getValue() instanceof Integer);
assertEquals(3, evaluated.getValue());
@@ -76,7 +76,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), p);
assertTrue(evaluated.getValue() instanceof Float);
assertEquals(3F, evaluated.getValue());
@@ -92,7 +92,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), p);
assertTrue(evaluated.getValue() instanceof Long);
assertEquals(3L, evaluated.getValue());
@@ -108,7 +108,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(null), p);
assertTrue(evaluated.getValue() instanceof Double);
assertEquals(2.0D, evaluated.getValue());
@@ -124,7 +124,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(null), p);
assertTrue(evaluated.getValue() instanceof Integer);
assertEquals(2, evaluated.getValue());
@@ -140,7 +140,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(null), p);
assertTrue(evaluated.getValue() instanceof Float);
assertEquals(2F, evaluated.getValue());
@@ -156,7 +156,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(null), p);
assertTrue(evaluated.getValue() instanceof Long);
assertEquals(2L, evaluated.getValue());
@@ -172,7 +172,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(null), p);
assertTrue(evaluated.getValue() instanceof Double);
assertEquals(-1.0D, evaluated.getValue());
@@ -188,7 +188,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(null), p);
assertTrue(evaluated.getValue() instanceof Integer);
assertEquals(-1, evaluated.getValue());
@@ -204,7 +204,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(null), p);
assertTrue(evaluated.getValue() instanceof Float);
assertEquals(-1F, evaluated.getValue());
@@ -220,7 +220,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(null), p);
assertTrue(evaluated.getValue() instanceof Long);
assertEquals(-1L, evaluated.getValue());
@@ -236,7 +236,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), p);
assertTrue(evaluated.getValue() instanceof Double);
assertEquals(1 / 2D, evaluated.getValue());
@@ -252,7 +252,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), p);
assertTrue(evaluated.getValue() instanceof Integer);
assertEquals(1 / 2, evaluated.getValue());
@@ -268,7 +268,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), p);
assertTrue(evaluated.getValue() instanceof Float);
assertEquals(0.5F, evaluated.getValue());
@@ -284,7 +284,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), p);
assertTrue(evaluated.getValue() instanceof Long);
assertEquals(0L, evaluated.getValue());
@@ -292,19 +292,19 @@ public class ArithmeticEvaluatorTest {
@Test(expected = IllegalArgumentException.class)
public void evaluateShouldThroughIllegalArgumentExceptionWhenInputIsNull() throws Exception {
- evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), null);
+ evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), null);
}
@Test(expected = IllegalArgumentException.class)
public void evaluateShouldThroughIllegalArgumentExceptionWhenInputsKeyIsNull() throws Exception {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(null, mock(Token.class));
- evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p);
+ evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), p);
}
@Test(expected = IllegalArgumentException.class)
public void evaluateShouldThroughIllegalArgumentExceptionWhenInputsValueIsNull() throws Exception {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(mock(Token.class), null);
- evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), p);
+ evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), p);
}
@Test
@@ -315,10 +315,10 @@ public class ArithmeticEvaluatorTest {
Token<Short> r = mock(Token.class);
when(r.getValue()).thenReturn((short) 3);
- Token<? extends Number> evaluated0 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), Pair.of(l, r));
- Token<? extends Number> evaluated1 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), Pair.of(l, r));
- Token<? extends Number> evaluated2 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), Pair.of(l, r));
- Token<? extends Number> evaluated3 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), Pair.of(l, r));
+ Token<? extends Number> evaluated0 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), Pair.of(l, r));
+ Token<? extends Number> evaluated1 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(null), Pair.of(l, r));
+ Token<? extends Number> evaluated2 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(null), Pair.of(l, r));
+ Token<? extends Number> evaluated3 = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), Pair.of(l, r));
assertTrue(evaluated0.getValue() instanceof Integer);
assertEquals(5, evaluated0.getValue());
@@ -343,7 +343,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), p);
assertTrue(evaluated.getValue() instanceof Integer);
assertEquals(2, evaluated.getValue());
@@ -359,7 +359,7 @@ public class ArithmeticEvaluatorTest {
Pair<Token<? extends Number>, Token<? extends Number>> p = Pair.of(l, r);
- Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), p);
+ Token<? extends Number> evaluated = evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), p);
assertTrue(evaluated.getValue() instanceof Integer);
assertEquals(1, evaluated.getValue());
@@ -403,10 +403,10 @@ public class ArithmeticEvaluatorTest {
}};
expectedReturnTypeMappings.forEach( (pair, expectedClass) -> {
- assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(), pair).getValue().getClass() == expectedClass);
- assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(), pair).getValue().getClass() == expectedClass);
- assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(), pair).getValue().getClass() == expectedClass);
- assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(), pair).getValue().getClass() == expectedClass);
+ assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.addition(null), pair).getValue().getClass() == expectedClass);
+ assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.division(null), pair).getValue().getClass() == expectedClass);
+ assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.subtraction(null), pair).getValue().getClass() == expectedClass);
+ assertTrue(evaluator.evaluate(ArithmeticEvaluator.ArithmeticEvaluatorFunctions.multiplication(null), pair).getValue().getClass() == expectedClass);
});
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluatorTest.java
index 612d311..0c20eb4 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/ComparisonExpressionWithOperatorEvaluatorTest.java
@@ -54,7 +54,7 @@ public class ComparisonExpressionWithOperatorEvaluatorTest {
StellarParser.ComparisonOpContext op = mock(StellarParser.ComparisonOpContext.class);
when(op.EQ()).thenReturn(mock(TerminalNode.class));
- Token<Boolean> evaluated = evaluator.evaluate(left, right, op);
+ Token<Boolean> evaluated = evaluator.evaluate(left, right, op, null);
assertTrue(evaluated.getValue());
}
@@ -70,7 +70,7 @@ public class ComparisonExpressionWithOperatorEvaluatorTest {
StellarParser.ComparisonOpContext op = mock(StellarParser.ComparisonOpContext.class);
when(op.NEQ()).thenReturn(mock(TerminalNode.class));
- Token<Boolean> evaluated = evaluator.evaluate(left, right, op);
+ Token<Boolean> evaluated = evaluator.evaluate(left, right, op, null);
assertFalse(evaluated.getValue());
}
@@ -86,7 +86,7 @@ public class ComparisonExpressionWithOperatorEvaluatorTest {
StellarParser.ComparisonOpContext op = mock(StellarParser.ComparisonOpContext.class);
when(op.LTE()).thenReturn(mock(TerminalNode.class));
- Token<Boolean> evaluated = evaluator.evaluate(left, right, op);
+ Token<Boolean> evaluated = evaluator.evaluate(left, right, op, null);
assertTrue(evaluated.getValue());
}
@@ -104,7 +104,7 @@ public class ComparisonExpressionWithOperatorEvaluatorTest {
StellarParser.ComparisonOpContext op = mock(StellarParser.ComparisonOpContext.class);
- evaluator.evaluate(left, right, op);
+ evaluator.evaluate(left, right, op, null);
}
@Test
@@ -121,6 +121,6 @@ public class ComparisonExpressionWithOperatorEvaluatorTest {
StellarParser.ComparisonOpContext op = mock(StellarParser.ComparisonOpContext.class);
when(op.LTE()).thenReturn(mock(TerminalNode.class));
- evaluator.evaluate(left, right, op);
+ evaluator.evaluate(left, right, op, null);
}
}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluatorTest.java
index 6f2fd40..3386f3f 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/DoubleLiteralEvaluatorTest.java
@@ -45,8 +45,8 @@ public class DoubleLiteralEvaluatorTest {
public void verifyHappyPathEvaluation() throws Exception {
when(context.getText()).thenReturn("100D");
- Token<? extends Number> evaluated = evaluator.evaluate(context);
- assertEquals(new Token<>(100D, Double.class), evaluated);
+ Token<? extends Number> evaluated = evaluator.evaluate(context, null);
+ assertEquals(new Token<>(100D, Double.class, null), evaluated);
verify(context).getText();
verifyNoMoreInteractions(context);
@@ -57,7 +57,7 @@ public class DoubleLiteralEvaluatorTest {
exception.expect(NumberFormatException.class);
when(context.getText()).thenReturn("");
- evaluator.evaluate(context);
+ evaluator.evaluate(context, null);
}
@Test
@@ -65,6 +65,6 @@ public class DoubleLiteralEvaluatorTest {
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot evaluate a context that is null.");
- evaluator.evaluate(null);
+ evaluator.evaluate(null, null);
}
}
[11/18] metron git commit: METRON-976 KafkaUtils doesn't handle
SASL_PLAINTEXT (justinleet via leet) closes apache/metron#600
Posted by ma...@apache.org.
METRON-976 KafkaUtils doesn't handle SASL_PLAINTEXT (justinleet via leet) closes apache/metron#600
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/61cbab46
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/61cbab46
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/61cbab46
Branch: refs/heads/Metron_0.4.0
Commit: 61cbab46e3e262e006e59c28eaa9192d0f7aaaab
Parents: 5b72da7
Author: justinleet <ju...@gmail.com>
Authored: Thu Jun 8 08:47:27 2017 -0400
Committer: YOUR NAME as In Apache <YO...@apache.org>
Committed: Thu Jun 8 08:47:27 2017 -0400
----------------------------------------------------------------------
.../apache/metron/common/utils/KafkaUtils.java | 15 +-
.../common/utils/KafkaUtilsEndpointTest.java | 64 --------
.../metron/common/utils/KafkaUtilsTest.java | 162 +++++++++++++++++++
3 files changed, 173 insertions(+), 68 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/61cbab46/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/KafkaUtils.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/KafkaUtils.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/KafkaUtils.java
index 04c1389..bbd8b30 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/KafkaUtils.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/KafkaUtils.java
@@ -44,6 +44,7 @@ public enum KafkaUtils {
framework.close();
}
}
+
public List<String> getBrokersFromZookeeper(CuratorFramework client) throws Exception {
List<String> ret = new ArrayList<>();
for(String id : client.getChildren().forPath("/brokers/ids")) {
@@ -68,12 +69,18 @@ public enum KafkaUtils {
return ret;
}
- public List<String> fromEndpoint(String url) throws URISyntaxException {
+ /*
+ The URL accepted is NOT a general URL, and is assumed to follow the format used by the Kafka structures in Zookeeper.
+ See: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
+ */
+ List<String> fromEndpoint(String url){
List<String> ret = new ArrayList<>();
if(url != null) {
- URI uri = new URI(url);
- int port = uri.getPort();
- ret.add(uri.getHost() + ((port > 0)?(":" + port):""));
+ Iterable<String> splits = Splitter.on("//").split(url);
+ if(Iterables.size(splits) == 2) {
+ String hostPort = Iterables.getLast(splits);
+ ret.add(hostPort);
+ }
}
return ret;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/61cbab46/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsEndpointTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsEndpointTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsEndpointTest.java
deleted file mode 100644
index 14a9e41..0000000
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsEndpointTest.java
+++ /dev/null
@@ -1,64 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.metron.common.utils;
-
-import org.junit.Assert;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.Parameterized;
-
-import java.net.URISyntaxException;
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.List;
-
-@RunWith(Parameterized.class)
-public class KafkaUtilsEndpointTest {
- static String[] hostnames = new String[] { "node1", "localhost", "192.168.0.1", "my.domain.com" };
- static String[] schemes = new String[] { "SSL", "PLAINTEXTSASL", "PLAINTEXT"};
- static String[] ports = new String[] { "6667", "9091", null};
- private String endpoint;
- private String expected;
-
- public KafkaUtilsEndpointTest(String endpoint, String expected) {
- this.endpoint = endpoint;
- this.expected = expected;
- }
-
- @Parameterized.Parameters
- public static Collection<Object[]> data() {
- List<Object[]> ret = new ArrayList<>();
- for(String scheme : schemes) {
- for(String hostname : hostnames) {
- for(String port : ports) {
- port = port != null?(":" + port):"";
- String expected = hostname + port;
- ret.add(new Object[]{scheme + "://" + expected, expected });
- }
- }
- }
- return ret;
- }
-
- @Test
- public void testEndpointParsing() throws URISyntaxException {
- Assert.assertEquals(expected, KafkaUtils.INSTANCE.fromEndpoint(endpoint).get(0));
- }
-
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/61cbab46/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsTest.java
new file mode 100644
index 0000000..72ac51e
--- /dev/null
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/utils/KafkaUtilsTest.java
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.utils;
+
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.when;
+
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.api.GetChildrenBuilder;
+import org.apache.curator.framework.api.GetDataBuilder;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+import org.mockito.Mock;
+import org.mockito.runners.MockitoJUnitRunner;
+
+@RunWith(Enclosed.class)
+public class KafkaUtilsTest {
+ @RunWith(MockitoJUnitRunner.class)
+ public static class ZkMockedUtils {
+ @Mock
+ CuratorFramework client;
+ @Mock
+ GetChildrenBuilder childrenBuilder;
+ @Mock
+ GetDataBuilder dataBuilder;
+
+ /**
+ * {
+ * "host": "192.168.1.148",
+ * "port": 9092
+ * }
+ */
+ @Multiline
+ public static String brokerWithHostPort;
+
+ @Test
+ public void testGetEndpointsFromZookeeperHostPort() throws Exception {
+ ArrayList<String> brokerIds = new ArrayList<>();
+ brokerIds.add("1");
+
+ when(client.getChildren()).thenReturn(childrenBuilder);
+ when(childrenBuilder.forPath("/brokers/ids")).thenReturn(brokerIds);
+ when(client.getData()).thenReturn(dataBuilder);
+ when(dataBuilder.forPath("/brokers/ids/1")).thenReturn(brokerWithHostPort.getBytes());
+
+ ArrayList<String> expected = new ArrayList<>();
+ expected.add("192.168.1.148:9092");
+ assertEquals(expected, (KafkaUtils.INSTANCE.getBrokersFromZookeeper(client)));
+ }
+
+ /**
+ * {
+ * "endpoints": ["PLAINTEXT://host1:9092", "SSL://host1:9093", "SASL_PLAINTEXT://host1:9094", "PLAINTEXTSASL://host1:9095"]
+ * }
+ */
+ @Multiline
+ public static String brokerWithEndpoints;
+
+ @Test
+ public void testGetEndpointsFromZookeeperEndpoints() throws Exception {
+ ArrayList<String> brokerIds = new ArrayList<>();
+ brokerIds.add("1");
+
+ when(client.getChildren()).thenReturn(childrenBuilder);
+ when(childrenBuilder.forPath("/brokers/ids")).thenReturn(brokerIds);
+ when(client.getData()).thenReturn(dataBuilder);
+ when(dataBuilder.forPath("/brokers/ids/1")).thenReturn(brokerWithEndpoints.getBytes());
+
+ ArrayList<String> expected = new ArrayList<>();
+ expected.add("host1:9092");
+ expected.add("host1:9093");
+ expected.add("host1:9094");
+ expected.add("host1:9095");
+ assertEquals(expected, (KafkaUtils.INSTANCE.getBrokersFromZookeeper(client)));
+ }
+
+ /**
+ * {
+ * "host": "192.168.1.148",
+ * "port": 9092,
+ * "endpoints": ["PLAINTEXT://host1:9092", "SSL://host1:9093"]
+ * }
+ */
+ @Multiline
+ public static String brokerWithHostPortAndEndpoints;
+
+ @Test
+ public void testGetEndpointsFromZookeeperHostPortAndEndpoints() throws Exception {
+ ArrayList<String> brokerIds = new ArrayList<>();
+ brokerIds.add("1");
+
+ when(client.getChildren()).thenReturn(childrenBuilder);
+ when(childrenBuilder.forPath("/brokers/ids")).thenReturn(brokerIds);
+ when(client.getData()).thenReturn(dataBuilder);
+ when(dataBuilder.forPath("/brokers/ids/1"))
+ .thenReturn(brokerWithHostPortAndEndpoints.getBytes());
+
+ ArrayList<String> expected = new ArrayList<>();
+ expected.add("192.168.1.148:9092");
+ assertEquals(expected, (KafkaUtils.INSTANCE.getBrokersFromZookeeper(client)));
+ }
+ }
+
+ @RunWith(Parameterized.class)
+ public static class ParameterizedEndPointParsing {
+ static String[] hostnames = new String[]{"node1", "localhost", "192.168.0.1", "my.domain.com"};
+ static String[] schemes = new String[]{"SSL", "PLAINTEXTSASL", "PLAINTEXT", "SASL_PLAINTEXT"};
+ static String[] ports = new String[]{"6667", "9091", null};
+
+ private String endpoint;
+ private String expected;
+
+ public ParameterizedEndPointParsing(String endpoint, String expected) {
+ this.endpoint = endpoint;
+ this.expected = expected;
+ }
+
+ @Parameters(name = "{index}:endpoint({0}={1})")
+ public static Collection<Object[]> data() {
+ List<Object[]> ret = new ArrayList<>();
+ for (String scheme : schemes) {
+ for (String hostname : hostnames) {
+ for (String port : ports) {
+ port = port != null ? (":" + port) : "";
+ String expected = hostname + port;
+ ret.add(new Object[]{scheme + "://" + expected, expected});
+ }
+ }
+ }
+ return ret;
+ }
+
+ @Test
+ public void testEndpointParsing() throws URISyntaxException {
+ assertEquals(expected, KafkaUtils.INSTANCE.fromEndpoint(endpoint).get(0));
+ }
+ }
+}
[13/18] metron git commit: METRON-980: Short circuit operations for
Stellar closes apache/metron#606
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluatorTest.java
index 4f7117a..f9ecff1 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/FloatLiteralEvaluatorTest.java
@@ -48,8 +48,8 @@ public class FloatLiteralEvaluatorTest {
public void verifyHappyPathEvaluation() throws Exception {
when(context.getText()).thenReturn("100f");
- Token<? extends Number> evaluated = evaluator.evaluate(context);
- assertEquals(new Token<>(100f, Float.class), evaluated);
+ Token<? extends Number> evaluated = evaluator.evaluate(context, null);
+ assertEquals(new Token<>(100f, Float.class, null), evaluated);
verify(context).getText();
verifyNoMoreInteractions(context);
@@ -60,7 +60,7 @@ public class FloatLiteralEvaluatorTest {
exception.expect(NumberFormatException.class);
when(context.getText()).thenReturn("");
- evaluator.evaluate(context);
+ evaluator.evaluate(context, null);
}
@Test
@@ -68,7 +68,7 @@ public class FloatLiteralEvaluatorTest {
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot evaluate a context that is null.");
- evaluator.evaluate(null);
+ evaluator.evaluate(null, null);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluatorTest.java
index 2fd081f..bef23d1 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/IntLiteralEvaluatorTest.java
@@ -48,8 +48,8 @@ public class IntLiteralEvaluatorTest {
public void verifyHappyPathEvaluation() throws Exception {
when(context.getText()).thenReturn("100");
- Token<? extends Number> evaluated = evaluator.evaluate(context);
- assertEquals(new Token<>(100, Integer.class), evaluated);
+ Token<? extends Number> evaluated = evaluator.evaluate(context, null);
+ assertEquals(new Token<>(100, Integer.class, null), evaluated);
verify(context).getText();
verifyNoMoreInteractions(context);
@@ -60,7 +60,7 @@ public class IntLiteralEvaluatorTest {
exception.expect(NumberFormatException.class);
when(context.getText()).thenReturn("");
- evaluator.evaluate(context);
+ evaluator.evaluate(context, null);
}
@Test
@@ -68,7 +68,7 @@ public class IntLiteralEvaluatorTest {
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot evaluate a context that is null.");
- evaluator.evaluate(null);
+ evaluator.evaluate(null, null);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluatorTest.java
index 21f75d4..e9c3aa7 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/LongLiteralEvaluatorTest.java
@@ -49,8 +49,8 @@ public class LongLiteralEvaluatorTest {
public void verifyHappyPathEvaluation() throws Exception {
when(context.getText()).thenReturn("100L");
- Token<? extends Number> evaluated = evaluator.evaluate(context);
- assertEquals(new Token<>(100L, Long.class), evaluated);
+ Token<? extends Number> evaluated = evaluator.evaluate(context, null);
+ assertEquals(new Token<>(100L, Long.class, null), evaluated);
verify(context).getText();
verifyNoMoreInteractions(context);
@@ -62,7 +62,7 @@ public class LongLiteralEvaluatorTest {
exception.expectMessage("Invalid format for long. Failed trying to parse a long with the following value: ");
when(context.getText()).thenReturn("");
- evaluator.evaluate(context);
+ evaluator.evaluate(context, null);
}
@Test
@@ -70,6 +70,6 @@ public class LongLiteralEvaluatorTest {
exception.expect(IllegalArgumentException.class);
exception.expectMessage("Cannot evaluate a context that is null.");
- evaluator.evaluate(null);
+ evaluator.evaluate(null, null);
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/30d0e2a6/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluatorTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluatorTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluatorTest.java
index 8317737..2065e6d 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluatorTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/evaluators/NumberLiteralEvaluatorTest.java
@@ -60,36 +60,36 @@ public class NumberLiteralEvaluatorTest {
@Test
public void verifyIntLiteralContextIsProperlyEvaluated() throws Exception {
StellarParser.IntLiteralContext context = mock(StellarParser.IntLiteralContext.class);
- NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap);
+ NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap, null);
- verify(intLiteralContextNumberEvaluator).evaluate(context);
+ verify(intLiteralContextNumberEvaluator).evaluate(context, null);
verifyZeroInteractions(doubleLiteralContextNumberEvaluator, floatLiteralContextNumberEvaluator, longLiteralContextNumberEvaluator);
}
@Test
public void verifyDoubleLiteralContextIsProperlyEvaluated() throws Exception {
StellarParser.DoubleLiteralContext context = mock(StellarParser.DoubleLiteralContext.class);
- NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap);
+ NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap, null);
- verify(doubleLiteralContextNumberEvaluator).evaluate(context);
+ verify(doubleLiteralContextNumberEvaluator).evaluate(context, null);
verifyZeroInteractions(intLiteralContextNumberEvaluator, floatLiteralContextNumberEvaluator, longLiteralContextNumberEvaluator);
}
@Test
public void verifyFloatLiteralContextIsProperlyEvaluated() throws Exception {
StellarParser.FloatLiteralContext context = mock(StellarParser.FloatLiteralContext.class);
- NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap);
+ NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap, null);
- verify(floatLiteralContextNumberEvaluator).evaluate(context);
+ verify(floatLiteralContextNumberEvaluator).evaluate(context, null);
verifyZeroInteractions(doubleLiteralContextNumberEvaluator, intLiteralContextNumberEvaluator, longLiteralContextNumberEvaluator);
}
@Test
public void verifyLongLiteralContextIsProperlyEvaluated() throws Exception {
StellarParser.LongLiteralContext context = mock(StellarParser.LongLiteralContext.class);
- NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap);
+ NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap, null);
- verify(longLiteralContextNumberEvaluator).evaluate(context);
+ verify(longLiteralContextNumberEvaluator).evaluate(context, null);
verifyZeroInteractions(doubleLiteralContextNumberEvaluator, floatLiteralContextNumberEvaluator, intLiteralContextNumberEvaluator);
}
@@ -100,7 +100,7 @@ public class NumberLiteralEvaluatorTest {
exception.expect(ParseException.class);
exception.expectMessage("Does not support evaluation for type " + context.getClass());
- NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap);
+ NumberLiteralEvaluator.INSTANCE.evaluate(context, instanceMap, null);
verifyZeroInteractions(longLiteralContextNumberEvaluator, doubleLiteralContextNumberEvaluator, floatLiteralContextNumberEvaluator, intLiteralContextNumberEvaluator);
}
[08/18] metron git commit: METRON-962 Configuration Based Unit Tests
and Add integration tests (justinleet via leet) closes apache/metron#612
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/resources/logData/IseParserTest.txt
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/resources/logData/IseParserTest.txt b/metron-platform/metron-parsers/src/test/resources/logData/IseParserTest.txt
new file mode 100644
index 0000000..54cf4ff
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/resources/logData/IseParserTest.txt
@@ -0,0 +1,308 @@
+Aug 6 17:26:31 10.34.84.145 Aug 7 00:45:43 stage-pdp01 CISE_Profiler 0000024855 1 0 2014-08-07 00:45:43.741 -07:00 0000288542 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=113, EndpointCertainityMetric=10, EndpointIPAddress=10.56.111.14, EndpointMacAddress=3C:97:0E:C3:F8:F1, EndpointMatchedPolicy=Nortel-Device, EndpointNADAddress=10.56.72.127, EndpointOUI=Wistron InfoComm(Kunshan)Co.\,Ltd., EndpointPolicy=Nortel-Device, EndpointProperty=StaticAssignment=false\,PostureApplicable=Yes\,PolicyVersion=402\,IdentityGroupID=0c1d9270-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=10\,BYODRegistration=Unknown\,FeedService=false\,EndPointPolicyID=49054ed0-68a6-11e1-bc72-0050568e013c\,FirstCollection=1407397543718\,MatchedPolicyID=49054ed0-68a6-11e1-bc72-0050568e013c\,TimeToProfile=19\,StaticGroupAssignment=false\,NmapSubnetScanID=0\,DeviceRegistrationStatus=NotRegistered\,PortalUser=, EndpointSourceEvent=SNMPQuery Probe, EndpointIdentityGroup=Profile
d, ProfilerServer=stage-pdp01.cisco.com,
+Aug 6 17:26:31 10.34.84.145 Aug 7 00:45:43 stage-pdp01 CISE_Profiler 0000024856 1 0 2014-08-07 00:45:43.786 -07:00 0000288543 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=113, EndpointCertainityMetric=10, EndpointIPAddress=10.56.111.14, EndpointMacAddress=3C:97:0E:C3:F8:F1, EndpointMatchedPolicy=Nortel-Device, EndpointNADAddress=10.56.72.127, EndpointOUI=Wistron InfoComm(Kunshan)Co.\,Ltd., EndpointPolicy=Nortel-Device, EndpointProperty=StaticAssignment=false\,PostureApplicable=Yes\,BYODRegistration=Unknown\,EndPointPolicyID=49054ed0-68a6-11e1-bc72-0050568e013c\,FirstCollection=1407397543718\,TimeToProfile=19\,LastNmapScanTime=0\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,UpdateTime=0\,PolicyVersion=402\,IdentityGroupID=0c1d9270-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=10\,FeedService=false\,MatchedPolicyID=49054ed0-68a6-11e1-bc72-0050568e013c\,NmapScanCount=0\,NmapSubnetScanID=0\,PortalUser=, EndpointSourceE
vent=SNMPQuery Probe, EndpointIdentityGroup=Profiled, ProfilerServer=stage-pdp01.cisco.com,
+Aug 6 20:00:52 10.42.7.64 Aug 7 03:20:05 npf-sjca-pdp02 CISE_Profiler 0000373185 1 0 2014-08-07 03:20:05.549 -07:00 0011310202 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=241, EndpointCertainityMetric=90, EndpointIPAddress=10.56.129.142, EndpointMacAddress=3C:A9:F4:46:75:CC, EndpointMatchedPolicy=Windows7-Workstation, EndpointNADAddress=10.56.129.4, EndpointOUI=Intel Corporate, EndpointPolicy=Windows7-Workstation, EndpointProperty=StaticAssignment=false\,Calling-Station-ID=3c-a9-f4-46-75-cc\,Device Identifier=\,PostureApplicable=Yes\,dhcp-class-identifier=MSFT 5.0\,host-name=EXAMPLE\,BYODRegistration=Unknown\,EndPointPolicyID=615ed410-68a6-11e1-bc72-0050568e013c\,FirstCollection=1406112353750\,TimeToProfile=11\,Framed-IP-Address=10.56.129.142\,LastNmapScanTime=0\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,NAS-Port-Type=Wireless - IEEE 802.11\,RegistrationTimeStamp=0\,UpdateTime=1407394245820\,PolicyVersion=403\,Identi
tyGroupID=5cb39b80-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=90\,FeedService=false\,MatchedPolicyID=615ed410-68a6-11e1-bc72-0050568e013c\,DestinationIPAddress=10.42.7.64\,CreateTime=1394526689397\,NmapScanCount=0\,NmapSubnetScanID=0\,AAA-Server=npf-sjca-pdp02\,PortalUser=, EndpointSourceEvent=RADIUS Probe, EndpointUserAgent=Mozilla/5.0 (Windows NT 6.1\\\; WOW64\\ rv:30.0) Gecko/20100101 Firefox/30.0, EndpointIdentityGroup=Workstation, ProfilerServer=npf.example.com,
+Aug 6 21:00:48 10.42.7.64 Aug 7 04:20:00 npf-sjca-pdp02 CISE_Profiler 0000373902 1 0 2014-08-07 04:20:00.983 -07:00 0011322557 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=241, EndpointCertainityMetric=30, EndpointIPAddress=10.56.129.142, EndpointMacAddress=3C:A9:F4:46:75:CC, EndpointMatchedPolicy=Microsoft-Workstation, EndpointNADAddress=10.56.129.4, EndpointOUI=Intel Corporate, EndpointPolicy=Microsoft-Workstation, EndpointProperty=StaticAssignment=false\,Calling-Station-ID=3c-a9-f4-46-75-cc\,Device Identifier=\,PostureApplicable=Yes\,dhcp-class-identifier=MSFT 5.0\,host-name=EXAMPLE\,BYODRegistration=Unknown\,EndPointPolicyID=5f4a24e0-68a6-11e1-bc72-0050568e013c\,FirstCollection=1406112353750\,TimeToProfile=11\,Framed-IP-Address=10.56.129.142\,LastNmapScanTime=0\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,NAS-Port-Type=Wireless - IEEE 802.11\,RegistrationTimeStamp=0\,UpdateTime=1407406806572\,PolicyVersion=403\,Iden
tityGroupID=5cb39b80-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=30\,FeedService=false\,MatchedPolicyID=5f4a24e0-68a6-11e1-bc72-0050568e013c\,DestinationIPAddress=10.42.7.64\,CreateTime=1394526689397\,NmapScanCount=0\,NmapSubnetScanID=0\,AAA-Server=npf-sjca-pdp02\,PortalUser=, EndpointSourceEvent=RADIUS Probe, EndpointUserAgent=MS-WebServices/1.0, EndpointIdentityGroup=Workstation, ProfilerServer=npf.example.com,
+Aug 6 22:22:50 10.42.7.64 Aug 7 05:42:03 npf-sjca-pdp02 CISE_Profiler 0000374846 1 0 2014-08-07 05:42:03.617 -07:00 0011340138 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=241, EndpointCertainityMetric=10, EndpointMacAddress=68:A8:6D:4E:0D:86, EndpointMatchedPolicy=Apple-Device, EndpointOUI=Apple, EndpointPolicy=Apple-Device, EndpointProperty=StaticAssignment=false\,PostureApplicable=Yes\,host-name=PEXAMPLE\,BYODRegistration=Unknown\,EndPointPolicyID=377d8ba0-68a6-11e1-bc72-0050568e013c\,FirstCollection=1407415322895\,TimeToProfile=717\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,PolicyVersion=403\,IdentityGroupID=abbbcac0-89e6-11e1-bf14-005056aa4dd7\,Total Certainty Factor=10\,ciaddr=0.0.0.0\,FeedService=false\,dhcp-parameter-request-list=1\, 3\, 6\, 15\, 119\, 95\, 252\, 44\, 46\,MatchedPolicyID=377d8ba0-68a6-11e1-bc72-0050568e013c\,NmapSubnetScanID=0\,PortalUser=, EndpointSourceEvent=DHCP Probe, EndpointIdentityGroup
=Apple-Device, ProfilerServer=npf.example.com,
+Aug 6 23:30:10 10.42.7.64 Aug 7 06:49:23 npf-sjca-pdp02 CISE_Profiler 0000375603 1 0 2014-08-07 06:49:23.920 -07:00 0011353768 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=241, EndpointCertainityMetric=90, EndpointIPAddress=10.56.129.142, EndpointMacAddress=3C:A9:F4:46:75:CC, EndpointMatchedPolicy=Windows7-Workstation, EndpointNADAddress=10.56.129.4, EndpointOUI=Intel Corporate, EndpointPolicy=Windows7-Workstation, EndpointProperty=StaticAssignment=false\,Calling-Station-ID=3c-a9-f4-46-75-cc\,Device Identifier=\,PostureApplicable=Yes\,dhcp-class-identifier=MSFT 5.0\,host-name=EXAMPLE\,BYODRegistration=Unknown\,EndPointPolicyID=615ed410-68a6-11e1-bc72-0050568e013c\,FirstCollection=1406112353750\,TimeToProfile=11\,Framed-IP-Address=10.56.129.142\,LastNmapScanTime=0\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,NAS-Port-Type=Wireless - IEEE 802.11\,RegistrationTimeStamp=0\,UpdateTime=1407410402099\,PolicyVersion=403\,Identi
tyGroupID=5cb39b80-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=90\,FeedService=false\,MatchedPolicyID=615ed410-68a6-11e1-bc72-0050568e013c\,DestinationIPAddress=10.42.7.64\,CreateTime=1394526689397\,NmapScanCount=0\,NmapSubnetScanID=0\,AAA-Server=npf-sjca-pdp02\,PortalUser=, EndpointSourceEvent=RADIUS Probe, EndpointUserAgent=Mozilla/5.0 (Windows NT 6.1\\\; WOW64\\ rv:30.0) Gecko/20100101 Firefox/30.0, EndpointIdentityGroup=Workstation, ProfilerServer=npf.example.com,
+Aug 6 23:30:48 10.42.7.64 Aug 7 06:50:01 npf-sjca-pdp02 CISE_Profiler 0000375611 1 0 2014-08-07 06:50:01.377 -07:00 0011353875 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=241, EndpointCertainityMetric=50, EndpointIPAddress=10.34.92.103, EndpointMacAddress=3C:A9:F4:29:FC:3C, EndpointMatchedPolicy=Microsoft-Workstation, EndpointNADAddress=10.34.76.212, EndpointOUI=Intel Corporate, EndpointPolicy=Microsoft-Workstation, EndpointProperty=StaticAssignment=false\,Calling-Station-ID=3c-a9-f4-29-fc-3c\,Device Identifier=\,PostureApplicable=Yes\,dhcp-class-identifier=MSFT 5.0\,host-name=EXAMPLE\,BYODRegistration=Unknown\,EndPointPolicyID=5f4a24e0-68a6-11e1-bc72-0050568e013c\,FirstCollection=1406109860322\,L4_DST_PORT=50428\,TimeToProfile=7\,Framed-IP-Address=10.34.92.103\,LastNmapScanTime=1380758278898\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,NAS-Port-Type=Wireless - IEEE 802.11\,RegistrationTimeStamp=0\,UpdateTime=140668603
4558\,PolicyVersion=403\,IdentityGroupID=5cb39b80-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=50\,operating-system=Microsoft Windows Vista SP0 - SP2\, Server 2008\, or Windows 7 Ultimate\,FeedService=false\,MatchedPolicyID=5f4a24e0-68a6-11e1-bc72-0050568e013c\,DestinationIPAddress=10.42.7.64\,CreateTime=1373657280926\,NmapScanCount=3\,NmapSubnetScanID=0\,AAA-Server=npf-sjca-pdp02\,PortalUser=, EndpointSourceEvent=RADIUS Probe, EndpointUserAgent=MS-WebServices/1.0, EndpointIdentityGroup=Workstation, ProfilerServer=npf.example.com,
+Aug 6 23:32:52 10.42.7.64 Aug 7 06:52:05 npf-sjca-pdp02 CISE_Profiler 0000375636 1 0 2014-08-07 06:52:05.272 -07:00 0011354313 80002 INFO Profiler: Profiler EndPoint profiling event occurred, ConfigVersionId=241, EndpointCertainityMetric=30, EndpointIPAddress=10.56.129.143, EndpointMacAddress=E8:2A:EA:23:5E:3D, EndpointMatchedPolicy=Microsoft-Workstation, EndpointNADAddress=10.56.129.4, EndpointOUI=Intel Corporate, EndpointPolicy=Microsoft-Workstation, EndpointProperty=StaticAssignment=false\,Calling-Station-ID=e8-2a-ea-23-5e-3d\,Device Identifier=\,PostureApplicable=Yes\,dhcp-class-identifier=MSFT 5.0\,host-name=ANOY-WS01\,BYODRegistration=Unknown\,EndPointPolicyID=5f4a24e0-68a6-11e1-bc72-0050568e013c\,FirstCollection=1406114784910\,TimeToProfile=7\,Framed-IP-Address=10.56.129.143\,LastNmapScanTime=0\,StaticGroupAssignment=false\,DeviceRegistrationStatus=NotRegistered\,NAS-Port-Type=Wireless - IEEE 802.11\,RegistrationTimeStamp=0\,UpdateTime=1407395211208\,PolicyVersion=403\,Ide
ntityGroupID=5cb39b80-68a6-11e1-bc72-0050568e013c\,Total Certainty Factor=30\,FeedService=false\,MatchedPolicyID=5f4a24e0-68a6-11e1-bc72-0050568e013c\,DestinationIPAddress=10.42.7.64\,CreateTime=1405408515121\,NmapScanCount=0\,NmapSubnetScanID=0\,AAA-Server=npf-sjca-pdp02\,PortalUser=, EndpointSourceEvent=RADIUS Probe, EndpointUserAgent=MS-WebServices/1.0, EndpointIdentityGroup=Workstation, ProfilerServer=npf.example.com,
+Aug 6 16:40:52 10.42.7.64 Aug 7 00:00:04 npf-sjca-pdp02 CISE_Failed_Attempts 0000370855 1 0 2014-08-07 00:00:04.527 -07:00 0011266584 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270932, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056EF53E323F4, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:40:57 10.42.7.63 Aug 7 00:00:09 npf-sjca-pdp01 CISE_Failed_Attempts 0001969834 1 0 2014-08-07 00:00:09.568 -07:00 0098648519 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2084839, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D4A53E323F9, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:41:24 10.34.84.145 Aug 7 00:00:36 stage-pdp01 CISE_Failed_Attempts 0000024616 1 0 2014-08-07 00:00:36.332 -07:00 0000287007 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19317, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:41:26 10.34.84.145 Aug 7 00:00:38 stage-pdp01 CISE_Failed_Attempts 0000024617 1 0 2014-08-07 00:00:38.336 -07:00 0000287011 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19318, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:41:28 10.34.84.145 Aug 7 00:00:40 stage-pdp01 CISE_Failed_Attempts 0000024618 1 0 2014-08-07 00:00:40.336 -07:00 0000287015 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19319, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:41:30 10.34.84.145 Aug 7 00:00:42 stage-pdp01 CISE_Failed_Attempts 0000024619 1 0 2014-08-07 00:00:42.340 -07:00 0000287019 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19320, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:41:32 10.34.84.145 Aug 7 00:00:44 stage-pdp01 CISE_Failed_Attempts 0000024620 1 0 2014-08-07 00:00:44.340 -07:00 0000287023 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19321, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:41:34 10.34.84.145 Aug 7 00:00:46 stage-pdp01 CISE_Failed_Attempts 0000024621 1 0 2014-08-07 00:00:46.344 -07:00 0000287027 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19322, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:42:02 10.42.7.64 Aug 7 00:01:14 npf-sjca-pdp02 CISE_Failed_Attempts 0000370865 1 0 2014-08-07 00:01:14.610 -07:00 0011266810 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=7, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270940, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F053E3243A, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:42:07 10.42.7.63 Aug 7 00:01:19 npf-sjca-pdp01 CISE_Failed_Attempts 0001969923 1 0 2014-08-07 00:01:19.665 -07:00 0098652715 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2084986, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D4B53E3243F, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:42:12 10.42.7.64 Aug 7 00:01:24 npf-sjca-pdp02 CISE_Failed_Attempts 0000370867 1 0 2014-08-07 00:01:24.701 -07:00 0011266815 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270941, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F153E32444, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:42:17 10.42.7.63 Aug 7 00:01:29 npf-sjca-pdp01 CISE_Failed_Attempts 0001969935 1 0 2014-08-07 00:01:29.746 -07:00 0098653362 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085007, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D4C53E32449, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:43:22 10.42.7.64 Aug 7 00:02:34 npf-sjca-pdp02 CISE_Failed_Attempts 0000370885 1 0 2014-08-07 00:02:34.792 -07:00 0011267367 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=4, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270956, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F353E3248A, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:43:27 10.42.7.63 Aug 7 00:02:39 npf-sjca-pdp01 CISE_Failed_Attempts 0001970043 1 0 2014-08-07 00:02:39.808 -07:00 0098657578 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085161, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D4D53E3248F, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:43:56 10.42.7.64 Aug 7 00:03:08 npf-sjca-pdp02 CISE_Failed_Attempts 0000370897 1 0 2014-08-07 00:03:08.902 -07:00 0011267657 5435 NOTICE RADIUS: NAS conducted several failed authentications of the same scenario, ConfigVersionId=240, Device IP Address=10.56.129.4, Device Port=32770, DestinationIPAddress=10.42.7.64, DestinationPort=1813, RadiusPacketType=AccountingRequest, UserName=yshchory, Protocol=Radius, RequestLatency=49, NetworkDeviceName=NTN-WLC1, User-Name=yshchory, NAS-IP-Address=10.56.129.4, NAS-Port=1, Framed-IP-Address=10.56.129.141, Class=CACS:0a388104000045cd53e2be75:npf-sjca-pdp02/195481465/270958, Called-Station-ID=6c-41-6a-5f-6e-c0, Calling-Station-ID=90-18-7c-7b-59-01, NAS-Identifier=ntn01-11a-wlc1, Acct-Status-Type=Interim-Update, Acct-Delay-Time=0, Acct-Input-Octets=2359603, Acct-Output-Octets=26928466, Acct-Session-Id=53e2be78/90:18:7c:7b:59:01/13844, Acct-Authentic=RADIUS, Acct-Session-Time=1466, Acct-Input-Packets=14866, Acct-Output-Packets=23043, und
efined-52=
+Aug 6 16:44:01 10.42.7.63 Aug 7 00:03:13 npf-sjca-pdp01 CISE_Failed_Attempts 0001970072 1 0 2014-08-07 00:03:13.112 -07:00 0098658804 5435 NOTICE RADIUS: NAS conducted several failed authentications of the same scenario, ConfigVersionId=133, Device IP Address=10.56.72.127, Device Port=1646, DestinationIPAddress=10.42.7.63, DestinationPort=1813, Protocol=Radius, NetworkDeviceName=ntn01-11a-sw4, User-Name=host/salfi-pc.cisco.com, NAS-IP-Address=10.56.72.127, NAS-Port=50212, Service-Type=Framed, Framed-IP-Address=10.56.111.14, Class=CACS:0A38487F00000397BDA7BCAC:npf-sjca-pdp02/195481465/270957, Called-Station-ID=00-26-99-28-5E-BB, Calling-Station-ID=3C-97-0E-C3-F8-F1, Acct-Status-Type=Interim-Update, Acct-Delay-Time=4, Acct-Input-Octets=225395, Acct-Output-Octets=761436, Acct-Session-Id=00000560, Acct-Authentic=RADIUS, Acct-Session-Time=43, Acct-Input-Packets=1163, Acct-Output-Packets=1080, NAS-Port-Type=Ethernet, NAS-Port-Id=GigabitEthernet2/12, undefined-151=F54C88B0, cisco-av-pair
=audit-session-id=0A38487F00000397BDA7BCAC, cisco-av-pair=connect-progress=Auth Open, AcsSessionID=npf-sjca-pdp01/195491152/2085221, FailureReason=11038 RADIUS Accounting-Request header contains invalid Authenticator field, Step=11004, Step=11017, Step=11038, Step=5435, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0A38487F00000397BDA7BCAC, TotalFailedAttempts=2, TotalFailedTime=42, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired,
+Aug 6 16:44:32 10.42.7.64 Aug 7 00:03:44 npf-sjca-pdp02 CISE_Failed_Attempts 0000370899 1 0 2014-08-07 00:03:44.851 -07:00 0011267663 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=7, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270963, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F453E324D0, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:44:36 10.34.84.145 Aug 7 00:03:48 stage-pdp01 CISE_Failed_Attempts 0000024632 1 0 2014-08-07 00:03:48.375 -07:00 0000287084 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19329, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:44:37 10.42.7.63 Aug 7 00:03:49 npf-sjca-pdp01 CISE_Failed_Attempts 0001970128 1 0 2014-08-07 00:03:49.893 -07:00 0098661643 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085307, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D4E53E324D5, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:44:38 10.34.84.145 Aug 7 00:03:50 stage-pdp01 CISE_Failed_Attempts 0000024633 1 0 2014-08-07 00:03:50.379 -07:00 0000287088 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19330, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:44:40 10.34.84.145 Aug 7 00:03:52 stage-pdp01 CISE_Failed_Attempts 0000024634 1 0 2014-08-07 00:03:52.379 -07:00 0000287092 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19331, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:44:42 10.34.84.145 Aug 7 00:03:54 stage-pdp01 CISE_Failed_Attempts 0000024635 1 0 2014-08-07 00:03:54.387 -07:00 0000287096 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19332, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:44:42 10.42.7.64 Aug 7 00:03:54 npf-sjca-pdp02 CISE_Failed_Attempts 0000370903 1 0 2014-08-07 00:03:54.924 -07:00 0011267670 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=4, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270964, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F553E324DA, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:44:44 10.34.84.145 Aug 7 00:03:56 stage-pdp01 CISE_Failed_Attempts 0000024636 1 0 2014-08-07 00:03:56.386 -07:00 0000287100 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19333, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:44:46 10.34.84.145 Aug 7 00:03:58 stage-pdp01 CISE_Failed_Attempts 0000024637 1 0 2014-08-07 00:03:58.390 -07:00 0000287104 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19334, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:44:47 10.42.7.63 Aug 7 00:03:59 npf-sjca-pdp01 CISE_Failed_Attempts 0001970140 1 0 2014-08-07 00:03:59.951 -07:00 0098662310 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085331, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D4F53E324DF, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:44:48 10.42.7.64 Aug 7 00:04:00 npf-sjca-pdp02 CISE_Failed_Attempts 0000370905 1 0 2014-08-07 00:04:00.526 -07:00 0011267674 5435 NOTICE RADIUS: NAS conducted several failed authentications of the same scenario, ConfigVersionId=240, Device IP Address=10.56.72.127, Device Port=1646, DestinationIPAddress=10.42.7.64, DestinationPort=1813, Protocol=Radius, NetworkDeviceName=ntn01-11a-sw4, User-Name=host/salfi-pc.cisco.com, NAS-IP-Address=10.56.72.127, NAS-Port=50212, Service-Type=Framed, Framed-IP-Address=169.254.53.87, Class=CACS:0A38487F00000397BDA7BCAC:npf-sjca-pdp02/195481465/270957, Called-Station-ID=00-26-99-28-5E-BB, Calling-Station-ID=3C-97-0E-C3-F8-F1, Acct-Status-Type=Interim-Update, Acct-Delay-Time=0, Acct-Input-Octets=1458615, Acct-Output-Octets=3836368, Acct-Session-Id=00000560, Acct-Authentic=RADIUS, Acct-Session-Time=95, Acct-Input-Packets=4505, Acct-Output-Packets=5619, NAS-Port-Type=Ethernet, NAS-Port-Id=GigabitEthernet2/12, undefined-151=F54C88B0, cisco-av-p
air=audit-session-id=0A38487F00000397BDA7BCAC, cisco-av-pair=connect-progress=Auth Open, AcsSessionID=npf-sjca-pdp02/195481465/270965, FailureReason=11038 RADIUS Accounting-Request header contains invalid Authenticator field, Step=11004, Step=11017, Step=11038, Step=5435, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0A38487F00000397BDA7BCAC, TotalFailedAttempts=2, TotalFailedTime=52, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired,
+Aug 6 16:45:52 10.42.7.64 Aug 7 00:05:04 npf-sjca-pdp02 CISE_Failed_Attempts 0000370920 1 0 2014-08-07 00:05:04.969 -07:00 0011267987 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=6, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270977, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F653E32520, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:45:58 10.42.7.63 Aug 7 00:05:09 npf-sjca-pdp01 CISE_Failed_Attempts 0001970212 1 0 2014-08-07 00:05:09.998 -07:00 0098665518 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085460, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5053E32525, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:47:03 10.42.7.64 Aug 7 00:06:15 npf-sjca-pdp02 CISE_Failed_Attempts 0000370931 1 0 2014-08-07 00:06:15.016 -07:00 0011268196 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270985, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F753E32567, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:47:08 10.42.7.63 Aug 7 00:06:20 npf-sjca-pdp01 CISE_Failed_Attempts 0001970324 1 0 2014-08-07 00:06:20.055 -07:00 0098669942 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085599, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5153E3256C, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:47:13 10.42.7.64 Aug 7 00:06:25 npf-sjca-pdp02 CISE_Failed_Attempts 0000370934 1 0 2014-08-07 00:06:25.097 -07:00 0011268209 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/270987, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F853E32571, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:47:18 10.42.7.63 Aug 7 00:06:30 npf-sjca-pdp01 CISE_Failed_Attempts 0001970335 1 0 2014-08-07 00:06:30.119 -07:00 0098670037 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085618, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5253E32576, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:47:48 10.34.84.145 Aug 7 00:07:00 stage-pdp01 CISE_Failed_Attempts 0000024649 1 0 2014-08-07 00:07:00.418 -07:00 0000287210 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19342, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:47:50 10.34.84.145 Aug 7 00:07:02 stage-pdp01 CISE_Failed_Attempts 0000024650 1 0 2014-08-07 00:07:02.421 -07:00 0000287214 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19343, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:47:52 10.34.84.145 Aug 7 00:07:04 stage-pdp01 CISE_Failed_Attempts 0000024651 1 0 2014-08-07 00:07:04.425 -07:00 0000287218 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19344, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:47:54 10.34.84.145 Aug 7 00:07:06 stage-pdp01 CISE_Failed_Attempts 0000024652 1 0 2014-08-07 00:07:06.429 -07:00 0000287222 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19345, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:47:56 10.34.84.145 Aug 7 00:07:08 stage-pdp01 CISE_Failed_Attempts 0000024653 1 0 2014-08-07 00:07:08.429 -07:00 0000287226 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19346, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:47:58 10.34.84.145 Aug 7 00:07:10 stage-pdp01 CISE_Failed_Attempts 0000024654 1 0 2014-08-07 00:07:10.433 -07:00 0000287230 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19347, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:48:23 10.42.7.64 Aug 7 00:07:35 npf-sjca-pdp02 CISE_Failed_Attempts 0000370955 1 0 2014-08-07 00:07:35.138 -07:00 0011268472 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271001, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056F953E325B7, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:48:28 10.42.7.63 Aug 7 00:07:40 npf-sjca-pdp01 CISE_Failed_Attempts 0001970420 1 0 2014-08-07 00:07:40.178 -07:00 0098673462 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085757, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5353E325BC, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:49:33 10.42.7.64 Aug 7 00:08:45 npf-sjca-pdp02 CISE_Failed_Attempts 0000370984 1 0 2014-08-07 00:08:45.219 -07:00 0011269071 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271016, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056FB53E325FD, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:49:38 10.42.7.63 Aug 7 00:08:50 npf-sjca-pdp01 CISE_Failed_Attempts 0001970519 1 0 2014-08-07 00:08:50.259 -07:00 0098677825 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085892, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5453E32602, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:49:43 10.42.7.64 Aug 7 00:08:55 npf-sjca-pdp02 CISE_Failed_Attempts 0000370986 1 0 2014-08-07 00:08:55.298 -07:00 0011269076 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271017, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056FC53E32607, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:49:48 10.42.7.63 Aug 7 00:09:00 npf-sjca-pdp01 CISE_Failed_Attempts 0001970524 1 0 2014-08-07 00:09:00.330 -07:00 0098678019 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2085909, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5553E3260C, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:50:53 10.42.7.64 Aug 7 00:10:05 npf-sjca-pdp02 CISE_Failed_Attempts 0000370999 1 0 2014-08-07 00:10:05.339 -07:00 0011269371 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271027, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056FD53E3264D, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:50:58 10.42.7.63 Aug 7 00:10:10 npf-sjca-pdp01 CISE_Failed_Attempts 0001970625 1 0 2014-08-07 00:10:10.388 -07:00 0098682297 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086061, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5653E32652, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:51:00 10.34.84.145 Aug 7 00:10:12 stage-pdp01 CISE_Failed_Attempts 0000024661 1 0 2014-08-07 00:10:12.492 -07:00 0000287258 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19354, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:51:02 10.34.84.145 Aug 7 00:10:14 stage-pdp01 CISE_Failed_Attempts 0000024662 1 0 2014-08-07 00:10:14.496 -07:00 0000287262 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19355, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:51:04 10.34.84.145 Aug 7 00:10:16 stage-pdp01 CISE_Failed_Attempts 0000024663 1 0 2014-08-07 00:10:16.496 -07:00 0000287266 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19356, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:51:06 10.34.84.145 Aug 7 00:10:18 stage-pdp01 CISE_Failed_Attempts 0000024664 1 0 2014-08-07 00:10:18.500 -07:00 0000287270 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19357, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:51:08 10.34.84.145 Aug 7 00:10:20 stage-pdp01 CISE_Failed_Attempts 0000024665 1 0 2014-08-07 00:10:20.504 -07:00 0000287274 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19358, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:51:10 10.34.84.145 Aug 7 00:10:22 stage-pdp01 CISE_Failed_Attempts 0000024667 1 0 2014-08-07 00:10:22.507 -07:00 0000287279 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19359, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:52:03 10.42.7.64 Aug 7 00:11:15 npf-sjca-pdp02 CISE_Failed_Attempts 0000371005 1 0 2014-08-07 00:11:15.432 -07:00 0011269421 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=4, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271031, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056FE53E32693, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:52:08 10.42.7.63 Aug 7 00:11:20 npf-sjca-pdp01 CISE_Failed_Attempts 0001970691 1 0 2014-08-07 00:11:20.468 -07:00 0098685176 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086181, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5753E32698, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:52:13 10.42.7.64 Aug 7 00:11:25 npf-sjca-pdp02 CISE_Failed_Attempts 0000371007 1 0 2014-08-07 00:11:25.515 -07:00 0011269426 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271032, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a0740000056FF53E3269D, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:52:18 10.42.7.63 Aug 7 00:11:30 npf-sjca-pdp01 CISE_Failed_Attempts 0001970708 1 0 2014-08-07 00:11:30.551 -07:00 0098685669 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=8, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086202, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5853E326A2, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:53:23 10.42.7.64 Aug 7 00:12:35 npf-sjca-pdp02 CISE_Failed_Attempts 0000371016 1 0 2014-08-07 00:12:35.547 -07:00 0011269586 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271040, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570053E326E3, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:53:28 10.42.7.63 Aug 7 00:12:40 npf-sjca-pdp01 CISE_Failed_Attempts 0001970802 1 0 2014-08-07 00:12:40.596 -07:00 0098689883 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086334, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5953E326E8, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:54:12 10.34.84.145 Aug 7 00:13:24 stage-pdp01 CISE_Failed_Attempts 0000024680 1 0 2014-08-07 00:13:24.527 -07:00 0000287388 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19368, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:54:14 10.34.84.145 Aug 7 00:13:26 stage-pdp01 CISE_Failed_Attempts 0000024681 1 0 2014-08-07 00:13:26.531 -07:00 0000287392 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19369, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:54:16 10.34.84.145 Aug 7 00:13:28 stage-pdp01 CISE_Failed_Attempts 0000024682 1 0 2014-08-07 00:13:28.534 -07:00 0000287396 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19370, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:54:18 10.34.84.145 Aug 7 00:13:30 stage-pdp01 CISE_Failed_Attempts 0000024683 1 0 2014-08-07 00:13:30.538 -07:00 0000287400 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19371, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:54:20 10.34.84.145 Aug 7 00:13:32 stage-pdp01 CISE_Failed_Attempts 0000024684 1 0 2014-08-07 00:13:32.538 -07:00 0000287404 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19372, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:54:22 10.34.84.145 Aug 7 00:13:34 stage-pdp01 CISE_Failed_Attempts 0000024685 1 0 2014-08-07 00:13:34.542 -07:00 0000287408 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19373, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:54:33 10.42.7.64 Aug 7 00:13:45 npf-sjca-pdp02 CISE_Failed_Attempts 0000371020 1 0 2014-08-07 00:13:45.628 -07:00 0011269631 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271044, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570153E32729, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:54:38 10.42.7.63 Aug 7 00:13:50 npf-sjca-pdp01 CISE_Failed_Attempts 0001970913 1 0 2014-08-07 00:13:50.668 -07:00 0098695334 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086486, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5A53E3272E, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:54:43 10.42.7.64 Aug 7 00:13:55 npf-sjca-pdp02 CISE_Failed_Attempts 0000371025 1 0 2014-08-07 00:13:55.694 -07:00 0011269740 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=7, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271048, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570253E32733, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:54:48 10.42.7.63 Aug 7 00:14:00 npf-sjca-pdp01 CISE_Failed_Attempts 0001970924 1 0 2014-08-07 00:14:00.705 -07:00 0098695591 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086505, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5B53E32738, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:55:53 10.42.7.64 Aug 7 00:15:05 npf-sjca-pdp02 CISE_Failed_Attempts 0000371036 1 0 2014-08-07 00:15:05.742 -07:00 0011270054 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=6, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271057, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570353E32779, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:55:58 10.42.7.63 Aug 7 00:15:10 npf-sjca-pdp01 CISE_Failed_Attempts 0001970997 1 0 2014-08-07 00:15:10.772 -07:00 0098698954 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086621, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5C53E3277E, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:57:03 10.42.7.64 Aug 7 00:16:15 npf-sjca-pdp02 CISE_Failed_Attempts 0000371051 1 0 2014-08-07 00:16:15.827 -07:00 0011270497 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=6, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271067, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570453E327BF, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:57:08 10.42.7.63 Aug 7 00:16:20 npf-sjca-pdp01 CISE_Failed_Attempts 0001971096 1 0 2014-08-07 00:16:20.857 -07:00 0098703837 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086806, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5D53E327C4, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:57:24 10.34.84.145 Aug 7 00:16:36 stage-pdp01 CISE_Failed_Attempts 0000024697 1 0 2014-08-07 00:16:36.602 -07:00 0000287553 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19384, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:57:26 10.34.84.145 Aug 7 00:16:38 stage-pdp01 CISE_Failed_Attempts 0000024698 1 0 2014-08-07 00:16:38.605 -07:00 0000287557 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19385, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:57:28 10.34.84.145 Aug 7 00:16:40 stage-pdp01 CISE_Failed_Attempts 0000024699 1 0 2014-08-07 00:16:40.609 -07:00 0000287561 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19386, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:57:30 10.34.84.145 Aug 7 00:16:42 stage-pdp01 CISE_Failed_Attempts 0000024700 1 0 2014-08-07 00:16:42.613 -07:00 0000287565 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19387, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:57:32 10.34.84.145 Aug 7 00:16:44 stage-pdp01 CISE_Failed_Attempts 0000024701 1 0 2014-08-07 00:16:44.613 -07:00 0000287569 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19388, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:57:34 10.34.84.145 Aug 7 00:16:46 stage-pdp01 CISE_Failed_Attempts 0000024702 1 0 2014-08-07 00:16:46.617 -07:00 0000287573 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19389, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 16:58:03 10.42.7.64 Aug 7 00:17:15 npf-sjca-pdp02 CISE_Failed_Attempts 0000371063 1 0 2014-08-07 00:17:15.966 -07:00 0011270832 5435 NOTICE RADIUS: NAS conducted several failed authentications of the same scenario, ConfigVersionId=240, Device IP Address=10.34.76.212, Device Port=32770, DestinationIPAddress=10.42.7.64, DestinationPort=1813, RadiusPacketType=AccountingRequest, UserName=hslai, Protocol=Radius, RequestLatency=25, NetworkDeviceName=sjcm-00a-npf-wlc1, User-Name=hslai, NAS-IP-Address=10.34.76.212, NAS-Port=1, Framed-IP-Address=10.34.94.11, Class=CACS:0a224cd40002fdf953e327f2:npf-sjca-pdp02/195481465/271072, Called-Station-ID=88-43-e1-62-1d-20, Calling-Station-ID=24-a2-e1-3b-4b-cb, NAS-Identifier=sjcm-00a-npf-wlc1, Acct-Status-Type=Interim-Update, Acct-Delay-Time=0, Acct-Input-Octets=5198, Acct-Output-Octets=4093, Acct-Session-Id=53e327f2/24:a2:e1:3b:4b:cb/174403, Acct-Authentic=RADIUS, Acct-Session-Time=9, Acct-Input-Packets=37, Acct-Output-Packets=13, undefined-52
=
+Aug 6 16:58:13 10.42.7.64 Aug 7 00:17:25 npf-sjca-pdp02 CISE_Failed_Attempts 0000371065 1 0 2014-08-07 00:17:25.902 -07:00 0011270838 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=4, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271076, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570553E32805, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:58:18 10.42.7.63 Aug 7 00:17:30 npf-sjca-pdp01 CISE_Failed_Attempts 0001971204 1 0 2014-08-07 00:17:30.916 -07:00 0098707928 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2086981, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5E53E3280A, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 16:59:23 10.42.7.64 Aug 7 00:18:35 npf-sjca-pdp02 CISE_Failed_Attempts 0000371070 1 0 2014-08-07 00:18:35.942 -07:00 0011271044 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271081, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570653E3284B, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 16:59:28 10.42.7.64 Aug 7 00:18:40 npf-sjca-pdp02 CISE_Failed_Attempts 0000371072 1 0 2014-08-07 00:18:40.669 -07:00 0011271053 5400 NOTICE Failed-Attempt: Authentication failed, ConfigVersionId=240, Device IP Address=10.56.129.4, Device Port=32770, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=istern, Protocol=Radius, RequestLatency=12, NetworkDeviceName=NTN-WLC1, User-Name=istern, NAS-IP-Address=10.56.129.4, NAS-Port=1, Service-Type=Framed, Framed-MTU=1300, State=37CPMSessionID=0a388104000045de53e2c750\;41SessionID=npf-sjca-pdp02/195481465/271077\;, Called-Station-ID=70-10-5c-f3-2f-80:alpha_example, Calling-Station-ID=f0-27-65-48-8c-8f, NAS-Identifier=ntn01-11a-wlc1, NAS-Port-Type=Wireless - IEEE 802.11, Tunnel-Type=(tag=0) VLAN, Tunnel-Medium-Type=(tag=0) 802, Tunnel-Private-Group-ID=(tag=0) 604, undefined-89=
+Aug 6 16:59:28 10.42.7.63 Aug 7 00:18:40 npf-sjca-pdp01 CISE_Failed_Attempts 0001971282 1 0 2014-08-07 00:18:40.981 -07:00 0098711291 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=2, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2087140, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D5F53E32850, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 17:00:33 10.42.7.64 Aug 7 00:19:46 npf-sjca-pdp02 CISE_Failed_Attempts 0000371080 1 0 2014-08-07 00:19:46.020 -07:00 0011271232 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=240, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.64, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=5, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp02/195481465/271087, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a07400000570753E32892, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Resp
onse={RadiusPacketType=Drop; },
+Aug 6 17:00:36 10.34.84.145 Aug 7 00:19:48 stage-pdp01 CISE_Failed_Attempts 0000024712 1 0 2014-08-07 00:19:48.660 -07:00 0000287604 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19396, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 17:00:38 10.34.84.145 Aug 7 00:19:50 stage-pdp01 CISE_Failed_Attempts 0000024713 1 0 2014-08-07 00:19:50.664 -07:00 0000287608 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19397, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 17:00:39 10.42.7.63 Aug 7 00:19:51 npf-sjca-pdp01 CISE_Failed_Attempts 0001971393 1 0 2014-08-07 00:19:51.042 -07:00 0098716185 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=133, Device IP Address=10.56.72.126, Device Port=1645, DestinationIPAddress=10.42.7.63, DestinationPort=1812, RadiusPacketType=AccessRequest, UserName=#CTSREQUEST#, Protocol=Radius, RequestLatency=1, NetworkDeviceName=ntn01-11a-sw3, User-Name=#CTSREQUEST#, NAS-IP-Address=10.56.72.126, Service-Type=Outbound, AcsSessionID=npf-sjca-pdp01/195491152/2087311, SelectedAccessService=NDAC_SGT_Service, FailureReason=11302 Received Secure RADIUS request without a cts-pac-opaque cisco-av-pair attribute, Step=11001, Step=11017, Step=15012, Step=11302, NetworkDeviceGroups=Location#All Locations#NTN, NetworkDeviceGroups=Device Type#All Device Types#Wired, CPMSessionID=0a2a073f00005D6053E32897, Model Name=4503, Location=Location#All Locations#NTN, Device Type=Device Type#All Device Types#Wired, Res
ponse={RadiusPacketType=Drop; },
+Aug 6 17:00:40 10.34.84.145 Aug 7 00:19:52 stage-pdp01 CISE_Failed_Attempts 0000024714 1 0 2014-08-07 00:19:52.664 -07:00 0000287612 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19398, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 17:00:42 10.34.84.145 Aug 7 00:19:54 stage-pdp01 CISE_Failed_Attempts 0000024715 1 0 2014-08-07 00:19:54.668 -07:00 0000287616 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19399, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 17:00:44 10.34.84.145 Aug 7 00:19:56 stage-pdp01 CISE_Failed_Attempts 0000024716 1 0 2014-08-07 00:19:56.672 -07:00 0000287620 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19400, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 17:00:46 10.34.84.145 Aug 7 00:19:58 stage-pdp01 CISE_Failed_Attempts 0000024717 1 0 2014-08-07 00:19:58.675 -07:00 0000287624 5405 NOTICE Failed-Attempt: RADIUS Request dropped, ConfigVersionId=113, Device IP Address=172.23.91.132, Device Port=32769, DestinationIPAddress=10.34.84.145, DestinationPort=1812, Protocol=Radius, User-Name=test, NAS-IP-Address=192.168.30.11, Service-Type=NAS Prompt, NAS-Identifier=Cisco_1b:e0:84, AcsSessionID=stage-pdp01/196593288/19401, FailureReason=11007 Could not locate Network Device or AAA Client, Step=11001, Step=11017, Step=11007, Step=5405,
+Aug 6 17:01:44 10.42.7.64 Aug 7 00:20:56 npf-sjca-pdp02 CISE_Failed_Attempts 0000371095 1 0 2014-08-07 00:20:56.06
<TRUNCATED>
[17/18] metron git commit: METRON-995: Temporary variables in stellar
enrichments which are maps do not function as expected closes
apache/metron#615
Posted by ma...@apache.org.
METRON-995: Temporary variables in stellar enrichments which are maps do not function as expected closes apache/metron#615
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/de2c871a
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/de2c871a
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/de2c871a
Branch: refs/heads/Metron_0.4.0
Commit: de2c871abf3e259311281743952eb465362cf90c
Parents: f523c17
Author: cstella <ce...@gmail.com>
Authored: Mon Jun 19 10:37:50 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Mon Jun 19 10:37:50 2017 -0400
----------------------------------------------------------------------
.../enrichment/handler/StellarConfig.java | 13 ++---
.../metron/common/stellar/StellarCompiler.java | 2 +
.../StellarEnrichmentConfigTest.java | 37 +++++++++++++
.../configuration/StellarEnrichmentTest.java | 44 +++++++++++++++-
.../metron/common/stellar/StellarTest.java | 5 ++
.../adapters/stellar/StellarAdapter.java | 35 ++++++++++---
.../adapters/stellar/StellarAdapterTest.java | 55 ++++++++++++++++++++
.../integration/EnrichmentIntegrationTest.java | 5 ++
.../main/config/zookeeper/enrichments/test.json | 4 +-
9 files changed, 185 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
index 7d49fbd..a888b81 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/handler/StellarConfig.java
@@ -17,18 +17,18 @@
*/
package org.apache.metron.common.configuration.enrichment.handler;
-import com.google.common.base.Splitter;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.Iterables;
+import com.google.common.base.Joiner;
import org.apache.metron.common.stellar.StellarAssignment;
import org.apache.metron.common.stellar.StellarProcessor;
import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.util.*;
import java.util.function.Function;
public class StellarConfig implements Config {
-
+ protected static final Logger _LOG = LoggerFactory.getLogger(StellarConfig.class);
@Override
public List<String> getSubgroups(Iterable<Map.Entry<String, Object>> config) {
boolean includeEmpty = false;
@@ -81,10 +81,10 @@ public class StellarConfig implements Config {
{
StellarProcessor processor = new StellarProcessor();
List<JSONObject> messages = new ArrayList<>();
- Map<String, String> defaultStellarStatementGroup = new HashMap<>();
+ List<String> defaultStellarStatementGroup = new ArrayList<>();
for(Map.Entry<String, Object> kv : config) {
if(kv.getValue() instanceof String) {
- defaultStellarStatementGroup.put(kv.getKey(), (String)kv.getValue());
+ defaultStellarStatementGroup.add((String)kv.getValue());
}
else if(kv.getValue() instanceof Map) {
JSONObject ret = new JSONObject();
@@ -103,6 +103,7 @@ public class StellarConfig implements Config {
ret.put("", getMessage(getFields(processor, defaultStellarStatementGroup), message));
messages.add(ret);
}
+ _LOG.debug("Stellar enrichment split: {}", messages );
return messages;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
index 136728d..f4e101b 100644
--- a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
+++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/StellarCompiler.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.common.stellar;
+import org.antlr.v4.runtime.ParserRuleContext;
import org.apache.commons.lang3.StringEscapeUtils;
import org.apache.metron.common.dsl.Context;
import org.apache.metron.common.dsl.Token;
@@ -347,6 +348,7 @@ public class StellarCompiler extends StellarBaseListener {
}, DeferredFunction.class, context));
}
+
@Override
public void exitVariable(StellarParser.VariableContext ctx) {
final FrameContext.Context context = getArgContext();
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
index 0883672..32fb00e 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentConfigTest.java
@@ -18,7 +18,9 @@
package org.apache.metron.common.configuration;
import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
import org.apache.metron.common.configuration.enrichment.handler.Configs;
@@ -33,6 +35,41 @@ import java.util.Map;
public class StellarEnrichmentConfigTest extends StellarEnrichmentTest {
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : [
+ "dga_model_endpoint := MAAS_GET_ENDPOINT('dga')",
+ "dga_result_map := MAAS_MODEL_APPLY( dga_model_endpoint, { 'host' : domain_without_subdomains } )",
+ "dga_result := MAP_GET('is_malicious', dga_result_map)",
+ "is_dga := dga_result != null && dga_result == 'dga'",
+ "dga_model_version := MAP_GET('version', dga_model_endpoint)",
+ "dga_model_endpoint := null",
+ "dga_result_map := null",
+ "dga_result := null"
+ ]
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String conf;
+
+ @Test
+ public void testSplitter_listWithTemporaryVariables() throws IOException {
+ JSONObject message = new JSONObject(ImmutableMap.of("domain_without_subdomains", "yahoo.com"));
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(conf, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ List<JSONObject> splits = Configs.STELLAR.splitByFields(message, null, x -> null, handler );
+ Assert.assertEquals(1, splits.size());
+ Map<String, Object> split = (Map<String, Object>)(splits.get(0)).get("");
+ Assert.assertEquals("yahoo.com", split.get("domain_without_subdomains"));
+ Assert.assertTrue(split.containsKey("dga_result"));
+ Assert.assertTrue(split.containsKey("dga_model_endpoint"));
+ Assert.assertTrue(split.containsKey("dga_result_map"));
+ }
@Test
public void testSplitter_default() throws IOException {
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
index 3af15cd..8eb9bbc 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/StellarEnrichmentTest.java
@@ -59,7 +59,25 @@ public class StellarEnrichmentTest {
*/
@Multiline
public static String defaultStellarConfig_list;
- public static List<String> DEFAULT_CONFIGS = ImmutableList.of(defaultStellarConfig_list, defaultStellarConfig_map);
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt4 := TO_LOWER(string)",
+ "stmt2 := TO_LOWER(stmt1)",
+ "stmt3 := TO_LOWER(string)",
+ "stmt4 := null"
+ ]
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String defaultStellarConfig_listWithTemp;
+ public static List<String> DEFAULT_CONFIGS = ImmutableList.of(defaultStellarConfig_list, defaultStellarConfig_map, defaultStellarConfig_listWithTemp);
/**
{
@@ -100,7 +118,29 @@ public class StellarEnrichmentTest {
*/
@Multiline
public static String groupedStellarConfig_list;
- public static List<String> GROUPED_CONFIGS = ImmutableList.of(groupedStellarConfig_list, groupedStellarConfig_map);
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := TO_LOWER(stmt1)"
+ ],
+ "group2" : [
+ "stmt3 := TO_LOWER(string)",
+ "stmt4 := TO_LOWER(string)",
+ "stmt4 := null"
+ ]
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String groupedStellarConfig_listWithTemp;
+ public static List<String> GROUPED_CONFIGS = ImmutableList.of(groupedStellarConfig_listWithTemp, groupedStellarConfig_list, groupedStellarConfig_map);
/**
{
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
index 98abd57..d549e19 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/stellar/StellarTest.java
@@ -183,8 +183,13 @@ public class StellarTest {
Assert.assertEquals(ImmutableSet.of("one", "two")
, processor.variablesUsed("if 1 < 2 then one else two"));
}
+ {
+ Assert.assertEquals(ImmutableSet.of("bar")
+ , processor.variablesUsed("MAP_GET('foo', { 'foo' : bar})"));
+ }
}
+
@Test
public void testFunctionEmptyArgs() {
{
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
index 53ef16f..4b78821 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
@@ -32,6 +32,8 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
import java.util.Map;
import java.util.function.Function;
@@ -101,7 +103,9 @@ public class StellarAdapter implements EnrichmentAdapter<CacheKey>,Serializable
JSONObject ret = new JSONObject();
Iterable<Map.Entry<String, Object>> stellarStatements = getStellarStatements(handler, field);
+ _LOG.debug("message := {}", message);
if(stellarStatements != null) {
+ List<String> mapEntries = new ArrayList<>();
for (Map.Entry<String, Object> kv : stellarStatements) {
if(kv.getKey() != null && kv.getValue() != null) {
if (kv.getValue() instanceof String) {
@@ -114,14 +118,11 @@ public class StellarAdapter implements EnrichmentAdapter<CacheKey>,Serializable
_PERF_LOG.debug("SLOW LOG: " + stellarStatement + " took" + duration + "ms");
}
}
+ _LOG.debug("{} := {} yields {}", kv.getKey(), stellarStatement , o);
if (o != null && o instanceof Map) {
- for (Map.Entry<Object, Object> valueKv : ((Map<Object, Object>) o).entrySet()) {
- String newKey = ((kv.getKey().length() > 0) ? kv.getKey() + "." : "") + valueKv.getKey();
- message.put(newKey, valueKv.getValue());
- ret.put(newKey, valueKv.getValue());
- }
+ mapEntries.add(kv.getKey());
}
- else if(o == null) {
+ if(o == null) {
message.remove(kv.getKey());
ret.remove(kv.getKey());
}
@@ -132,6 +133,28 @@ public class StellarAdapter implements EnrichmentAdapter<CacheKey>,Serializable
}
}
}
+ /*
+ We need to handle the map entries separately now.
+ We want to explode them out, so if "var" is
+ {
+ "foo" : "bar"
+ }
+ then we want "var.foo" == "bar"
+ and no "var"
+ */
+ for(String mapEntry : mapEntries) {
+ String key = mapEntry;
+ Map<Object, Object> value = (Map<Object, Object>) ret.get(key);
+ if(value != null) {
+ _LOG.debug("Exploding map: {} == {}", key, value);
+ for (Map.Entry<Object, Object> valueKv : value.entrySet()) {
+ String newKey = ((key.length() > 0) ? key + "." : "") + valueKv.getKey();
+ ret.put(newKey, valueKv.getValue());
+ }
+ //removing the map from downstream
+ ret.remove(key);
+ }
+ }
}
return ret;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
index 5624d4b..724221d 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapterTest.java
@@ -18,6 +18,7 @@
package org.apache.metron.enrichment.adapters.stellar;
import com.google.common.collect.ImmutableList;
+import org.adrianwalker.multilinestring.Multiline;
import org.apache.metron.common.configuration.StellarEnrichmentTest;
import org.apache.metron.common.configuration.enrichment.EnrichmentConfig;
import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
@@ -30,6 +31,8 @@ import org.json.simple.JSONObject;
import org.junit.Assert;
import org.junit.Test;
+import java.util.List;
+
public class StellarAdapterTest extends StellarEnrichmentTest {
StellarProcessor processor = new StellarProcessor();
@@ -131,4 +134,56 @@ public class StellarAdapterTest extends StellarEnrichmentTest {
Assert.assertEquals(2, enriched.size());
}
}
+
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : {
+ "group1" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := { 'foo' : source.type }"
+ ]
+ }
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String mapConfig_subgroup;
+ /**
+ {
+ "fieldMap": {
+ "stellar" : {
+ "config" : [
+ "stmt1 := TO_UPPER(source.type)",
+ "stmt2 := { 'foo' : source.type }"
+ ]
+ }
+ }
+ }
+ */
+ @Multiline
+ public static String mapConfig_default;
+
+ private void testMapEnrichment(String config, String field) throws Exception {
+ JSONObject message = getMessage();
+ EnrichmentConfig enrichmentConfig = JSONUtils.INSTANCE.load(config, EnrichmentConfig.class);
+ Assert.assertNotNull(enrichmentConfig.getEnrichmentConfigs().get("stellar"));
+ ConfigHandler handler = enrichmentConfig.getEnrichmentConfigs().get("stellar");
+ JSONObject enriched = enrich(message, field, handler);
+ Assert.assertEquals(2, enriched.size());
+ Assert.assertEquals("stellar_test", enriched.get("stmt2.foo"));
+ Assert.assertEquals("stellar_test".toUpperCase(), enriched.get("stmt1"));
+ }
+
+ @Test
+ public void testMapEnrichment_subgroup() throws Exception {
+ testMapEnrichment(mapConfig_subgroup, "group1");
+ }
+
+ @Test
+ public void testMapEnrichment_default() throws Exception {
+ testMapEnrichment(mapConfig_default, "");
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
index 9634805..2149cb9 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
@@ -260,6 +260,11 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
Assert.assertNotNull(jsonDoc.get(DST_IP));
Assert.assertNotNull(jsonDoc.get("ALL_CAPS"));
+ Assert.assertNotNull(jsonDoc.get("map.blah"));
+ Assert.assertNull(jsonDoc.get("map"));
+ Assert.assertNotNull(jsonDoc.get("one"));
+ Assert.assertEquals(1, jsonDoc.get("one"));
+ Assert.assertEquals(1, jsonDoc.get("map.blah"));
Assert.assertNotNull(jsonDoc.get("foo"));
Assert.assertEquals("TEST", jsonDoc.get("ALL_CAPS"));
Assert.assertNotNull(jsonDoc.get("bar"));
http://git-wip-us.apache.org/repos/asf/metron/blob/de2c871a/metron-platform/metron-integration-test/src/main/config/zookeeper/enrichments/test.json
----------------------------------------------------------------------
diff --git a/metron-platform/metron-integration-test/src/main/config/zookeeper/enrichments/test.json b/metron-platform/metron-integration-test/src/main/config/zookeeper/enrichments/test.json
index 1037b69..bba6679 100644
--- a/metron-platform/metron-integration-test/src/main/config/zookeeper/enrichments/test.json
+++ b/metron-platform/metron-integration-test/src/main/config/zookeeper/enrichments/test.json
@@ -16,7 +16,9 @@
"stellar" : {
"config" : {
"numeric" : {
- "foo": "1 + 1"
+ "map" : "{ 'blah' : 1}"
+ ,"one" : "MAP_GET('blah', map)"
+ ,"foo": "1 + 1"
}
,"ALL_CAPS" : "TO_UPPER(source.type)"
,"src_enrichment" : {
[10/18] metron git commit: METRON-962 Configuration Based Unit Tests
and Add integration tests (justinleet via leet) closes apache/metron#612
Posted by ma...@apache.org.
METRON-962 Configuration Based Unit Tests and Add integration tests (justinleet via leet) closes apache/metron#612
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/5b72da7b
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/5b72da7b
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/5b72da7b
Branch: refs/heads/Metron_0.4.0
Commit: 5b72da7be0521fc16e887dd4a1c958ceaeaa6510
Parents: 605a3cc
Author: justinleet <ju...@gmail.com>
Authored: Wed Jun 7 10:19:56 2017 -0400
Committer: YOUR NAME as In Apache <YO...@apache.org>
Committed: Wed Jun 7 10:19:56 2017 -0400
----------------------------------------------------------------------
.../common/cli/ConfigurationsUtilsTest.java | 2 +-
.../configuration/ConfigurationsTest.java | 2 +-
.../SensorEnrichmentConfigTest.java | 2 +-
.../metron/enrichment/bolt/SplitBoltTest.java | 2 +-
.../bolt/ThreatIntelJoinBoltTest.java | 2 +-
.../bolt/ThreatIntelSplitterBoltTest.java | 3 +-
.../metron/parsers/asa/GrokAsaParser.java | 285 -----------------
.../parsers/fireeye/BasicFireEyeParser.java | 2 +-
.../metron/parsers/AbstractConfigTest.java | 300 ------------------
.../parsers/AbstractParserConfigTest.java | 92 ++++++
.../metron/parsers/AbstractSchemaTest.java | 204 ------------
.../metron/parsers/AbstractTestContext.java | 190 ------------
.../apache/metron/parsers/GrokParserTest.java | 2 +-
.../metron/parsers/MessageParserTest.java | 2 +-
.../metron/parsers/asa/GrokAsaParserTest.java | 169 ----------
.../metron/parsers/bro/BasicBroParserTest.java | 62 ++--
.../metron/parsers/bro/BroParserTest.java | 164 ----------
.../metron/parsers/cef/CEFParserTest.java | 95 +++---
.../parsers/fireeye/BasicFireEyeParserTest.java | 157 ++--------
.../integration/ParserIntegrationTest.java | 2 +-
.../metron/parsers/ise/BasicIseParserTest.java | 176 ++---------
.../parsers/lancope/BasicLancopeParserTest.java | 148 ++-------
.../BasicPaloAltoFirewallParserTest.java | 164 ++--------
.../sourcefire/BasicSourcefireParserTest.java | 164 ++--------
.../resources/config/GrokAsaParserTest.config | 20 --
.../resources/logData/FireEyeParserTest.txt | 8 +
.../test/resources/logData/IseParserTest.txt | 308 +++++++++++++++++++
.../resources/logData/LancopeParserTest.txt | 1 +
.../logData/PaloAltoFirewallParserTest.txt | 2 +
.../resources/logData/SourcefireParserTest.txt | 3 +
.../apache/metron/pcap/utils/PcapUtilsTest.java | 2 +-
31 files changed, 648 insertions(+), 2087 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
index 0645619..0c72183 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/cli/ConfigurationsUtilsTest.java
@@ -17,12 +17,12 @@
*/
package org.apache.metron.common.cli;
-import junit.framework.Assert;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.test.TestingServer;
import org.apache.metron.TestConstants;
import org.apache.metron.common.configuration.ConfigurationsUtils;
import org.apache.metron.common.utils.JSONUtils;
+import org.junit.Assert;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/ConfigurationsTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/ConfigurationsTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/ConfigurationsTest.java
index 4720929..992f0a7 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/ConfigurationsTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/ConfigurationsTest.java
@@ -17,10 +17,10 @@
*/
package org.apache.metron.common.configuration;
-import junit.framework.Assert;
import nl.jqno.equalsverifier.EqualsVerifier;
import nl.jqno.equalsverifier.Warning;
import org.junit.Test;
+import org.junit.Assert;
import java.io.IOException;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/SensorEnrichmentConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/SensorEnrichmentConfigTest.java b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/SensorEnrichmentConfigTest.java
index 305182b..5a77b4f 100644
--- a/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/SensorEnrichmentConfigTest.java
+++ b/metron-platform/metron-common/src/test/java/org/apache/metron/common/configuration/SensorEnrichmentConfigTest.java
@@ -17,12 +17,12 @@
*/
package org.apache.metron.common.configuration;
-import junit.framework.Assert;
import nl.jqno.equalsverifier.EqualsVerifier;
import nl.jqno.equalsverifier.Warning;
import org.apache.commons.io.IOUtils;
import org.apache.metron.TestConstants;
import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import org.junit.Assert;
import org.junit.Test;
import java.io.BufferedReader;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/SplitBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/SplitBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/SplitBoltTest.java
index d0ca475..41d34de 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/SplitBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/SplitBoltTest.java
@@ -22,10 +22,10 @@ import org.apache.storm.topology.OutputFieldsDeclarer;
import org.apache.storm.tuple.Tuple;
import org.apache.storm.tuple.Values;
import com.google.common.collect.ImmutableList;
-import junit.framework.Assert;
import org.apache.metron.common.configuration.ConfigurationType;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
import org.json.simple.JSONObject;
+import org.junit.Assert;
import org.junit.Test;
import java.util.HashMap;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
index 0f3cc8c..04617ff 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
@@ -18,7 +18,6 @@
package org.apache.metron.enrichment.bolt;
import com.fasterxml.jackson.databind.JsonMappingException;
-import junit.framework.Assert;
import org.adrianwalker.multilinestring.Multiline;
import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
import org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
@@ -31,6 +30,7 @@ import org.apache.storm.tuple.Tuple;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelSplitterBoltTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelSplitterBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelSplitterBoltTest.java
index 6898c89..4feba2e 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelSplitterBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelSplitterBoltTest.java
@@ -17,14 +17,13 @@
*/
package org.apache.metron.enrichment.bolt;
-import junit.framework.Assert;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
+import org.junit.Assert;
import org.junit.Test;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.HashMap;
-import java.util.List;
import java.util.Map;
public class ThreatIntelSplitterBoltTest extends BaseEnrichmentBoltTest {
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/GrokAsaParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/GrokAsaParser.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/GrokAsaParser.java
deleted file mode 100644
index 4f1c8b0..0000000
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/GrokAsaParser.java
+++ /dev/null
@@ -1,285 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.parsers.asa;
-
-import oi.thekraken.grok.api.Grok;
-import oi.thekraken.grok.api.Match;
-import oi.thekraken.grok.api.exception.GrokException;
-import org.apache.commons.io.IOUtils;
-import org.apache.metron.parsers.BasicParser;
-import org.json.simple.JSONObject;
-
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.text.ParseException;
-import java.text.SimpleDateFormat;
-import java.util.*;
-
-public class GrokAsaParser extends BasicParser {
-
- private static final long serialVersionUID = 945353287115350798L;
- private transient Grok grok;
- Map<String, String> patternMap;
- private transient Map<String, Grok> grokMap;
- private transient InputStream pattern_url;
-
- public static final String PREFIX = "stream2file";
- public static final String SUFFIX = ".tmp";
-
- public static File stream2file(InputStream in) throws IOException {
- final File tempFile = File.createTempFile(PREFIX, SUFFIX);
- tempFile.deleteOnExit();
- try (FileOutputStream out = new FileOutputStream(tempFile)) {
- IOUtils.copy(in, out);
- }
- return tempFile;
- }
-
- public GrokAsaParser() throws Exception {
- // pattern_url = Resources.getResource("patterns/asa");
-
- pattern_url = getClass().getClassLoader().getResourceAsStream(
- "src/main/patterns/asa");
-
- File file = stream2file(pattern_url);
- grok = Grok.create(file.getPath());
-
- patternMap = getPatternMap();
- grokMap = getGrokMap();
-
- grok.compile("%{CISCO_TAGGED_SYSLOG}");
- }
-
- public GrokAsaParser(String filepath) throws Exception {
-
- grok = Grok.create(filepath);
- // grok.getNamedRegexCollection().put("ciscotag","CISCOFW302013_302014_302015_302016");
- grok.compile("%{CISCO_TAGGED_SYSLOG}");
-
- }
-
- public GrokAsaParser(String filepath, String pattern) throws Exception {
-
- grok = Grok.create(filepath);
- grok.compile("%{" + pattern + "}");
- }
-
- private Map<String, Object> getMap(String pattern, String text)
- throws GrokException {
-
- Grok g = grokMap.get(pattern);
- if (g != null) {
- Match gm = g.match(text);
- gm.captures();
- return gm.toMap();
- } else {
- return new HashMap<String, Object>();
- }
-
- }
-
- private Map<String, Grok> getGrokMap() throws GrokException, IOException {
- Map<String, Grok> map = new HashMap<String, Grok>();
-
- for (Map.Entry<String, String> entry : patternMap.entrySet()) {
- File file = stream2file(pattern_url);
- Grok grok = Grok.create(file.getPath());
- grok.compile("%{" + entry.getValue() + "}");
-
- map.put(entry.getValue(), grok);
-
- }
-
- return map;
- }
-
- private Map<String, String> getPatternMap() {
- Map<String, String> map = new HashMap<String, String>();
-
- map.put("ASA-2-106001", "CISCOFW106001");
- map.put("ASA-2-106006", "CISCOFW106006_106007_106010");
- map.put("ASA-2-106007", "CISCOFW106006_106007_106010");
- map.put("ASA-2-106010", "CISCOFW106006_106007_106010");
- map.put("ASA-3-106014", "CISCOFW106014");
- map.put("ASA-6-106015", "CISCOFW106015");
- map.put("ASA-1-106021", "CISCOFW106021");
- map.put("ASA-4-106023", "CISCOFW106023");
- map.put("ASA-5-106100", "CISCOFW106100");
- map.put("ASA-6-110002", "CISCOFW110002");
- map.put("ASA-6-302010", "CISCOFW302010");
- map.put("ASA-6-302013", "CISCOFW302013_302014_302015_302016");
- map.put("ASA-6-302014", "CISCOFW302013_302014_302015_302016");
- map.put("ASA-6-302015", "CISCOFW302013_302014_302015_302016");
- map.put("ASA-6-302016", "CISCOFW302013_302014_302015_302016");
- map.put("ASA-6-302020", "CISCOFW302020_302021");
- map.put("ASA-6-302021", "CISCOFW302020_302021");
- map.put("ASA-6-305011", "CISCOFW305011");
- map.put("ASA-3-313001", "CISCOFW313001_313004_313008");
- map.put("ASA-3-313004", "CISCOFW313001_313004_313008");
- map.put("ASA-3-313008", "CISCOFW313001_313004_313008");
- map.put("ASA-4-313005", "CISCOFW313005");
- map.put("ASA-4-402117", "CISCOFW402117");
- map.put("ASA-4-402119", "CISCOFW402119");
- map.put("ASA-4-419001", "CISCOFW419001");
- map.put("ASA-4-419002", "CISCOFW419002");
- map.put("ASA-4-500004", "CISCOFW500004");
- map.put("ASA-6-602303", "CISCOFW602303_602304");
- map.put("ASA-6-602304", "CISCOFW602303_602304");
- map.put("ASA-7-710001", "CISCOFW710001_710002_710003_710005_710006");
- map.put("ASA-7-710002", "CISCOFW710001_710002_710003_710005_710006");
- map.put("ASA-7-710003", "CISCOFW710001_710002_710003_710005_710006");
- map.put("ASA-7-710005", "CISCOFW710001_710002_710003_710005_710006");
- map.put("ASA-7-710006", "CISCOFW710001_710002_710003_710005_710006");
- map.put("ASA-6-713172", "CISCOFW713172");
- map.put("ASA-4-733100", "CISCOFW733100");
- map.put("ASA-6-305012", "CISCOFW305012");
- map.put("ASA-7-609001", "CISCOFW609001");
- map.put("ASA-7-609002", "CISCOFW609002");
-
- return map;
- }
-
- public static Long convertToEpoch(String m, String d, String ts,
- boolean adjust_timezone) throws ParseException {
- d = d.trim();
-
- if (d.length() <= 2)
- d = "0" + d;
-
- Date date = new SimpleDateFormat("MMM", Locale.ENGLISH).parse(m);
- Calendar cal = Calendar.getInstance();
- cal.setTime(date);
- String month = String.valueOf(cal.get(Calendar.MONTH));
- int year = Calendar.getInstance().get(Calendar.YEAR);
-
- if (month.length() <= 2)
- month = "0" + month;
-
- String coglomerated_ts = year + "-" + month + "-" + d + " " + ts;
-
- System.out.println(coglomerated_ts);
-
- SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
-
- if (adjust_timezone)
- sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
-
- date = sdf.parse(coglomerated_ts);
- long timeInMillisSinceEpoch = date.getTime();
-
- return timeInMillisSinceEpoch;
- }
-
- @Override
- public void configure(Map<String, Object> parserConfig) {
-
- }
-
- @Override
- public void init() {
- // pattern_url = Resources.getResource("patterns/asa");
-
- pattern_url = getClass().getClassLoader().getResourceAsStream(
- "src/main/patterns/asa");
-
- File file = null;
- try {
- file = stream2file(pattern_url);
- } catch (IOException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- try {
- grok = Grok.create(file.getPath());
- } catch (GrokException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
-
- patternMap = getPatternMap();
- try {
- grokMap = getGrokMap();
- } catch (GrokException | IOException e1) {
- // TODO Auto-generated catch block
- e1.printStackTrace();
- }
-
- try {
- grok.compile("%{CISCO_TAGGED_SYSLOG}");
- } catch (GrokException e) {
- // TODO Auto-generated catch block
- e.printStackTrace();
- }
- }
-
- @Override
- public List<JSONObject> parse(byte[] raw_message) {
-
- String toParse = "";
- JSONObject toReturn;
- List<JSONObject> messages = new ArrayList<>();
- try {
-
- toParse = new String(raw_message, "UTF-8");
-
- System.out.println("Received message: " + toParse);
-
- Match gm = grok.match(toParse);
- gm.captures();
-
- toReturn = new JSONObject();
-
- toReturn.putAll(gm.toMap());
-
- String str = toReturn.get("ciscotag").toString();
- String pattern = patternMap.get(str);
-
- Map<String, Object> response = getMap(pattern, toParse);
-
- toReturn.putAll(response);
-
- //System.out.println("*******I MAPPED: " + toReturn);
- long timestamp = convertToEpoch(toReturn.get("MONTH").toString(), toReturn
- .get("MONTHDAY").toString(),
- toReturn.get("TIME").toString(),
- true);
- toReturn.put("timestamp", timestamp);
-
- toReturn.remove("MONTHDAY");
- toReturn.remove("TIME");
- toReturn.remove("MINUTE");
- toReturn.remove("HOUR");
- toReturn.remove("YEAR");
- toReturn.remove("SECOND");
-
- toReturn.put("ip_src_addr", toReturn.remove("IPORHOST"));
- toReturn.put("original_string", toParse);
- messages.add(toReturn);
- return messages;
-
- } catch (Exception e) {
- e.printStackTrace();
- return null;
- }
-
- }
-
-
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
index 498248a..839b8f9 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
@@ -185,7 +185,7 @@ public class BasicFireEyeParser extends BasicParser {
if (ip_dst_port != null)
toReturn.put("ip_dst_port", ip_dst_port);
- System.out.println(toReturn);
+// System.out.println(toReturn);
return toReturn;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractConfigTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractConfigTest.java
deleted file mode 100644
index 26ce051..0000000
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractConfigTest.java
+++ /dev/null
@@ -1,300 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.parsers;
-import java.io.BufferedReader;
-import java.io.File;
-import java.io.FileReader;
-import java.net.URL;
-import java.util.ArrayList;
-import java.util.Map;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.commons.configuration.ConfigurationException;
-import org.apache.commons.configuration.PropertiesConfiguration;
-
-import com.fasterxml.jackson.databind.JsonNode;
-import com.github.fge.jackson.JsonLoader;
-import com.github.fge.jsonschema.core.report.ProcessingReport;
-import com.github.fge.jsonschema.main.JsonSchemaFactory;
-import com.github.fge.jsonschema.main.JsonValidator;
-
-/**
- * <ul>
- * <li>Title: </li>
- * <li>Description: The class <code>AbstractConfigTest</code> is
- * an abstract base class for implementing JUnit tests that need to use
- * config to connect to ZooKeeper and HBase. The <code>setup</code> method will attempt to
- * load a properties from a file, located in src/test/resources,
- * with the same name as the class.</li>
- * <li>Created: Oct 10, 2014</li>
- * </ul>
- * @version $Revision: 1.1 $
- */
-public class AbstractConfigTest extends AbstractTestContext {
- /**
- * The configPath.
- */
- protected String configPath=null;
-
- /**
- * The configName.
- */
- protected String configName=null;
-
- /**
- * The config.
- */
- private Configuration config=null;
-
- /**
- * The settings.
- */
- Map<String, String> settings=null;
-
- /**
- * The schemaJsonString.
- */
- private String schemaJsonString = null;
- /**
- * Any Object for mavenMode
- * @parameter
- * expression="${mode}"
- * default-value="local"
- */
- private Object mode="local";
-
- /**
- * Constructs a new <code>AbstractConfigTest</code> instance.
- * @throws Exception
- */
- public AbstractConfigTest() throws Exception {
- super.setUp();
- }
-
- /**
- * Constructs a new <code>AbstractTestContext</code> instance.
- * @param name the name of the test case.
- */
- public AbstractConfigTest(String name) {
- super(name);
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#setUp()
- */
- protected void setUp(String configName) throws Exception {
- super.setUp();
- this.setConfigPath("src/test/resources/config/"+getClass().getSimpleName()+".config");
- try {
- this.setConfig(new PropertiesConfiguration(this.getConfigPath()));
-
- Map configOptions= SettingsLoader.getConfigOptions((PropertiesConfiguration)this.config, configName+"=");
- this.setSettings(SettingsLoader.getConfigOptions((PropertiesConfiguration)this.config, configName + "."));
- this.getSettings().put(configName, (String) configOptions.get(configName));
- } catch (ConfigurationException e) {
- e.printStackTrace();
- throw new Exception("Config not found !!"+e);
- }
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#tearDown()
- */
- @Override
- protected void tearDown() throws Exception {
-
- }
-
-
- /**
- * validateJsonData
- * @param jsonSchema
- * @param jsonData
- * @return
- * @throws Exception
- */
-
- protected boolean validateJsonData(final String jsonSchema, final String jsonData)
- throws Exception {
-
- final JsonNode d = JsonLoader.fromString(jsonData);
- final JsonNode s = JsonLoader.fromString(jsonSchema);
-
- final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
- JsonValidator v = factory.getValidator();
-
- ProcessingReport report = v.validate(s, d);
- System.out.println(report);
-
- return report.toString().contains("success");
- }
-
- protected String readSchemaFromFile(URL schema_url) throws Exception {
- BufferedReader br = new BufferedReader(new FileReader(
- schema_url.getFile()));
- String line;
- StringBuilder sb = new StringBuilder();
- while ((line = br.readLine()) != null) {
- System.out.println(line);
- sb.append(line);
- }
- br.close();
-
- String schema_string = sb.toString().replaceAll("\n", "");
- schema_string = schema_string.replaceAll(" ", "");
-
- System.out.println("Read in schema: " + schema_string);
-
- return schema_string;
- }
-
- protected String[] readTestDataFromFile(String test_data_url) throws Exception {
- BufferedReader br = new BufferedReader(new FileReader(
- new File(test_data_url)));
- ArrayList<String> inputDataLines = new ArrayList<String>();
-
- String line;
- while ((line = br.readLine()) != null) {
- System.out.println(line);
- inputDataLines.add(line.toString().replaceAll("\n", ""));
- }
- br.close();
- String[] inputData = new String[inputDataLines.size()];
- inputData = inputDataLines.toArray(inputData);
-
- return inputData;
- }
- /**
- * Skip Tests
- */
- @Override
- public boolean skipTests(Object mode){
- if(mode.toString().equals("local")){
- return true;
- }else {
- return false;
- }
- }
-
- /**
- * Returns the mode.
- * @return the mode.
- */
-
- @Override
- public Object getMode() {
- return mode;
- }
-
- /**
- * Sets the mode.
- * @param mode the mode.
- */
-
- @Override
- public void setMode(Object mode) {
-
- this.mode = mode;
- }
-
-
- /**
- * @param schemaJsonString
- */
- public void setSchemaJsonString(String schemaJsonString) {
- this.schemaJsonString=schemaJsonString;
- }
-
-
- /**
- * @return
- */
- public String getSchemaJsonString() {
- return this.schemaJsonString;
- }
-
- /**
- * Returns the configPath.
- * @return the configPath.
- */
- public String getConfigPath() {
- return configPath;
- }
-
- /**
- * Sets the configPath.
- * @param configPath the configPath.
- */
- public void setConfigPath(String configPath) {
- this.configPath = configPath;
- }
- /**
- * Returns the config.
- * @return the config.
- */
-
- public Configuration getConfig() {
- return config;
- }
-
- /**
- * Sets the config.
- * @param config the config.
- */
-
- public void setConfig(Configuration config) {
-
- this.config = config;
- }
- /**
- * Returns the settings.
- * @return the settings.
- */
-
- public Map<String, String> getSettings() {
- return settings;
- }
-
- /**
- * Sets the settings.
- * @param settings the settings.
- */
-
- public void setSettings(Map<String, String> settings) {
- this.settings = settings;
- }
- /**
- * Returns the configName.
- * @return the configName.
- */
- public String getConfigName() {
- return configName;
- }
-
- /**
- * Sets the configName.
- * @param configName the configName.
- */
- public void setConfigName(String configName) {
- this.configName = configName;
- }
-}
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractParserConfigTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractParserConfigTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractParserConfigTest.java
new file mode 100644
index 0000000..dd43cc3
--- /dev/null
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractParserConfigTest.java
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.github.fge.jackson.JsonLoader;
+import com.github.fge.jsonschema.core.exceptions.ProcessingException;
+import com.github.fge.jsonschema.core.report.ProcessingReport;
+import com.github.fge.jsonschema.main.JsonSchemaFactory;
+import com.github.fge.jsonschema.main.JsonValidator;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.net.URL;
+import java.util.ArrayList;
+
+public class AbstractParserConfigTest {
+
+ protected static BasicParser parser = null;
+ protected static String[] inputStrings;
+ private String schemaJsonString = null;
+
+ protected boolean validateJsonData(final String jsonSchema, final String jsonData)
+ throws IOException, ProcessingException {
+
+ final JsonNode d = JsonLoader.fromString(jsonData);
+ final JsonNode s = JsonLoader.fromString(jsonSchema);
+
+ final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
+ JsonValidator v = factory.getValidator();
+
+ ProcessingReport report = v.validate(s, d);
+
+ return report.toString().contains("success");
+ }
+
+ protected String readSchemaFromFile(URL schema_url) throws Exception {
+ BufferedReader br = new BufferedReader(new FileReader(
+ schema_url.getFile()));
+ String line;
+ StringBuilder sb = new StringBuilder();
+ while ((line = br.readLine()) != null) {
+ sb.append(line);
+ }
+ br.close();
+
+ String schema_string = sb.toString().replaceAll("\n", "");
+ schema_string = schema_string.replaceAll(" ", "");
+
+ return schema_string;
+ }
+
+ protected String[] readTestDataFromFile(String test_data_url) throws Exception {
+ BufferedReader br = new BufferedReader(new FileReader(new File(test_data_url)));
+ ArrayList<String> inputDataLines = new ArrayList<>();
+
+ String line;
+ while ((line = br.readLine()) != null) {
+ inputDataLines.add(line.replaceAll("\n", ""));
+ }
+ br.close();
+ String[] inputData = new String[inputDataLines.size()];
+ inputData = inputDataLines.toArray(inputData);
+
+ return inputData;
+ }
+
+ public void setSchemaJsonString(String schemaJsonString) {
+ this.schemaJsonString = schemaJsonString;
+ }
+
+
+ public String getSchemaJsonString() {
+ return this.schemaJsonString;
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractSchemaTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractSchemaTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractSchemaTest.java
deleted file mode 100644
index b1cc597..0000000
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractSchemaTest.java
+++ /dev/null
@@ -1,204 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.parsers;
-import java.io.BufferedReader;
-import java.io.FileReader;
-import java.net.URL;
-
-import com.fasterxml.jackson.databind.JsonNode;
-import com.github.fge.jackson.JsonLoader;
-import com.github.fge.jsonschema.core.report.ProcessingReport;
-import com.github.fge.jsonschema.main.JsonSchemaFactory;
-import com.github.fge.jsonschema.main.JsonValidator;
-
- /**
- * <ul>
- * <li>Title: </li>
- * <li>Description: The class <code>AbstractSchemaTest</code> is
- * an abstract base class for implementing JUnit tests that need to load a
- * Json Schema. The <code>setup</code> method will attempt to
- * load a properties from a file, located in src/test/resources,
- * with the same name as the class.</li>
- * <li>Created: Aug 7, 2014</li>
- * </ul>
- * @version $Revision: 1.1 $
- */
-public class AbstractSchemaTest extends AbstractConfigTest{
-
-
- /**
- * The schemaJsonString.
- */
- private String schemaJsonString = null;
- /**
- * Any Object for mavenMode
- * @parameter
- * expression="${mode}"
- * default-value="local"
- */
- private Object mode="local";
-
- /**
- * Constructs a new <code>AbstractTestContext</code> instance.
- * @throws Exception
- */
- public AbstractSchemaTest() throws Exception {
- super.setUp();
- }
-
- /**
- * Constructs a new <code>AbstractTestContext</code> instance.
- * @param name the name of the test case.
- */
- public AbstractSchemaTest(String name) {
- super(name);
- try{
- if(System.getProperty("mode")!=null){
- setMode(System.getProperty("mode") );
- }else
- {
- setMode("local");
- }
- }catch(Exception ex){
- setMode("local");
- }
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#setUp()
- */
- @Override
- protected void setUp() throws Exception {
- super.setUp();
-
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#tearDown()
- */
- @Override
- protected void tearDown() throws Exception {
-
- }
-
-
- /**
- * validateJsonData
- * @param jsonSchema
- * @param jsonData
- * @return
- * @throws Exception
- */
-
- @Override
- protected boolean validateJsonData(final String jsonSchema, final String jsonData)
- throws Exception {
-
- final JsonNode d = JsonLoader.fromString(jsonData);
- final JsonNode s = JsonLoader.fromString(jsonSchema);
-
- final JsonSchemaFactory factory = JsonSchemaFactory.byDefault();
- JsonValidator v = factory.getValidator();
-
- ProcessingReport report = v.validate(s, d);
- System.out.println(report);
-
- return report.toString().contains("success");
- }
-
- @Override
- protected String readSchemaFromFile(URL schema_url) throws Exception {
- BufferedReader br = new BufferedReader(new FileReader(
- schema_url.getFile()));
- String line;
- StringBuilder sb = new StringBuilder();
- while ((line = br.readLine()) != null) {
- System.out.println(line);
- sb.append(line);
- }
- br.close();
-
- String schema_string = sb.toString().replaceAll("\n", "");
- schema_string = schema_string.replaceAll(" ", "");
-
- System.out.println("Read in schema: " + schema_string);
-
- return schema_string;
-
- }
-
- /**
- * Skip Tests
- */
- @Override
- public boolean skipTests(Object mode){
- if(mode.toString().equals("local")){
- return true;
- }else {
- return false;
- }
- }
-
- /**
- * Returns the mode.
- * @return the mode.
- */
-
- @Override
- public Object getMode() {
- return mode;
- }
-
- /**
- * Sets the mode.
- * @param mode the mode.
- */
-
- @Override
- public void setMode(Object mode) {
-
- this.mode = mode;
- }
-
-
- /**
-
- * @param schemaJsonString
- */
-
- @Override
- public void setSchemaJsonString(String schemaJsonString) {
- this.schemaJsonString=schemaJsonString;
- }
-
-
- /**
-
- * @return
- */
-
- @Override
- public String getSchemaJsonString() {
- return this.schemaJsonString;
- }
-
-}
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractTestContext.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractTestContext.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractTestContext.java
deleted file mode 100644
index 134b896..0000000
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/AbstractTestContext.java
+++ /dev/null
@@ -1,190 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.parsers;
-import org.junit.After;
-import org.junit.Test;
-
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.IOException;
-import java.io.InputStream;
-import java.util.Properties;
-
- /**
- * <ul>
- * <li>Title: </li>
- * <li>Description: The class <code>AbstractTestContext</code> is
- * an abstract base class for implementing JUnit tests that need to load a
- * test properties. The <code>setup</code> method will attempt to
- * load a properties from a file, located in src/test/resources,
- * with the same name as the class.</li>
- * <li>Created: Aug 7, 2014</li>
- * </ul>
- * @version $Revision: 1.1 $
- */
-public class AbstractTestContext {
- /**
- * The testProps.
- */
- protected File testPropFile=null;
-
- /**
- * The properties loaded for test.
- */
- protected Properties testProperties=new Properties();
-
- /**
- * Any Object for mavenMode
- * @parameter
- * expression="${mode}"
- * default-value="global"
- */
- private Object mode="local";
-
- /**
- * Constructs a new <code>AbstractTestContext</code> instance.
- */
- public AbstractTestContext() {
- super();
- }
-
- /**
- * Constructs a new <code>AbstractTestContext</code> instance.
- * @param name the name of the test case.
- */
- public AbstractTestContext(String name) {
- try{
- if(System.getProperty("mode")!=null){
- setMode(System.getProperty("mode") );
- }else
- {
- setMode("local");
- }
- }catch(Exception ex){
- setMode("local");
- }
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#setUp()
- */
- protected void setUp() throws Exception {
- InputStream input=null;
- File directory = new File("src/test/resources");
- if (!directory.isDirectory()) {
- return;
- }
- File file = new File(directory, getClass().getSimpleName() + ".properties");
- if (!file.canRead()) {
- return;
- }
- setTestPropFile(file);
- try{
- input=new FileInputStream(file);
- testProperties.load(input);
- }catch(IOException ex){
- ex.printStackTrace();
- throw new Exception("failed to load properties");
- }
-
-
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#tearDown()
- */
- @After
- protected void tearDown() throws Exception {
-
- }
-
- /**
- * Returns the testProperties.
- * @return the testProperties.
- */
-
- public Properties getTestProperties() {
- return testProperties;
- }
-
- /**
- * Sets the testProperties.
- * @param testProperties the testProperties.
- */
-
- public void setTestProperties(Properties testProperties) {
-
- this.testProperties = testProperties;
- }
- /**
- * Returns the testPropFile.
- * @return the testPropFile.
- */
-
- public File getTestPropFile() {
- return testPropFile;
- }
-
- /**
- * Sets the testPropFile.
- * @param testPropFile the testPropFile.
- */
-
- public void setTestPropFile(File testPropFile) {
-
- this.testPropFile = testPropFile;
- }
-
- /**
- * Skip Tests
- */
- public boolean skipTests(Object mode){
- if(mode.toString().equals("local")){
- return true;
- }else {
- return false;
- }
- }
-
- /**
- * Returns the mode.
- * @return the mode.
- */
-
- public Object getMode() {
- return mode;
- }
-
- /**
- * Sets the mode.
- * @param mode the mode.
- */
-
- public void setMode(Object mode) {
-
- this.mode = mode;
- }
-
- protected void assertNotNull() {}
- protected void assertNotNull(Object o) {}
-
- }
-
-
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GrokParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GrokParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GrokParserTest.java
index 9590d34..1a50dea 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GrokParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/GrokParserTest.java
@@ -19,10 +19,10 @@ package org.apache.metron.parsers;
import com.google.common.collect.MapDifference;
import com.google.common.collect.Maps;
-import junit.framework.Assert;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
+import org.junit.Assert;
import org.junit.Test;
import java.io.IOException;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MessageParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MessageParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MessageParserTest.java
index 1d2af78..9769baa 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MessageParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MessageParserTest.java
@@ -18,8 +18,8 @@
package org.apache.metron.parsers;
-import junit.framework.Assert;
import org.apache.metron.parsers.interfaces.MessageParser;
+import org.junit.Assert;
import org.junit.Test;
import java.util.ArrayList;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/asa/GrokAsaParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/asa/GrokAsaParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/asa/GrokAsaParserTest.java
deleted file mode 100644
index 9ed0b54..0000000
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/asa/GrokAsaParserTest.java
+++ /dev/null
@@ -1,169 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.parsers.asa;
-
-import java.util.Iterator;
-import java.util.Map;
-
-import org.apache.metron.parsers.sourcefire.BasicSourcefireParser;
-import org.json.simple.JSONObject;
-import org.json.simple.parser.JSONParser;
-import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractConfigTest;
-import org.junit.Assert;
-
-
-/**
- * <ul>
- * <li>Title: </li>
- * <li>Description: </li>
- * <li>Created: Feb 17, 2015 by: </li>
- * </ul>
- * @author $Author: $
- * @version $Revision: 1.1 $
- */
-public class GrokAsaParserTest extends AbstractConfigTest{
- /**
- * The grokAsaStrings.
- */
- private static String[] grokAsaStrings=null;
-
- /**
- * The grokAsaParser.
- */
-
- private GrokAsaParser grokAsaParser=null;
-
- /**
- * Constructs a new <code>GrokAsaParserTest</code> instance.
- * @throws Exception
- */
-
- public GrokAsaParserTest() throws Exception {
- super();
-
- }
- /**
- * @throws java.lang.Exception
- */
- public static void setUpBeforeClass() throws Exception {
- }
-
- /**
- * @throws java.lang.Exception
- */
- public static void tearDownAfterClass() throws Exception {
- setGrokAsaStrings(null);
- }
-
- /*
- * (non-Javadoc)
- * @see junit.framework.TestCase#setUp()
- */
- @Override
- public void setUp() throws Exception {
- super.setUp("org.apache.metron.parsers.asa.GrokAsaParserTest");
- setGrokAsaStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- grokAsaParser = new GrokAsaParser();
- }
-
- /**
- *
- *
- * @throws java.lang.Exception
- */
- @Override
- public void tearDown() throws Exception {
- grokAsaParser = null;
- }
-
- /**
- * Test method for {@link BasicSourcefireParser#parse(byte[])}.
- */
- @SuppressWarnings({ "rawtypes" })
- public void testParse() {
-
- for (String grokAsaString : getGrokAsaStrings()) {
- JSONObject parsed = grokAsaParser.parse(grokAsaString.getBytes()).get(0);
- Assert.assertNotNull(parsed);
-
- System.out.println(parsed);
- JSONParser parser = new JSONParser();
-
- Map json=null;
- try {
- json = (Map) parser.parse(parsed.toJSONString());
- } catch (ParseException e) {
- e.printStackTrace();
- }
- //Ensure JSON returned is not null/empty
- Assert.assertNotNull(json);
-
- Iterator iter = json.entrySet().iterator();
-
-
- while (iter.hasNext()) {
- Map.Entry entry = (Map.Entry) iter.next();
- Assert.assertNotNull(entry);
-
- String key = (String) entry.getKey();
- Assert.assertNotNull(key);
-
- String value = (String) json.get("CISCO_TAGGED_SYSLOG").toString();
- Assert.assertNotNull(value);
- }
- }
- }
-
- /**
- * Returns GrokAsa Input String
- */
- public static String[] getGrokAsaStrings() {
- return grokAsaStrings;
- }
-
-
- /**
- * Sets GrokAsa Input String
- */
- public static void setGrokAsaStrings(String[] strings) {
- GrokAsaParserTest.grokAsaStrings = strings;
- }
-
- /**
- * Returns the grokAsaParser.
- * @return the grokAsaParser.
- */
-
- public GrokAsaParser getGrokAsaParser() {
- return grokAsaParser;
- }
-
-
- /**
- * Sets the grokAsaParser.
- * @param grokAsaParser the grokAsaParser.
- */
-
- public void setGrokAsaParser(GrokAsaParser grokAsaParser) {
-
- this.grokAsaParser = grokAsaParser;
- }
-
- }
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
index 7aefe88..7b09746 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
@@ -17,7 +17,6 @@
*/
package org.apache.metron.parsers.bro;
-import junit.framework.TestCase;
import org.apache.commons.lang3.tuple.Pair;
import org.apache.log4j.Level;
import org.apache.metron.test.utils.UnitTestHelper;
@@ -25,32 +24,32 @@ import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
+import org.junit.AfterClass;
import org.junit.Assert;
import java.util.Map;
+import org.junit.BeforeClass;
+import org.junit.Test;
-public class BasicBroParserTest extends TestCase {
+public class BasicBroParserTest {
+ private BasicBroParser broParser = new BasicBroParser();
+ private JSONParser jsonParser = new JSONParser();
- /**
- * The parser.
- */
- private BasicBroParser broParser = null;
- private JSONParser jsonParser = null;
+ @BeforeClass
+ public static void setup() {
+ UnitTestHelper.setLog4jLevel(BasicBroParser.class, Level.FATAL);
+ }
- /**
- * Constructs a new <code>BasicBroParserTest</code> instance.
- *
- * @throws Exception
- */
- public BasicBroParserTest() throws Exception {
- broParser = new BasicBroParser();
- jsonParser = new JSONParser();
+ @AfterClass
+ public static void teardown() {
+ UnitTestHelper.setLog4jLevel(BasicBroParser.class, Level.ERROR);
}
/**
* This test is included as a gut-check about our formatting expectations using the Java JDK
* https://docs.oracle.com/javase/tutorial/i18n/format/decimalFormat.html
*/
+ @Test
public void testDecimalFormatAssumptions() {
Pair[] pairs = {
Pair.of(12345678d, "12345678.0"),
@@ -63,10 +62,11 @@ public class BasicBroParserTest extends TestCase {
Pair.of(12345678.111111d, "12345678.111111")
};
for (Pair pair : pairs) {
- assertEquals("Format did not match", pair.getRight(), BasicBroParser.DECIMAL_FORMAT.get().format(pair.getLeft()));
+ Assert.assertEquals("Format did not match", pair.getRight(), BasicBroParser.DECIMAL_FORMAT.get().format(pair.getLeft()));
}
}
+ @Test
public void testUnwrappedBroMessage() throws ParseException {
String rawMessage = "{\"timestamp\":1449511228.474,\"uid\":\"CFgSLp4HgsGqXnNjZi\",\"source_ip\":\"104.130.172.191\",\"source_port\":33893,\"dest_ip\":\"69.20.0.164\",\"dest_port\":53,\"proto\":\"udp\",\"trans_id\":3514,\"rcode\":3,\"rcode_name\":\"NXDOMAIN\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":false,\"Z\":0,\"rejected\":false,\"sensor\":\"cloudbro\",\"type\":\"dns\"}";
@@ -92,6 +92,7 @@ public class BasicBroParserTest extends TestCase {
}
@SuppressWarnings("rawtypes")
+ @Test
public void testHttpBroMessage() throws ParseException {
String rawMessage = "{\"http\":{\"ts\":1402307733.473,\"uid\":\"CTo78A11g7CYbbOHvj\",\"id.orig_h\":\"192.249.113.37\",\"id.orig_p\":58808,\"id.resp_h\":\"72.163.4.161\",\"id.resp_p\":80,\"trans_depth\":1,\"method\":\"GET\",\"host\":\"www.cisco.com\",\"uri\":\"/\",\"user_agent\":\"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\",\"request_body_len\":0,\"response_body_len\":25523,\"status_code\":200,\"status_msg\":\"OK\",\"tags\":[],\"resp_fuids\":[\"FJDyMC15lxUn5ngPfd\"],\"resp_mime_types\":[\"text/html\"]}}";
@@ -121,6 +122,7 @@ public class BasicBroParserTest extends TestCase {
* more compactly as 123.0
*/
@SuppressWarnings("rawtypes")
+ @Test
public void testHttpBroMessageWithZeroDecimalTruncation() throws ParseException {
{
String rawMessage = "{\"http\": {\"ts\":1467657279,\"uid\":\"CMYLzP3PKiwZAgBa51\",\"id.orig_h\":\"192.168.138.158\",\"id.orig_p\":49206,\"id.resp_h\":\"95.163.121.204\"," +
@@ -177,6 +179,7 @@ public class BasicBroParserTest extends TestCase {
}
@SuppressWarnings("rawtypes")
+ @Test
public void testHttpDecimalBroMessage() throws ParseException {
String rawMessage = "{\"http\":{\"ts\":1457149494.166991,\"uid\":\"CTo78A11g7CYbbOHvj\",\"id.orig_h\":\"192.249.113.37\",\"id.orig_p\":58808,\"id.resp_h\":\"72.163.4.161\",\"id.resp_p\":80,\"trans_depth\":1,\"method\":\"GET\",\"host\":\"www.cisco.com\",\"uri\":\"/\",\"user_agent\":\"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\",\"request_body_len\":0,\"response_body_len\":25523,\"status_code\":200,\"status_msg\":\"OK\",\"tags\":[],\"resp_fuids\":[\"FJDyMC15lxUn5ngPfd\"],\"resp_mime_types\":[\"text/html\"]}}";
Map rawMessageMap = (Map) jsonParser.parse(rawMessage);
@@ -201,6 +204,7 @@ public class BasicBroParserTest extends TestCase {
@SuppressWarnings("rawtypes")
+ @Test
public void testDnsBroMessage() throws ParseException {
String rawMessage = "{\"dns\":{\"ts\":1402308259.609,\"uid\":\"CuJT272SKaJSuqO0Ia\",\"id.orig_h\":\"10.122.196.204\",\"id.orig_p\":33976,\"id.resp_h\":\"144.254.71.184\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":62418,\"query\":\"www.cisco.com\",\"qclass\":1,\"qclass_name\":\"C_INTERNET\",\"qtype\":28,\"qtype_name\":\"AAAA\",\"rcode\":0,\"rcode_name\":\"NOERROR\",\"AA\":true,\"TC\":false,\"RD\":true,\"RA\":true,\"Z\":0,\"answers\":[\"www.cisco.com.akadns.net\",\"origin-www.cisco.com\",\"2001:420:1201:2::a\"],\"TTLs\":[3600.0,289.0,14.0],\"rejected\":false}}";
@@ -223,6 +227,7 @@ public class BasicBroParserTest extends TestCase {
}
@SuppressWarnings("rawtypes")
+ @Test
public void testFilesBroMessage() throws ParseException {
String rawMessage = "{\"files\":{\"analyzers\": [\"X509\",\"MD5\",\"SHA1\"],\"conn_uids\":[\"C4tygJ3qxJBEJEBCeh\"],\"depth\": 0,\"duration\": 0.0,\"fuid\":\"FZEBC33VySG0nHSoO9\",\"is_orig\": false,\"local_orig\": false,\"md5\": \"eba37166385e3ef42464ed9752e99f1b\",\"missing_bytes\": 0,\"overflow_bytes\": 0,\"rx_hosts\": [\"10.220.15.205\"],\"seen_bytes\": 1136,\"sha1\": \"73e42686657aece354fbf685712361658f2f4357\",\"source\": \"SSL\",\"timedout\": false,\"ts\": 1425845251.334,\"tx_hosts\": [\"68.171.237.7\"]}}";
@@ -244,6 +249,7 @@ public class BasicBroParserTest extends TestCase {
}
@SuppressWarnings("rawtypes")
+ @Test
public void testProtocolKeyCleanedUp() throws ParseException {
String rawMessage = "{\"ht*tp\":{\"ts\":1402307733.473,\"uid\":\"CTo78A11g7CYbbOHvj\",\"id.orig_h\":\"192.249.113.37\",\"id.orig_p\":58808,\"id.resp_h\":\"72.163.4.161\",\"id.resp_p\":80,\"trans_depth\":1,\"method\":\"GET\",\"host\":\"www.cisco.com\",\"uri\":\"/\",\"user_agent\":\"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3\",\"request_body_len\":0,\"response_body_len\":25523,\"status_code\":200,\"status_msg\":\"OK\",\"tags\":[],\"resp_fuids\":[\"FJDyMC15lxUn5ngPfd\"],\"resp_mime_types\":[\"text/html\"]}}";
@@ -259,23 +265,13 @@ public class BasicBroParserTest extends TestCase {
Assert.assertTrue(broJson.get("original_string").toString().startsWith("HTTP"));
}
- public void testBadMessage() throws ParseException{
- UnitTestHelper.setLog4jLevel(BasicBroParser.class, Level.FATAL);
- try {
- broParser.parse("{ \"foo\" : \"bar\"}".getBytes());
- Assert.fail("Should have marked this as a bad message.");
- }
- catch(IllegalStateException ise) {
-
- }
- //non json
- try {
- broParser.parse("foo bar".getBytes());
- Assert.fail("Should have marked this as a bad message.");
- }
- catch(IllegalStateException ise) {
+ @Test(expected=IllegalStateException.class)
+ public void testBadMessage() throws ParseException {
+ broParser.parse("{ \"foo\" : \"bar\"}".getBytes());
+ }
- }
- UnitTestHelper.setLog4jLevel(BasicBroParser.class, Level.ERROR);
+ @Test(expected=IllegalStateException.class)
+ public void testBadMessageNonJson() {
+ broParser.parse("foo bar".getBytes());
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BroParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BroParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BroParserTest.java
deleted file mode 100644
index 8d81e14..0000000
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BroParserTest.java
+++ /dev/null
@@ -1,164 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.parsers.bro;
-
-
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.Map;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-
-import org.json.simple.JSONObject;
-import org.json.simple.parser.JSONParser;
-import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractConfigTest;
-import org.junit.Assert;
-
-/**
- * <ul>
- * <li>Title: Test For BroParser</li>
- * <li>Description: </li>
- * <li>Created: July 8, 2014</li>
- * </ul>
- * @version $Revision: 1.0 $
- */
-
- /**
- * <ul>
- * <li>Title: </li>
- * <li>Description: </li>
- * <li>Created: Feb 20, 2015 </li>
- * </ul>
- * @author $Author: $
- * @version $Revision: 1.1 $
- */
-public class BroParserTest extends AbstractConfigTest {
-
-
- /**
- * The inputStrings.
- */
- private static String[] inputStrings;
-
- /**
- * The parser.
- */
- private BasicBroParser parser=null;
-
- /**
- * Constructs a new <code>BroParserTest</code> instance.
- * @throws Exception
- */
- public BroParserTest() throws Exception {
- super();
- }
-
-
- /**
- * @throws java.lang.Exception
- */
- public static void setUpBeforeClass() throws Exception {
- }
-
- /**
- * @throws java.lang.Exception
- */
- public static void tearDownAfterClass() throws Exception {
- }
-
- /**
- * @throws java.lang.Exception
- */
- @Override
- public void setUp() throws Exception {
- super.setUp("org.apache.metron.parsers.bro.BroParserTest");
- setInputStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- parser = new BasicBroParser();
- }
-
- /**
- * @throws ParseException
- * Tests for Parse Method
- * Parses Static json String and checks if any spl chars are present in parsed string.
- */
- @SuppressWarnings({ "unused", "rawtypes" })
- public void testParse() throws ParseException {
-
- for (String inputString : getInputStrings()) {
- JSONObject cleanJson = parser.parse(inputString.getBytes()).get(0);
- Assert.assertNotNull(cleanJson);
- System.out.println(cleanJson);
-
- Pattern p = Pattern.compile("[^\\._a-z0-9 ]",
- Pattern.CASE_INSENSITIVE);
-
- JSONParser parser = new JSONParser();
-
- Map json = (Map) cleanJson;
- Map output = new HashMap();
- Iterator iter = json.entrySet().iterator();
-
- while (iter.hasNext()) {
- Map.Entry entry = (Map.Entry) iter.next();
- String key = (String) entry.getKey();
-
- Matcher m = p.matcher(key);
- boolean b = m.find();
- // Test False
- Assert.assertFalse(b);
- }
- }
-
- }
-
- /**
- * Returns Input String
- */
- public static String[] getInputStrings() {
- return inputStrings;
- }
-
- /**
- * Sets SourceFire Input String
- */
- public static void setInputStrings(String[] strings) {
- BroParserTest.inputStrings = strings;
- }
-
- /**
- * Returns the parser.
- * @return the parser.
- */
-
- public BasicBroParser getParser() {
- return parser;
- }
-
-
- /**
- * Sets the parser.
- * @param parser the parser.
- */
-
- public void setParser(BasicBroParser parser) {
-
- this.parser = parser;
- }
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
index 88509dd..a1fe2e8 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/cef/CEFParserTest.java
@@ -31,6 +31,7 @@ import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
import org.junit.Assert;
+import org.junit.Before;
import org.junit.Test;
import com.fasterxml.jackson.core.JsonProcessingException;
@@ -42,14 +43,12 @@ import com.github.fge.jsonschema.main.JsonSchemaFactory;
import com.github.fge.jsonschema.main.JsonValidator;
import com.google.common.io.Resources;
-import junit.framework.TestCase;
-
-public class CEFParserTest extends TestCase {
+public class CEFParserTest {
private static final Charset UTF_8 = Charset.forName("utf-8");
private CEFParser parser;
- @Override
+ @Before
public void setUp() {
parser = new CEFParser();
parser.init();
@@ -58,48 +57,52 @@ public class CEFParserTest extends TestCase {
@Test
public void testInvalid() {
List<JSONObject> obj = parse("test test test nonsense\n");
- assertEquals(0, obj.size());
+ Assert.assertEquals(0, obj.size());
}
@Test
public void testEscaping() {
for (JSONObject obj : parse(
"Sep 19 08:26:10 host CEF:0|security|threatmanager|1.0|100|detected a \\ in packet|10|src=10.0.0.1 act=blocked a \\ dst=1.1.1.1")) {
- assertEquals("10.0.0.1", obj.get("ip_src_addr"));
- assertEquals("blocked a \\", obj.get("deviceAction"));
- assertEquals("1.1.1.1", obj.get("ip_dst_addr"));
+ Assert.assertEquals("10.0.0.1", obj.get("ip_src_addr"));
+ Assert.assertEquals("blocked a \\", obj.get("deviceAction"));
+ Assert.assertEquals("1.1.1.1", obj.get("ip_dst_addr"));
}
}
+ @Test
public void testBasicHeader() {
for (JSONObject obj : parse(
"CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
- assertEquals("Security", obj.get("DeviceVendor"));
- assertEquals("threatmanager", obj.get("DeviceProduct"));
- assertEquals("1.0", obj.get("DeviceVersion"));
- assertEquals("100", obj.get("DeviceEvent"));
- assertEquals("worm successfully stopped", obj.get("Name"));
- assertEquals(10, obj.get("Severity"));
+ Assert.assertEquals("Security", obj.get("DeviceVendor"));
+ Assert.assertEquals("threatmanager", obj.get("DeviceProduct"));
+ Assert.assertEquals("1.0", obj.get("DeviceVersion"));
+ Assert.assertEquals("100", obj.get("DeviceEvent"));
+ Assert.assertEquals("worm successfully stopped", obj.get("Name"));
+ Assert.assertEquals(10, obj.get("Severity"));
}
}
+ @Test
public void testBasicExtensions() {
for (JSONObject obj : parse(
"CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
- assertEquals("10.0.0.1", obj.get("ip_src_addr"));
- assertEquals("2.1.2.2", obj.get("ip_dst_addr"));
- assertEquals(1232, obj.get("ip_src_port"));
+ Assert.assertEquals("10.0.0.1", obj.get("ip_src_addr"));
+ Assert.assertEquals("2.1.2.2", obj.get("ip_dst_addr"));
+ Assert.assertEquals(1232, obj.get("ip_src_port"));
}
}
+ @Test
public void testCustomLabelWithSpace() {
for (JSONObject obj : parse(
"CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232 custom=Text with space customLabel=Label with space")) {
- assertEquals(true, obj.containsKey("Label with space"));
- assertEquals("Text with space", obj.get("Label with space"));
+ Assert.assertEquals(true, obj.containsKey("Label with space"));
+ Assert.assertEquals("Text with space", obj.get("Label with space"));
}
}
+ @Test
public void testTimestampPriority() throws java.text.ParseException {
long correctTime = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
.getTime();
@@ -108,33 +111,34 @@ public class CEFParserTest extends TestCase {
for (JSONObject obj : parse(
"CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt=May 1 2016 09:29:11.356 -0400 dst=2.1.2.2 spt=1232")) {
- assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
- assertEquals(correctTime, obj.get("timestamp"));
+ Assert.assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ Assert.assertEquals(correctTime, obj.get("timestamp"));
}
for (JSONObject obj : parse(
"2016-06-01T09:29:11.356-04:00 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt=May 1 2016 09:29:11.356 -0400 dst=2.1.2.2 spt=1232")) {
- assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
- assertEquals(correctTime, obj.get("timestamp"));
+ Assert.assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ Assert.assertEquals(correctTime, obj.get("timestamp"));
}
for (JSONObject obj : parse(
"2016-05-01T09:29:11.356-04:00 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
- assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
- assertEquals(correctTime, obj.get("timestamp"));
+ Assert.assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ Assert.assertEquals(correctTime, obj.get("timestamp"));
}
for (JSONObject obj : parse(
"CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232")) {
- assertNotNull(obj.get("timestamp"));
+ Assert.assertNotNull(obj.get("timestamp"));
}
}
+ @Test
public void testRtValueAsEpochTimestamp() throws java.text.ParseException {
long correctTime = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSz").parse("2016-05-01T09:29:11.356-0400")
.getTime();
for (JSONObject obj : parse("CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt="
+ String.valueOf(correctTime) + " dst=2.1.2.2 spt=1232")) {
- assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
- assertEquals(correctTime, obj.get("timestamp"));
+ Assert.assertEquals(new Date(correctTime), new Date((long) obj.get("timestamp")));
+ Assert.assertEquals(correctTime, obj.get("timestamp"));
}
}
@@ -142,11 +146,12 @@ public class CEFParserTest extends TestCase {
SimpleDateFormat sdf = new SimpleDateFormat("MMM dd HH:mm:ss.SSS");
for (JSONObject obj : parse("CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 rt="
+ sdf.format(input.getTime()) + " dst=2.1.2.2 spt=1232")) {
- assertEquals(expected.getTimeInMillis(), obj.get("timestamp"));
- assertEquals(expected.getTime(), new Date((long) obj.get("timestamp")));
+ Assert.assertEquals(expected.getTimeInMillis(), obj.get("timestamp"));
+ Assert.assertEquals(expected.getTime(), new Date((long) obj.get("timestamp")));
}
}
+ @Test
public void testMissingYearFromDate() throws java.text.ParseException {
Calendar current = Calendar.getInstance();
Calendar correct = Calendar.getInstance();
@@ -156,6 +161,7 @@ public class CEFParserTest extends TestCase {
runMissingYear(correct, current);
}
+ @Test
public void testFourDayFutureBecomesPast() {
Calendar current = Calendar.getInstance();
Calendar correct = Calendar.getInstance();
@@ -168,17 +174,20 @@ public class CEFParserTest extends TestCase {
runMissingYear(correct, current);
}
+ @Test
public void testCEFParserAdallom() throws Exception {
runTest("adallom", Resources.readLines(Resources.getResource(getClass(), "adallom.cef"), UTF_8),
Resources.toString(Resources.getResource(getClass(), "adallom.schema"), UTF_8));
}
+ @Test
public void testCEFParserCyberArk() throws Exception {
runTest("cyberark", Resources.readLines(Resources.getResource(getClass(), "cyberark.cef"), UTF_8),
Resources.toString(Resources.getResource(getClass(), "cyberark.schema"), UTF_8),
Resources.toString(Resources.getResource(getClass(), "cyberark.json"), UTF_8));
}
+ @Test
public void testCEFParserWAF() throws Exception {
URL waf_url = Resources.getResource(getClass(), "waf.cef");
runTest("waf", Resources.readLines(waf_url, UTF_8),
@@ -199,11 +208,10 @@ public class CEFParserTest extends TestCase {
private void runTest(String name, List<String> lines, String schema, String targetJson) throws Exception {
for (String inputString : lines) {
JSONObject parsed = parse(inputString).get(0);
- assertNotNull(parsed);
- assertNotNull(parsed.get("timestamp"));
- assertTrue((long) parsed.get("timestamp") > 0);
+ Assert.assertNotNull(parsed);
+ Assert.assertNotNull(parsed.get("timestamp"));
+ Assert.assertTrue((long) parsed.get("timestamp") > 0);
- System.out.println(parsed);
JSONParser parser = new JSONParser();
Map<?, ?> json = null;
@@ -249,14 +257,14 @@ public class CEFParserTest extends TestCase {
List<JSONObject> parse = parse(sample);
JSONObject obj = parse.get(0);
- assertEquals("TestVendor", obj.get("DeviceVendor"));
- assertEquals(1423441663000L, obj.get("timestamp"));
- assertEquals("9223372036854775807", obj.get("Test Long"));
- assertEquals(obj.get("Test FP Number"), String.valueOf(1.234F));
- assertEquals("00:00:0c:07:ac:00", obj.get("smac"));
- assertEquals("2001:cdba::3257:9652", obj.get("Test IPv6"));
- assertEquals("test test test chocolate", obj.get("Test String"));
- assertEquals("123.123.123.123", obj.get("destinationTranslatedAddress"));
+ Assert.assertEquals("TestVendor", obj.get("DeviceVendor"));
+ Assert.assertEquals(1423441663000L, obj.get("timestamp"));
+ Assert.assertEquals("9223372036854775807", obj.get("Test Long"));
+ Assert.assertEquals(obj.get("Test FP Number"), String.valueOf(1.234F));
+ Assert.assertEquals("00:00:0c:07:ac:00", obj.get("smac"));
+ Assert.assertEquals("2001:cdba::3257:9652", obj.get("Test IPv6"));
+ Assert.assertEquals("test test test chocolate", obj.get("Test String"));
+ Assert.assertEquals("123.123.123.123", obj.get("destinationTranslatedAddress"));
JsonNode inner = new ObjectMapper().readTree((String) obj.get("JSON payload"));
Assert.assertEquals("chocolate!", inner.get("test_test_test").asText());
@@ -270,14 +278,13 @@ public class CEFParserTest extends TestCase {
JsonValidator v = factory.getValidator();
ProcessingReport report = v.validate(s, d);
- System.out.println(report);
return report.toString().contains("success");
}
private List<JSONObject> parse(String string) {
List<JSONObject> parse = parser.parse(string.getBytes(Charset.forName("utf-8")));
- assertNotNull(parse);
+ Assert.assertNotNull(parse);
return parse;
}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/fireeye/BasicFireEyeParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/fireeye/BasicFireEyeParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/fireeye/BasicFireEyeParserTest.java
index 0368b5e..69a6dbd 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/fireeye/BasicFireEyeParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/fireeye/BasicFireEyeParserTest.java
@@ -17,147 +17,44 @@
*/
package org.apache.metron.parsers.fireeye;
-
-
-import java.util.Iterator;
import java.util.Map;
-
+import java.util.Map.Entry;
+import org.apache.metron.parsers.AbstractParserConfigTest;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractConfigTest;
import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
-/**
- * <ul>
- * <li>Title: Test For SourceFireParser</li>
- * <li>Description: </li>
- * <li>Created: July 8, 2014</li>
- * </ul>
- * @version $Revision: 1.0 $
- */
-public class BasicFireEyeParserTest extends AbstractConfigTest
-{
- /**
- * The inputStrings.
- */
- private static String[] inputStrings;
-
- /**
- * The parser.
- */
- private BasicFireEyeParser parser=null;
-
-
- /**
- * Constructs a new <code>BasicFireEyeParserTest</code> instance.
- * @throws Exception
- */
- public BasicFireEyeParserTest() throws Exception {
- super();
- }
-
+public class BasicFireEyeParserTest extends AbstractParserConfigTest {
- /**
- * @throws java.lang.Exception
- */
- public static void setUpBeforeClass() throws Exception {
- }
+ @Before
+ public void setUp() throws Exception {
+ inputStrings = super.readTestDataFromFile("src/test/resources/logData/FireEyeParserTest.txt");
+ parser = new BasicFireEyeParser();
+ }
- /**
- * @throws java.lang.Exception
- */
- public static void tearDownAfterClass() throws Exception {
- }
+ @SuppressWarnings({"rawtypes"})
+ @Test
+ public void testParse() throws ParseException {
+ for (String inputString : inputStrings) {
+ JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
+ Assert.assertNotNull(parsed);
- /**
- * @throws java.lang.Exception
- */
- @Override
- public void setUp() throws Exception {
- super.setUp("org.apache.metron.parsers.fireeye.BasicFireEyeParserTest");
- setInputStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- parser = new BasicFireEyeParser();
- }
+ JSONParser parser = new JSONParser();
- /**
- *
- *
- * @throws java.lang.Exception
- */
- @Override
- public void tearDown() throws Exception {
- parser = null;
- setInputStrings(null);
- }
+ Map json = (Map) parser.parse(parsed.toJSONString());
- /**
- * Test method for
- *
- *
- *
- *
- *
- * {@link BasicFireEyeParser#parse(byte[])}.
- */
- @SuppressWarnings({ "rawtypes"})
- public void testParse() {
- for (String inputString : getInputStrings()) {
- JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
- Assert.assertNotNull(parsed);
-
- JSONParser parser = new JSONParser();
+ Assert.assertNotNull(json);
+ Assert.assertFalse(json.isEmpty());
- Map json=null;
- try {
- json = (Map) parser.parse(parsed.toJSONString());
- } catch (ParseException e) {
- e.printStackTrace();
- }
- Iterator iter = json.entrySet().iterator();
-
- Assert.assertNotNull(json);
- Assert.assertFalse(json.isEmpty());
-
-
- while (iter.hasNext()) {
- Map.Entry entry = (Map.Entry) iter.next();
- String key = (String) entry.getKey();
- String value = (String) json.get(key).toString();
- Assert.assertNotNull(value);
- }
- }
- }
-
- /**
- * Returns Input String
- */
- public static String[] getInputStrings() {
- return inputStrings;
- }
-
- /**
- * Sets SourceFire Input String
- */
- public static void setInputStrings(String[] strings) {
- BasicFireEyeParserTest.inputStrings = strings;
- }
-
- /**
- * Returns the parser.
- * @return the parser.
- */
- public BasicFireEyeParser getParser() {
- return parser;
+ for (Object o : json.entrySet()) {
+ Entry entry = (Entry) o;
+ String key = (String) entry.getKey();
+ String value = json.get(key).toString();
+ Assert.assertNotNull(value);
+ }
}
-
- /**
- * Sets the parser.
- * @param parser the parser.
- */
- public void setParser(BasicFireEyeParser parser) {
-
- this.parser = parser;
- }
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
index fe6475d..b20445e 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/ParserIntegrationTest.java
@@ -18,7 +18,6 @@
package org.apache.metron.parsers.integration;
import com.google.common.base.Function;
-import junit.framework.Assert;
import org.apache.metron.TestConstants;
import org.apache.metron.common.Constants;
import org.apache.metron.enrichment.integration.components.ConfigUploadComponent;
@@ -31,6 +30,7 @@ import org.apache.metron.integration.utils.TestUtils;
import org.apache.metron.parsers.integration.components.ParserTopologyComponent;
import org.apache.metron.test.TestDataType;
import org.apache.metron.test.utils.SampleDataUtils;
+import org.junit.Assert;
import org.junit.Test;
import javax.annotation.Nullable;
http://git-wip-us.apache.org/repos/asf/metron/blob/5b72da7b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/ise/BasicIseParserTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/ise/BasicIseParserTest.java b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/ise/BasicIseParserTest.java
index c448c70..050a2d2 100644
--- a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/ise/BasicIseParserTest.java
+++ b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/ise/BasicIseParserTest.java
@@ -17,156 +17,40 @@
*/
package org.apache.metron.parsers.ise;
+import com.github.fge.jsonschema.core.exceptions.ProcessingException;
import java.io.IOException;
import java.net.URL;
import java.util.Map;
-
+import org.apache.metron.parsers.AbstractParserConfigTest;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
-import org.json.simple.parser.ParseException;
-
-import org.apache.metron.parsers.AbstractSchemaTest;
import org.junit.Assert;
-
-
-/**
- * <ul>
- * <li>Title: Basic ISE Parser</li>
- * <li>Description: Junit Test Case for BasicISE Parser</li>
- * <li>Created: AUG 25, 2014</li>
- * </ul>
- *
- * @version $Revision: 1.1 $
- */
-
-public class BasicIseParserTest extends AbstractSchemaTest {
- /**
- * The inputStrings.
- */
- private static String[] inputStrings;
-
- /**
- * The parser.
- */
- private static BasicIseParser parser = null;
-
-
- /**
- * Constructs a new <code>BasicIseParserTest</code> instance.
- *
- * @param name
- */
-
- public BasicIseParserTest(String name) {
- super(name);
- }
-
- /**
- *
- * @throws java.lang.Exception
- */
- protected static void setUpBeforeClass() throws Exception {
- }
-
- /**
- *
- * @throws java.lang.Exception
- */
- protected static void tearDownAfterClass() throws Exception {
- }
-
- /*
- * (non-Javadoc)
- *
- * @see junit.framework.TestCase#setUp()
- */
-
- @Override
- protected void setUp() throws Exception {
- super.setUp("org.apache.metron.parsers.lancope.BasicLancopeParserTest");
- setInputStrings(super.readTestDataFromFile(this.getConfig().getString("logFile")));
- BasicIseParserTest.setIseParser(new BasicIseParser());
-
- URL schema_url = getClass().getClassLoader().getResource(
- "TestSchemas/IseSchema.json");
- super.setSchemaJsonString(super.readSchemaFromFile(schema_url));
- }
-
- /*
- * (non-Javadoc)
- *
- * @see junit.framework.TestCase#tearDown()
- */
-
- @Override
- protected void tearDown() throws Exception {
- super.tearDown();
- }
-
- /**
- * Test method for
- * {@link BasicIseParser#parse(byte[])}.
- *
- * @throws IOException
- * @throws Exception
- */
- public void testParse() throws ParseException, IOException, Exception {
- for (String inputString : getInputStrings()) {
- JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
- assertNotNull(parsed);
-
- System.out.println(parsed);
- JSONParser parser = new JSONParser();
-
- Map<?, ?> json=null;
- try {
- json = (Map<?, ?>) parser.parse(parsed.toJSONString());
- Assert.assertEquals(true, validateJsonData(super.getSchemaJsonString(), json.toString()));
- } catch (ParseException e) {
- e.printStackTrace();
- }
- }
- }
-
- /**
- * Returns the iseParser.
- *
- * @return the iseParser.
- */
-
- public BasicIseParser getIseParser() {
- return parser;
- }
-
- /**
- * Sets the iseParser.
- *
- * @param parser
- */
-
-
- public static void setIseParser(BasicIseParser parser) {
-
- BasicIseParserTest.parser = parser;
- }
- /**
- * Returns the inputStrings.
- * @return the inputStrings.
- */
-
- public static String[] getInputStrings() {
- return inputStrings;
- }
-
- /**
- * Sets the inputStrings.
- * @param inputStrings the inputStrings.
- */
-
- public static void setInputStrings(String[] inputStrings) {
- BasicIseParserTest.inputStrings = inputStrings;
- }
-
-
-
+import org.junit.Before;
+import org.junit.Test;
+
+public class BasicIseParserTest extends AbstractParserConfigTest {
+
+ @Before
+ public void setUp() throws Exception {
+ inputStrings = super.readTestDataFromFile("src/test/resources/logData/IseParserTest.txt");
+ parser = new BasicIseParser();
+
+ URL schema_url = getClass().getClassLoader().getResource(
+ "TestSchemas/IseSchema.json");
+ super.setSchemaJsonString(super.readSchemaFromFile(schema_url));
+ }
+
+ @Test
+ public void testParse()
+ throws org.json.simple.parser.ParseException, IOException, ProcessingException {
+ for (String inputString : inputStrings) {
+ JSONObject parsed = parser.parse(inputString.getBytes()).get(0);
+ Assert.assertNotNull(parsed);
+
+ JSONParser parser = new JSONParser();
+
+ Map<?, ?> json = (Map<?, ?>) parser.parse(parsed.toJSONString());
+ Assert.assertTrue(validateJsonData(getSchemaJsonString(), json.toString()));
+ }
+ }
}
[04/18] metron git commit: METRON-978 Management UI init script uses
hard-coded METRON_HOME (merrimanr) closes apache/metron#604
Posted by ma...@apache.org.
METRON-978 Management UI init script uses hard-coded METRON_HOME (merrimanr) closes apache/metron#604
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/43e529fe
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/43e529fe
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/43e529fe
Branch: refs/heads/Metron_0.4.0
Commit: 43e529fe003b84c585555be9f25a4ff49d769132
Parents: d5abcf4
Author: merrimanr <me...@gmail.com>
Authored: Mon Jun 5 16:48:49 2017 -0500
Committer: merrimanr <me...@apache.org>
Committed: Mon Jun 5 16:48:49 2017 -0500
----------------------------------------------------------------------
metron-interface/metron-config/scripts/metron-management-ui | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/43e529fe/metron-interface/metron-config/scripts/metron-management-ui
----------------------------------------------------------------------
diff --git a/metron-interface/metron-config/scripts/metron-management-ui b/metron-interface/metron-config/scripts/metron-management-ui
index 32b4e9e..34c0cb6 100644
--- a/metron-interface/metron-config/scripts/metron-management-ui
+++ b/metron-interface/metron-config/scripts/metron-management-ui
@@ -33,6 +33,11 @@ METRON_LOG_DIR="/var/log/metron"
METRON_PID_DIR="/var/run/metron"
METRON_USER="metron"
METRON_GROUP="metron"
+METRON_SYSCONFIG="/etc/sysconfig/metron"
+if [ -f "$METRON_SYSCONFIG" ]; then
+ set -a
+ . "$METRON_SYSCONFIG"
+fi
PIDFILE="$METRON_PID_DIR/$NAME.pid"
LOCKFILE=/var/lock/subsys/$NAME
[03/18] metron git commit: METRON-979 markdown errors in site-book,
part 2 (JonZeolla via mattf-horton) closes mattf-horton/metron#7
closes apache/metron#605
Posted by ma...@apache.org.
METRON-979 markdown errors in site-book, part 2 (JonZeolla via mattf-horton) closes mattf-horton/metron#7 closes apache/metron#605
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/d5abcf42
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/d5abcf42
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/d5abcf42
Branch: refs/heads/Metron_0.4.0
Commit: d5abcf429dad3c284a5f2469c0fb99713844be02
Parents: 1e9acc7
Author: JonZeolla <ze...@gmail.com>
Authored: Mon Jun 5 14:17:46 2017 -0700
Committer: mattf <ma...@apache.org>
Committed: Mon Jun 5 14:17:46 2017 -0700
----------------------------------------------------------------------
Upgrading.md | 18 ++++-----
metron-deployment/Kerberos-manual-setup.md | 4 ++
metron-deployment/amazon-ec2/README.md | 53 ++++++++++++-------------
metron-platform/metron-parsers/README.md | 2 -
4 files changed, 39 insertions(+), 38 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/d5abcf42/Upgrading.md
----------------------------------------------------------------------
diff --git a/Upgrading.md b/Upgrading.md
index 3864ae3..6fb2486 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -2,6 +2,15 @@
This document constitutes a per-version listing of changes of
configuration which are non-backwards compatible.
+## 0.3.1 to 0.4.0
+
+### [METRON-671: Refactor existing Ansible deployment to use Ambari MPack](https://issues.apache.org/jira/browse/METRON-671)
+
+#### Description
+Since the Ansible Deployment uses the MPack, RPMs must be built prior to deployment. As a result,
+[Docker](https://www.docker.com/) is required to perform a Quick-Dev, Full-Dev or Ansible deployment.
+This effectively limits the build environment to Docker supported [platforms](https://docs.docker.com/engine/installation/#platform-support-matrix).
+
## 0.3.0 to 0.3.1
### [METRON-664: Make the index configuration per-writer with enabled/disabled](https://issues.apache.org/jira/browse/METRON-664)
@@ -151,12 +160,3 @@ When using Ansible to deploy the latest Metron bits to an existing installation,
#### Workaround
Set the `topology.classpath` property for storm in Ambari to `/etc/hbase/conf:/etc/hadoop/conf`
-
-## 0.3.1 to PLACEHOLDER
-
-### [METRON-671: Refactor existing Ansible deployment to use Ambari MPack](https://issues.apache.org/jira/browse/METRON-671)
-
-#### Description
-Since the Ansible Deployment uses the MPack, RPMs must be built prior to deployment. As a result,
-[Docker](https://www.docker.com/) is required to perform a Quick-Dev, Full-Dev or Ansible deployment.
-This effectively limits the build environment to Docker supported [platforms](https://docs.docker.com/engine/installation/#platform-support-matrix).
http://git-wip-us.apache.org/repos/asf/metron/blob/d5abcf42/metron-deployment/Kerberos-manual-setup.md
----------------------------------------------------------------------
diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md
index 4efebc6..cdfbdfb 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -381,7 +381,11 @@ Push Data
```
cat sample-bro.txt | ${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+<<<<<<< HEAD
--broker-list ${BROKERLIST} \
+=======
+ --broker-list ${BROKERLIST}
+>>>>>>> d550b9eff34d931aeae7151713c0e07f28719d4c
--security-protocol SASL_PLAINTEXT \
--topic bro
```
http://git-wip-us.apache.org/repos/asf/metron/blob/d5abcf42/metron-deployment/amazon-ec2/README.md
----------------------------------------------------------------------
diff --git a/metron-deployment/amazon-ec2/README.md b/metron-deployment/amazon-ec2/README.md
index b28efdd..7b8be93 100644
--- a/metron-deployment/amazon-ec2/README.md
+++ b/metron-deployment/amazon-ec2/README.md
@@ -158,33 +158,32 @@ The dashboard expects fields to be of a certain type. If the index templates ha
If you see this error, please report your findings by creating a JIRA or dropping an email to the Metron Users mailing list. Follow these steps to work around the problem.
-(1) Define which Elasticsearch host to interact with. Any Elasticsearch host should work.
-```
-export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
-```
-
-(2) Confirm the index templates are in fact missing.
-```
-curl -s -XGET $ES_HOST/_template
-```
-
-(3) Manually load the index templates.
-```
-cd metron-deployment
-curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
-curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
-curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
-```
-
-(4) Delete the existing indexes. Only a new index will use the templates defined in the previous step.
-
-```
-curl -s -XDELETE "$ES_HOST/yaf_index*"
-curl -s -XDELETE "$ES_HOST/bro_index*"
-curl -s -XDELETE "$ES_HOST/snort_index*"
-```
-
-(5) Open up Kibana and wait for the new indexes to be created. The dashboard should now work.
+1. Define which Elasticsearch host to interact with. Any Elasticsearch host should work.
+ ```
+ export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
+ ```
+
+1. Confirm the index templates are in fact missing.
+ ```
+ curl -s -XGET $ES_HOST/_template
+ ```
+
+1. Manually load the index templates.
+ ```
+ cd metron-deployment
+ curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
+ curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
+ curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
+ ```
+
+1. Delete the existing indexes. Only a new index will use the templates defined in the previous step.
+ ```
+ curl -s -XDELETE "$ES_HOST/yaf_index*"
+ curl -s -XDELETE "$ES_HOST/bro_index*"
+ curl -s -XDELETE "$ES_HOST/snort_index*"
+ ```
+
+1. Open up Kibana and wait for the new indexes to be created. The dashboard should now work.
### Error: 'No handler was ready to authenticate...Check your credentials'
http://git-wip-us.apache.org/repos/asf/metron/blob/d5abcf42/metron-platform/metron-parsers/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md
index 97e8e5f..e5d97e0 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -38,7 +38,6 @@ All Metron messages follow a specific format in order to ingest a message. If a
```
{"message" : message content}
-
```
Where appropriate there is also a standardization around the 5-tuple JSON fields. This is done so the topology correlation engine further down stream can correlate messages from different topologies by these fields. We are currently working on expanding the message standardization beyond these fields, but this feature is not yet availabe. The standard field names are as follows:
@@ -66,7 +65,6 @@ So putting it all together a typical Metron message with all 5-tuple fields pres
"original_string": xxx,
"additional-field 1": xxx,
}
-
}
```
[18/18] metron git commit: METRON-996 Performance improvement for ASA
parser closes apache/incubator-metron#617
Posted by ma...@apache.org.
METRON-996 Performance improvement for ASA parser closes apache/incubator-metron#617
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/b76bcd5f
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/b76bcd5f
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/b76bcd5f
Branch: refs/heads/Metron_0.4.0
Commit: b76bcd5f2405db122c2bf9d19128f9efb20b8458
Parents: de2c871
Author: Simon Elliston Ball <si...@simonellistonball.com>
Authored: Mon Jun 19 11:30:58 2017 -0400
Committer: cstella <ce...@gmail.com>
Committed: Mon Jun 19 11:30:58 2017 -0400
----------------------------------------------------------------------
.../metron/parsers/asa/BasicAsaParser.java | 366 ++++++++++---------
1 file changed, 191 insertions(+), 175 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/b76bcd5f/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
index ad1db83..daac141 100644
--- a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
+++ b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/asa/BasicAsaParser.java
@@ -33,187 +33,203 @@ import java.io.*;
import java.time.Clock;
import java.time.ZoneId;
import java.util.*;
+import java.util.Map.Entry;
public class BasicAsaParser extends BasicParser {
- protected static final Logger LOG = LoggerFactory.getLogger(BasicAsaParser.class);
-
- private Grok asaGrok;
- protected Clock deviceClock;
-
- private static final Map<String, String> patternMap = ImmutableMap.<String, String>builder()
- .put("ASA-2-106001", "CISCOFW106001")
- .put("ASA-2-106006", "CISCOFW106006_106007_106010")
- .put("ASA-2-106007", "CISCOFW106006_106007_106010")
- .put("ASA-2-106010", "CISCOFW106006_106007_106010")
- .put("ASA-3-106014", "CISCOFW106014")
- .put("ASA-6-106015", "CISCOFW106015")
- .put("ASA-1-106021", "CISCOFW106021")
- .put("ASA-4-106023", "CISCOFW106023")
- .put("ASA-5-106100", "CISCOFW106100")
- .put("ASA-6-110002", "CISCOFW110002")
- .put("ASA-6-302010", "CISCOFW302010")
- .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
- .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
- .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
- .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
- .put("ASA-6-302020", "CISCOFW302020_302021")
- .put("ASA-6-302021", "CISCOFW302020_302021")
- .put("ASA-6-305011", "CISCOFW305011")
- .put("ASA-3-313001", "CISCOFW313001_313004_313008")
- .put("ASA-3-313004", "CISCOFW313001_313004_313008")
- .put("ASA-3-313008", "CISCOFW313001_313004_313008")
- .put("ASA-4-313005", "CISCOFW313005")
- .put("ASA-4-402117", "CISCOFW402117")
- .put("ASA-4-402119", "CISCOFW402119")
- .put("ASA-4-419001", "CISCOFW419001")
- .put("ASA-4-419002", "CISCOFW419002")
- .put("ASA-4-500004", "CISCOFW500004")
- .put("ASA-6-602303", "CISCOFW602303_602304")
- .put("ASA-6-602304", "CISCOFW602303_602304")
- .put("ASA-7-710001", "CISCOFW710001_710002_710003_710005_710006")
- .put("ASA-7-710002", "CISCOFW710001_710002_710003_710005_710006")
- .put("ASA-7-710003", "CISCOFW710001_710002_710003_710005_710006")
- .put("ASA-7-710005", "CISCOFW710001_710002_710003_710005_710006")
- .put("ASA-7-710006", "CISCOFW710001_710002_710003_710005_710006")
- .put("ASA-6-713172", "CISCOFW713172")
- .put("ASA-4-733100", "CISCOFW733100")
- .put("ASA-6-305012", "CISCOFW305012")
- .put("ASA-7-609001", "CISCOFW609001")
- .put("ASA-7-609002", "CISCOFW609002")
- .put("ASA-5-713041", "CISCOFW713041")
- .build();
-
- @Override
- public void configure(Map<String, Object> parserConfig) {
- String timeZone = (String) parserConfig.get("deviceTimeZone");
- if (timeZone != null)
- deviceClock = Clock.system(ZoneId.of(timeZone));
- else {
- deviceClock = Clock.systemUTC();
- LOG.warn("[Metron] No device time zone provided; defaulting to UTC");
- }
+ protected static final Logger LOG = LoggerFactory.getLogger(BasicAsaParser.class);
+
+ protected Clock deviceClock;
+ private String syslogPattern = "%{CISCO_TAGGED_SYSLOG}";
+
+ private Grok syslogGrok;
+
+ private static final Map<String, String> patternMap = ImmutableMap.<String, String> builder()
+ .put("ASA-2-106001", "CISCOFW106001")
+ .put("ASA-2-106006", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106007", "CISCOFW106006_106007_106010")
+ .put("ASA-2-106010", "CISCOFW106006_106007_106010")
+ .put("ASA-3-106014", "CISCOFW106014")
+ .put("ASA-6-106015", "CISCOFW106015")
+ .put("ASA-1-106021", "CISCOFW106021")
+ .put("ASA-4-106023", "CISCOFW106023")
+ .put("ASA-5-106100", "CISCOFW106100")
+ .put("ASA-6-110002", "CISCOFW110002")
+ .put("ASA-6-302010", "CISCOFW302010")
+ .put("ASA-6-302013", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302014", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302015", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302016", "CISCOFW302013_302014_302015_302016")
+ .put("ASA-6-302020", "CISCOFW302020_302021")
+ .put("ASA-6-302021", "CISCOFW302020_302021")
+ .put("ASA-6-305011", "CISCOFW305011")
+ .put("ASA-3-313001", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313004", "CISCOFW313001_313004_313008")
+ .put("ASA-3-313008", "CISCOFW313001_313004_313008")
+ .put("ASA-4-313005", "CISCOFW313005")
+ .put("ASA-4-402117", "CISCOFW402117")
+ .put("ASA-4-402119", "CISCOFW402119")
+ .put("ASA-4-419001", "CISCOFW419001")
+ .put("ASA-4-419002", "CISCOFW419002")
+ .put("ASA-4-500004", "CISCOFW500004")
+ .put("ASA-6-602303", "CISCOFW602303_602304")
+ .put("ASA-6-602304", "CISCOFW602303_602304")
+ .put("ASA-7-710001", "CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710002", "CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710003", "CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710005", "CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-7-710006", "CISCOFW710001_710002_710003_710005_710006")
+ .put("ASA-6-713172", "CISCOFW713172")
+ .put("ASA-4-733100", "CISCOFW733100")
+ .put("ASA-6-305012", "CISCOFW305012")
+ .put("ASA-7-609001", "CISCOFW609001")
+ .put("ASA-7-609002", "CISCOFW609002")
+ .put("ASA-5-713041", "CISCOFW713041")
+ .build();
+
+ private Map<String, Grok> grokers = new HashMap<String, Grok>(patternMap.size());
+
+ @Override
+ public void configure(Map<String, Object> parserConfig) {
+ String timeZone = (String) parserConfig.get("deviceTimeZone");
+ if (timeZone != null)
+ deviceClock = Clock.system(ZoneId.of(timeZone));
+ else {
+ deviceClock = Clock.systemUTC();
+ LOG.warn("[Metron] No device time zone provided; defaulting to UTC");
+ }
+ }
+
+ private void addGrok(String key, String pattern) throws GrokException {
+ Grok grok = new Grok();
+ InputStream patternStream = this.getClass().getResourceAsStream("/patterns/asa");
+ grok.addPatternFromReader(new InputStreamReader(patternStream));
+ grok.compile("%{" + pattern + "}");
+ grokers.put(key, grok);
+ }
+
+ @Override
+ public void init() {
+ syslogGrok = new Grok();
+ InputStream syslogStream = this.getClass().getResourceAsStream("/patterns/asa");
+ try {
+ syslogGrok.addPatternFromReader(new InputStreamReader(syslogStream));
+ syslogGrok.compile(syslogPattern);
+ } catch (GrokException e) {
+ LOG.error("[Metron] Failed to load grok patterns from jar", e);
+ throw new RuntimeException(e.getMessage(), e);
+ }
+
+ for (Entry<String, String> pattern : patternMap.entrySet()) {
+ try {
+ addGrok(pattern.getKey(), pattern.getValue());
+ } catch (GrokException e) {
+ LOG.error("[Metron] Failed to load grok pattern %s for ASA tag %s", pattern.getValue(), pattern.getKey());
+ }
}
- @Override
- public void init() {
- asaGrok = new Grok();
- InputStream patternStream = this.getClass().getResourceAsStream("/patterns/asa");
- try {
- asaGrok.addPatternFromReader(new InputStreamReader(patternStream));
- } catch (GrokException e) {
- LOG.error("[Metron] Failed to load grok patterns from jar", e);
- throw new RuntimeException(e.getMessage(), e);
- }
- LOG.info("[Metron] CISCO ASA Parser Initialized");
+ LOG.info("[Metron] CISCO ASA Parser Initialized");
+ }
+
+ @Override
+ public List<JSONObject> parse(byte[] rawMessage) {
+ String logLine = "";
+ String messagePattern = "";
+ JSONObject metronJson = new JSONObject();
+ List<JSONObject> messages = new ArrayList<>();
+ Map<String, Object> syslogJson = new HashMap<String, Object>();
+
+ try {
+ logLine = new String(rawMessage, "UTF-8");
+ } catch (UnsupportedEncodingException e) {
+ LOG.error("[Metron] Could not read raw message", e);
+ throw new RuntimeException(e.getMessage(), e);
}
- @Override
- public List<JSONObject> parse(byte[] rawMessage) {
- String logLine = "";
- String syslogPattern = "%{CISCO_TAGGED_SYSLOG}";
- String messagePattern = "";
- JSONObject metronJson = new JSONObject();
- List<JSONObject> messages = new ArrayList<>();
- Map<String, Object> syslogJson = new HashMap<String, Object>();
-
- try {
- logLine = new String(rawMessage, "UTF-8");
- } catch (UnsupportedEncodingException e) {
- LOG.error("[Metron] Could not read raw message", e);
- throw new RuntimeException(e.getMessage(), e);
- }
-
- try {
- LOG.debug("[Metron] Started parsing raw message: {}", logLine);
-
- asaGrok.compile(syslogPattern);
- Match syslogMatch = asaGrok.match(logLine);
- syslogMatch.captures();
- if(!syslogMatch.isNull()) {
- syslogJson = syslogMatch.toMap();
- LOG.trace("[Metron] Grok CISCO ASA syslog matches: {}", syslogMatch.toJson());
-
- metronJson.put(Constants.Fields.ORIGINAL.getName(), logLine);
- metronJson.put(Constants.Fields.TIMESTAMP.getName(),
- SyslogUtils.parseTimestampToEpochMillis((String) syslogJson.get("CISCOTIMESTAMP"), deviceClock));
- metronJson.put("ciscotag", syslogJson.get("CISCOTAG"));
- metronJson.put("syslog_severity", SyslogUtils.getSeverityFromPriority((int) syslogJson.get("syslog_pri")));
- metronJson.put("syslog_facility", SyslogUtils.getFacilityFromPriority((int) syslogJson.get("syslog_pri")));
-
-
- if (syslogJson.get("syslog_host")!=null) {
- metronJson.put("syslog_host", syslogJson.get("syslog_host"));
- }
- if (syslogJson.get("syslog_prog")!=null) {
- metronJson.put("syslog_prog", syslogJson.get("syslog_prog"));
- }
-
- }
- else
- throw new RuntimeException(String.format("[Metron] Message '%s' does not match pattern '%s'", logLine, syslogPattern));
- } catch (GrokException e) {
- LOG.error(String.format("[Metron] Could not compile grok pattern '%s'", syslogPattern), e);
- throw new RuntimeException(e.getMessage(), e);
- } catch (ParseException e) {
- LOG.error("[Metron] Could not parse message timestamp", e);
- throw new RuntimeException(e.getMessage(), e);
- } catch (RuntimeException e) {
- LOG.error(e.getMessage(), e);
- throw new RuntimeException(e.getMessage(), e);
- }
-
- try {
- messagePattern = patternMap.get(syslogJson.get("CISCOTAG"));
- if (messagePattern == null)
- LOG.info("[Metron] No pattern for ciscotag '{}'", syslogJson.get("CISCOTAG"));
- else {
- asaGrok.compile("%{" + messagePattern + "}");
- Match messageMatch = asaGrok.match((String) syslogJson.get("message"));
- messageMatch.captures();
- if (!messageMatch.isNull()) {
- Map<String, Object> messageJson = messageMatch.toMap();
- LOG.trace("[Metron] Grok CISCO ASA message matches: {}", messageMatch.toJson());
-
- String src_ip = (String) messageJson.get("src_ip");
- if (src_ip != null)
- metronJson.put(Constants.Fields.SRC_ADDR.getName(), src_ip);
-
- Integer src_port = (Integer) messageJson.get("src_port");
- if (src_port != null)
- metronJson.put(Constants.Fields.SRC_PORT.getName(), src_port);
-
- String dst_ip = (String) messageJson.get("dst_ip");
- if (dst_ip != null)
- metronJson.put(Constants.Fields.DST_ADDR.getName(), dst_ip);
-
- Integer dst_port = (Integer) messageJson.get("dst_port");
- if (dst_port != null)
- metronJson.put(Constants.Fields.DST_PORT.getName(), dst_port);
-
- String protocol = (String) messageJson.get("protocol");
- if (protocol != null)
- metronJson.put(Constants.Fields.PROTOCOL.getName(), protocol.toLowerCase());
-
- String action = (String) messageJson.get("action");
- if (action != null)
- metronJson.put("action", action.toLowerCase());
- }
- else
- LOG.warn("[Metron] Message '{}' did not match pattern for ciscotag '{}'", logLine, syslogJson.get("CISCOTAG"));
- }
-
- LOG.debug("[Metron] Final normalized message: {}", metronJson.toString());
-
- } catch (GrokException e) {
- LOG.error(String.format("[Metron] Could not compile grok pattern '%s'", messagePattern), e);
- throw new RuntimeException(e.getMessage(), e);
- } catch (RuntimeException e) {
- LOG.error(e.getMessage(), e);
- throw new RuntimeException(e.getMessage(), e);
- }
-
- messages.add(metronJson);
- return messages;
+ try {
+ LOG.debug("[Metron] Started parsing raw message: {}", logLine);
+ Match syslogMatch = syslogGrok.match(logLine);
+ syslogMatch.captures();
+ if (!syslogMatch.isNull()) {
+ syslogJson = syslogMatch.toMap();
+ LOG.trace("[Metron] Grok CISCO ASA syslog matches: {}", syslogMatch.toJson());
+
+ metronJson.put(Constants.Fields.ORIGINAL.getName(), logLine);
+ metronJson.put(Constants.Fields.TIMESTAMP.getName(),
+ SyslogUtils.parseTimestampToEpochMillis((String) syslogJson.get("CISCOTIMESTAMP"), deviceClock));
+ metronJson.put("ciscotag", syslogJson.get("CISCOTAG"));
+ metronJson.put("syslog_severity", SyslogUtils.getSeverityFromPriority((int) syslogJson.get("syslog_pri")));
+ metronJson.put("syslog_facility", SyslogUtils.getFacilityFromPriority((int) syslogJson.get("syslog_pri")));
+
+ if (syslogJson.get("syslog_host") != null) {
+ metronJson.put("syslog_host", syslogJson.get("syslog_host"));
+ }
+ if (syslogJson.get("syslog_prog") != null) {
+ metronJson.put("syslog_prog", syslogJson.get("syslog_prog"));
+ }
+
+ } else
+ throw new RuntimeException(
+ String.format("[Metron] Message '%s' does not match pattern '%s'", logLine, syslogPattern));
+ } catch (ParseException e) {
+ LOG.error("[Metron] Could not parse message timestamp", e);
+ throw new RuntimeException(e.getMessage(), e);
+ } catch (RuntimeException e) {
+ LOG.error(e.getMessage(), e);
+ throw new RuntimeException(e.getMessage(), e);
}
+
+ try {
+ messagePattern = (String) syslogJson.get("CISCOTAG");
+ Grok asaGrok = grokers.get(messagePattern);
+
+ if (asaGrok == null)
+ LOG.info("[Metron] No pattern for ciscotag '{}'", syslogJson.get("CISCOTAG"));
+ else {
+
+ String messageContent = (String) syslogJson.get("message");
+ Match messageMatch = asaGrok.match(messageContent);
+ messageMatch.captures();
+ if (!messageMatch.isNull()) {
+ Map<String, Object> messageJson = messageMatch.toMap();
+ LOG.trace("[Metron] Grok CISCO ASA message matches: {}", messageMatch.toJson());
+
+ String src_ip = (String) messageJson.get("src_ip");
+ if (src_ip != null)
+ metronJson.put(Constants.Fields.SRC_ADDR.getName(), src_ip);
+
+ Integer src_port = (Integer) messageJson.get("src_port");
+ if (src_port != null)
+ metronJson.put(Constants.Fields.SRC_PORT.getName(), src_port);
+
+ String dst_ip = (String) messageJson.get("dst_ip");
+ if (dst_ip != null)
+ metronJson.put(Constants.Fields.DST_ADDR.getName(), dst_ip);
+
+ Integer dst_port = (Integer) messageJson.get("dst_port");
+ if (dst_port != null)
+ metronJson.put(Constants.Fields.DST_PORT.getName(), dst_port);
+
+ String protocol = (String) messageJson.get("protocol");
+ if (protocol != null)
+ metronJson.put(Constants.Fields.PROTOCOL.getName(), protocol.toLowerCase());
+
+ String action = (String) messageJson.get("action");
+ if (action != null)
+ metronJson.put("action", action.toLowerCase());
+ } else
+ LOG.warn("[Metron] Message '{}' did not match pattern for ciscotag '{}'", logLine,
+ syslogJson.get("CISCOTAG"));
+ }
+
+ LOG.debug("[Metron] Final normalized message: {}", metronJson.toString());
+
+ } catch (RuntimeException e) {
+ LOG.error(e.getMessage(), e);
+ throw new RuntimeException(e.getMessage(), e);
+ }
+
+ messages.add(metronJson);
+ return messages;
+ }
}
[02/18] metron git commit: METRON-977: metron-statistics enrichment
example uses old indexing syntax (mmiklavc) closes apache/metron#601
Posted by ma...@apache.org.
METRON-977: metron-statistics enrichment example uses old indexing syntax (mmiklavc) closes apache/metron#601
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/1e9acc72
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/1e9acc72
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/1e9acc72
Branch: refs/heads/Metron_0.4.0
Commit: 1e9acc72e3642c372cb84cab52fee91e96276e22
Parents: e31ed2f
Author: mmiklavc <mi...@gmail.com>
Authored: Mon Jun 5 17:11:08 2017 -0400
Committer: Michael Miklavcic <mi...@gmail.com>
Committed: Mon Jun 5 17:11:08 2017 -0400
----------------------------------------------------------------------
metron-analytics/metron-statistics/README.md | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/metron/blob/1e9acc72/metron-analytics/metron-statistics/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/README.md b/metron-analytics/metron-statistics/README.md
index 74e78b0..f822845 100644
--- a/metron-analytics/metron-statistics/README.md
+++ b/metron-analytics/metron-statistics/README.md
@@ -334,8 +334,6 @@ Create the following in
```
{
- "index": "mad",
- "batchSize": 1,
"enrichment": {
"fieldMap": {
"stellar" : {
@@ -364,6 +362,23 @@ PROFILE_GET( 'sketchy_mad', 'global', PROFILE_FIXED(10, 'MINUTES')) ), value)"
}
```
+We also need an indexing configuration. Create the following in
+`$METRON_HOME/config/zookeeper/indexing/mad.json`:
+```
+{
+ "hdfs" : {
+ "index": "mad",
+ "batchSize": 1,
+ "enabled" : true
+ },
+ "elasticsearch" : {
+ "index": "mad",
+ "batchSize": 1,
+ "enabled" : true
+ }
+}
+```
+
#### The Profiler
We can set up the profiler to track the MAD statistical state required