You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@metron.apache.org by le...@apache.org on 2019/01/17 18:00:42 UTC
[metron] branch master updated: METRON-1929 Build GET_ASN Stellar
function (justinleet) closes apache/metron#1299
This is an automated email from the ASF dual-hosted git repository.
leet pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new f43035c METRON-1929 Build GET_ASN Stellar function (justinleet) closes apache/metron#1299
f43035c is described below
commit f43035c02ef01f07ff382bbf136eb1bada727fbb
Author: justinleet <ju...@gmail.com>
AuthorDate: Thu Jan 17 12:59:58 2019 -0500
METRON-1929 Build GET_ASN Stellar function (justinleet) closes apache/metron#1299
---
Upgrading.md | 8 +
dependencies_with_url.csv | 4 +-
.../compose/hadoop/docker-entrypoint.sh | 2 +-
.../packaging/ambari/metron-mpack/README.md | 4 +-
.../configuration/metron-enrichment-env.xml | 8 +-
.../CURRENT/package/scripts/enrichment_commands.py | 45 ++--
.../CURRENT/package/scripts/enrichment_master.py | 4 +-
.../CURRENT/package/scripts/params/params_linux.py | 4 +-
.../package/scripts/params/status_params.py | 2 +-
.../METRON/CURRENT/themes/metron_theme.json | 10 +
.../packaging/docker/rpm-docker/SPECS/metron.spec | 2 +-
.../metron-alerts/cypress/fixtures/config.json | 2 +-
metron-platform/metron-data-management/README.md | 19 +-
...tLoader.java => MaxmindDbEnrichmentLoader.java} | 110 ++++++---
...richment_load.sh => maxmind_enrichment_load.sh} | 2 +-
...est.java => MaxmindDbEnrichmentLoaderTest.java} | 67 ++++--
metron-platform/metron-enrichment/pom.xml | 2 +-
.../metron/enrichment/adapters/geo/GeoAdapter.java | 9 +-
.../adapters/maxmind/MaxMindDatabase.java | 133 +++++++++++
.../adapters/maxmind/MaxMindDbUtilities.java | 127 ++++++++++
.../adapters/maxmind/asn/GeoLiteAsnDatabase.java | 164 +++++++++++++
.../geo/GeoLiteCityDatabase.java} | 169 +++++++------
.../{ => maxmind}/geo/hash/DistanceStrategies.java | 2 +-
.../{ => maxmind}/geo/hash/DistanceStrategy.java | 2 +-
.../{ => maxmind}/geo/hash/GeoHashUtil.java | 6 +-
.../enrichment/bolt/ThreatIntelJoinBolt.java | 10 +-
.../enrichment/bolt/UnifiedEnrichmentBolt.java | 8 +-
...tFunctions.java => AsnEnrichmentFunctions.java} | 73 +++---
.../enrichment/stellar/GeoEnrichmentFunctions.java | 20 +-
.../enrichment/stellar/GeoHashFunctions.java | 12 +-
.../adapters/geo/GeoLiteDatabaseTest.java | 190 ---------------
.../maxmind/asn/GeoLiteAsnDatabaseTest.java | 157 ++++++++++++
.../adapters/{ => maxmind}/geo/GeoAdapterTest.java | 8 +-
.../maxmind/geo/GeoLiteCityDatabaseTest.java | 263 +++++++++++++++++++++
.../enrichment/bolt/GenericEnrichmentBoltTest.java | 7 +-
.../enrichment/bolt/ThreatIntelJoinBoltTest.java | 9 +-
.../integration/EnrichmentIntegrationTest.java | 10 +-
.../stellar/AsnEnrichmentFunctionsTest.java | 173 ++++++++++++++
.../stellar/GeoEnrichmentFunctionsTest.java | 83 +++++--
.../resources/GeoLite/GeoIP2-City-Test-2.mmdb.gz | Bin 10011 -> 0 bytes
.../src/test/resources/GeoLite/GeoLite2-ASN.tar.gz | Bin 0 -> 3460335 bytes
...IP2-City-Test.mmdb.gz => GeoLite2-City.mmdb.gz} | Bin
.../test/resources/GeoLite/GeoLite2-City.tar.gz | Bin 0 -> 26788236 bytes
metron-stellar/stellar-common/README.md | 7 +
44 files changed, 1462 insertions(+), 475 deletions(-)
diff --git a/Upgrading.md b/Upgrading.md
index 1b2324b..fcd5f27 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -19,6 +19,14 @@ limitations under the License.
This document constitutes a per-version listing of changes of
configuration which are non-backwards compatible.
+## 0.7.0 to 0.7.1
+### [METRON-1929: Build GET_ASN Stellar function](https://issues.apache.org/jira/browse/METRON-1929)
+The script for `geo_enrichment_load.sh` has been renamed, and now is `maxmind_enrichment_load.sh`. A couple changes should happen for users who are upgrading.
+
+* The MaxMind GeoLite2 ASN database should be loaded onto HDFS at /apps/metron/asn/default/GeoLite2-ASN.tar.gz OR the global configuration property `asn.hdfs.file` can be set to point to a custom HDFS location.
+* Any custom scripts or tasks that use this script should be updated. In addition, this updated script also retrieves the GeoLite2 ASN database. The `-ra` flag can be used to provide a custom location for this database if offline install is needed. Otherwise, it will retrieve the latest from MaxMind.
+
+
## 0.6.0 to 0.7.0
### [METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API](https://issues.apache.org/jira/browse/METRON-1834)
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index 5856a3a..e16650d 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -28,8 +28,8 @@ com.jayway.jsonpath:json-path:jar:2.3.0:compile,Apache v2,https://github.com/jso
com.jayway.jsonpath:json-path:jar:2.4.0:compile,Apache v2,https://github.com/json-path/JsonPath
net.minidev:accessors-smart:jar:1.2:compile,Apache v2,https://github.com/netplex/json-smart-v2
net.minidev:json-smart:jar:2.3:compile,Apache v2,https://github.com/netplex/json-smart-v2
-com.maxmind.db:maxmind-db:jar:1.2.1:compile,CC-BY-SA 3.0,https://github.com/maxmind/MaxMind-DB
-com.maxmind.geoip2:geoip2:jar:2.8.0:compile,Apache v2,https://github.com/maxmind/GeoIP2-java
+com.maxmind.db:maxmind-db:jar:1.2.2:compile,CC-BY-SA 3.0,https://github.com/maxmind/MaxMind-DB
+com.maxmind.geoip2:geoip2:jar:2.12.0:compile,Apache v2,https://github.com/maxmind/GeoIP2-java
com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile,CDDL,http://jaxb.java.net/
com.sun.xml.bind:jaxb-impl:jar:2.2.5-2:compile,CDDL,http://jaxb.java.net/
com.twitter:jsr166e:jar:1.1.0:compile,CC0 1.0 Universal,http://github.com/twitter/jsr166e
diff --git a/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh b/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh
index 97b9809..940451c 100755
--- a/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh
+++ b/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh
@@ -32,7 +32,7 @@ $HADOOP_PREFIX/bin/hdfs dfs -mkdir -p /apps/metron
$HADOOP_PREFIX/bin/hdfs dfs -mkdir -p /apps/metron/geo/default
# download geo database to hdfs
-curl http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz | $HADOOP_PREFIX/bin/hdfs dfs -put - /apps/metron/geo/default/GeoLite2-City.mmdb.gz
+curl http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz | $HADOOP_PREFIX/bin/hdfs dfs -put - /apps/metron/geo/default/GeoLite2-City.mmdb.gz
# pass through CMD as PID 1
exec "$@"
diff --git a/metron-deployment/packaging/ambari/metron-mpack/README.md b/metron-deployment/packaging/ambari/metron-mpack/README.md
index 3459db2..9e3de6b 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/README.md
+++ b/metron-deployment/packaging/ambari/metron-mpack/README.md
@@ -183,7 +183,9 @@ A custom action is available in Ambari to import Zeppelin dashboards. See the [m
#### Offline Installation
-Retrieval of the GeoIP database is the only point during installation that reaches out to the internet. For an offline installation, the URL for the GeoIP database can be manually set to a local path on the file system such as `file:///home/root/geoip/GeoLite2-City.mmdb.gz`.
+Retrieval of the GeoIP and ASN databases (both from MaxMind) is the only point during installation that reaches out to the internet. For an offline installation, the URL for the databases can be manually set to a local path on the file system such as `file:///home/root/geoip/GeoLite2-City.tar.gz`.
+
+The properties for configuration are `geoip_url` and `asn_url` in the `Enrichment` section.
The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in `/localrepo`.
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
index 69dce3f..950db6a 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
@@ -24,11 +24,17 @@
-->
<property>
<name>geoip_url</name>
- <value>http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz</value>
+ <value>http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz</value>
<description>Location of the GeoIP data to load.</description>
<display-name>GEOIP Load Datafile URL</display-name>
</property>
<property>
+ <name>asn_url</name>
+ <value>http://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz</value>
+ <description>Location of the ASN data to load.</description>
+ <display-name>ASN Load Datafile URL</display-name>
+ </property>
+ <property>
<name>enrichment_host_known_hosts</name>
<description>List of Known Hosts for Host Enrichment</description>
<value>[{"ip":"10.1.128.236", "local":"YES", "type":"webserver", "asset_value" : "important"},{"ip":"10.1.128.237", "local":"UNKNOWN", "type":"unknown", "asset_value" : "important"},{"ip":"10.60.10.254", "local":"YES", "type":"printer", "asset_value" : "important"}]</value>
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index a1bdbed..1b71e60 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -35,7 +35,7 @@ class EnrichmentCommands:
__kafka_acl_configured = False
__hbase_configured = False
__hbase_acl_configured = False
- __geo_configured = False
+ __maxmind_configured = False
def __init__(self, params):
if params is None:
@@ -47,7 +47,7 @@ class EnrichmentCommands:
self.__kafka_acl_configured = os.path.isfile(self.__params.enrichment_kafka_acl_configured_flag_file)
self.__hbase_configured = os.path.isfile(self.__params.enrichment_hbase_configured_flag_file)
self.__hbase_acl_configured = os.path.isfile(self.__params.enrichment_hbase_acl_configured_flag_file)
- self.__geo_configured = os.path.isfile(self.__params.enrichment_geo_configured_flag_file)
+ self.__maxmind_configured = os.path.isfile(self.__params.enrichment_maxmind_configured_flag_file)
def __get_topics(self):
return [self.__enrichment_topic, self.__params.enrichment_error_topic]
@@ -67,8 +67,8 @@ class EnrichmentCommands:
def is_hbase_acl_configured(self):
return self.__hbase_acl_configured
- def is_geo_configured(self):
- return self.__geo_configured
+ def is_maxmind_configured(self):
+ return self.__maxmind_configured
def set_kafka_configured(self):
metron_service.set_configured(self.__params.metron_user, self.__params.enrichment_kafka_configured_flag_file, "Setting Kafka configured to True for enrichment")
@@ -82,11 +82,11 @@ class EnrichmentCommands:
def set_hbase_acl_configured(self):
metron_service.set_configured(self.__params.metron_user, self.__params.enrichment_hbase_acl_configured_flag_file, "Setting HBase ACL configured to True for enrichment")
- def set_geo_configured(self):
- metron_service.set_configured(self.__params.metron_user, self.__params.enrichment_geo_configured_flag_file, "Setting GEO configured to True for enrichment")
+ def set_maxmind_configured(self):
+ metron_service.set_configured(self.__params.metron_user, self.__params.enrichment_maxmind_configured_flag_file, "Setting Maxmind databases configured to True for enrichment")
- def init_geo(self):
- Logger.info("Creating HDFS location for GeoIP database")
+ def init_maxmind(self):
+ Logger.info("Creating HDFS locations for MaxMind databases")
self.__params.HdfsResource(self.__params.geoip_hdfs_dir,
type="directory",
action="create_on_execute",
@@ -95,20 +95,32 @@ class EnrichmentCommands:
mode=0755,
)
- Logger.info("Creating and loading GeoIp database")
- command_template = """{0}/bin/geo_enrichment_load.sh \
+ self.__params.HdfsResource(self.__params.asn_hdfs_dir,
+ type="directory",
+ action="create_on_execute",
+ owner=self.__params.metron_user,
+ group=self.__params.metron_group,
+ mode=0755,
+ )
+
+ Logger.info("Creating and loading Maxmind databases")
+ command_template = """{0}/bin/maxmind_enrichment_load.sh \
-g {1} \
- -r {2} \
- -z {3}"""
+ -a {2} \
+ -r {3} \
+ -ra {4} \
+ -z {5}"""
command = command_template.format(self.__params.metron_home,
self.__params.geoip_url,
+ self.__params.asn_url,
self.__params.geoip_hdfs_dir,
+ self.__params.asn_hdfs_dir,
self.__params.zookeeper_quorum
)
Logger.info("Executing command " + command)
Execute(command, user=self.__params.metron_user, tries=1, logoutput=True)
- Logger.info("Done intializing GeoIP data")
- self.set_geo_configured()
+ Logger.info("Done intializing Maxmind databases")
+ self.set_maxmind_configured()
def init_kafka_topics(self):
Logger.info('Creating Kafka topics for enrichment')
@@ -238,7 +250,10 @@ class EnrichmentCommands:
:param env: Environment
"""
Logger.info("Checking for Geo database")
- metron_service.check_hdfs_file_exists(self.__params, self.__params.geoip_hdfs_dir + "/GeoLite2-City.mmdb.gz")
+ metron_service.check_hdfs_file_exists(self.__params, self.__params.geoip_hdfs_dir + "/GeoLite2-City.tar.gz")
+
+ Logger.info("Checking for ASN database")
+ metron_service.check_hdfs_file_exists(self.__params, self.__params.asn_hdfs_dir + "/GeoLite2-ASN.tar.gz")
Logger.info('Checking Kafka topics for Enrichment')
metron_service.check_kafka_topics(self.__params, self.__get_topics())
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
index cada1d2..3267c9b 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
@@ -76,8 +76,8 @@ class Enrichment(Script):
commands.create_hbase_tables()
if params.security_enabled and not commands.is_hbase_acl_configured():
commands.set_hbase_acls()
- if not commands.is_geo_configured():
- commands.init_geo()
+ if not commands.is_maxmind_configured():
+ commands.init_maxmind()
commands.start_enrichment_topology(env)
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index a543d79..e97c0e8 100755
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -47,6 +47,7 @@ metron_home = status_params.metron_home
parsers = status_params.parsers
parser_error_topic = config['configurations']['metron-parsers-env']['parser_error_topic']
geoip_hdfs_dir = "/apps/metron/geo/default/"
+asn_hdfs_dir = "/apps/metron/asn/default/"
metron_user = status_params.metron_user
metron_group = config['configurations']['metron-env']['metron_group']
metron_log_dir = config['configurations']['metron-env']['metron_log_dir']
@@ -89,7 +90,7 @@ enrichment_kafka_configured_flag_file = status_params.enrichment_kafka_configure
enrichment_kafka_acl_configured_flag_file = status_params.enrichment_kafka_acl_configured_flag_file
enrichment_hbase_configured_flag_file = status_params.enrichment_hbase_configured_flag_file
enrichment_hbase_acl_configured_flag_file = status_params.enrichment_hbase_acl_configured_flag_file
-enrichment_geo_configured_flag_file = status_params.enrichment_geo_configured_flag_file
+enrichment_maxmind_configured_flag_file = status_params.enrichment_maxmind_configured_flag_file
indexing_configured_flag_file = status_params.indexing_configured_flag_file
indexing_acl_configured_flag_file = status_params.indexing_acl_configured_flag_file
indexing_hbase_configured_flag_file = status_params.indexing_hbase_configured_flag_file
@@ -307,6 +308,7 @@ threat_triage_score_field = config['configurations']['metron-rest-env']['threat_
# Enrichment
metron_enrichment_topology = status_params.metron_enrichment_topology
geoip_url = config['configurations']['metron-enrichment-env']['geoip_url']
+asn_url = config['configurations']['metron-enrichment-env']['asn_url']
enrichment_host_known_hosts = config['configurations']['metron-enrichment-env']['enrichment_host_known_hosts']
# Enrichment - Kafka
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
index 2563646..a9fc8f8 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/status_params.py
@@ -45,7 +45,7 @@ enrichment_kafka_configured_flag_file = metron_zookeeper_config_path + '/../metr
enrichment_kafka_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_kafka_acl_configured'
enrichment_hbase_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_hbase_configured'
enrichment_hbase_acl_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_hbase_acl_configured'
-enrichment_geo_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_geo_configured'
+enrichment_maxmind_configured_flag_file = metron_zookeeper_config_path + '/../metron_enrichment_maxmind_configured'
enrichment_hbase_table = config['configurations']['metron-enrichment-env']['enrichment_hbase_table']
enrichment_hbase_cf = config['configurations']['metron-enrichment-env']['enrichment_hbase_cf']
diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
index 6749101..65b2767 100644
--- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
+++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
@@ -536,6 +536,10 @@
"subsection-name": "subsection-enrichment-adapters"
},
{
+ "config": "metron-enrichment-env/asn_url",
+ "subsection-name": "subsection-enrichment-adapters"
+ },
+ {
"config": "metron-enrichment-env/enrichment_host_known_hosts",
"subsection-name": "subsection-enrichment-adapters"
},
@@ -1087,6 +1091,12 @@
}
},
{
+ "config": "metron-enrichment-env/asn_url",
+ "widget": {
+ "type": "text-field"
+ }
+ },
+ {
"config": "metron-enrichment-env/enrichment_host_known_hosts",
"widget": {
"type": "text-field"
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index 8f0cd9d..ddd8170 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -271,7 +271,7 @@ This package installs the Metron Parser files
%dir %{metron_home}/bin
%dir %{metron_home}/lib
%{metron_home}/bin/Whois_CSV_to_JSON.py
-%{metron_home}/bin/geo_enrichment_load.sh
+%{metron_home}/bin/maxmind_enrichment_load.sh
%{metron_home}/bin/flatfile_loader.sh
%{metron_home}/bin/flatfile_summarizer.sh
%{metron_home}/bin/prune_elasticsearch_indices.sh
diff --git a/metron-interface/metron-alerts/cypress/fixtures/config.json b/metron-interface/metron-alerts/cypress/fixtures/config.json
index 190e514..e3d17ad 100644
--- a/metron-interface/metron-alerts/cypress/fixtures/config.json
+++ b/metron-interface/metron-alerts/cypress/fixtures/config.json
@@ -19,5 +19,5 @@
"enrichment.writer.batchTimeout":"0",
"profiler.writer.batchSize":"15",
"profiler.writer.batchTimeout":"0",
- "geo.hdfs.file":"/apps/metron/geo/default/GeoLite2-City.mmdb.gz"
+ "geo.hdfs.file":"/apps/metron/geo/default/GeoLite2-City.tar.gz"
}
\ No newline at end of file
diff --git a/metron-platform/metron-data-management/README.md b/metron-platform/metron-data-management/README.md
index 1f8e52d..efb8e42 100644
--- a/metron-platform/metron-data-management/README.md
+++ b/metron-platform/metron-data-management/README.md
@@ -355,19 +355,22 @@ The parameters for the utility are as follows:
### GeoLite2 Loader
-The shell script `$METRON_HOME/bin/geo_enrichment_load.sh` will retrieve MaxMind GeoLite2 data and load data into HDFS, and update the configuration.
+The shell script `$METRON_HOME/bin/maxmind_enrichment_load.sh` will retrieve MaxMind GeoLite2 data and load data into HDFS, and update the configuration.
+The script will retrieve both the City and ASN databases.
THIS SCRIPT WILL NOT UPDATE AMBARI'S GLOBAL.JSON, JUST THE ZK CONFIGS. CHANGES WILL GO INTO EFFECT, BUT WILL NOT PERSIST PAST AN AMBARI RESTART UNTIL UPDATED THERE.
The parameters for the utility are as follows:
-| Short Code | Long Code | Is Required? | Description |
-|------------|---------------------|--------------|--------------------------------------------------------------------------------------------------|
-| -h | | No | Generate the help screen/set of options |
-| -g | --geo_url | No | GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz |
-| -r | --remote_dir | No | HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/\<epoch millis\>/ |
-| -t | --tmp_dir | No | Directory for landing the temporary GeoIP data - defaults to /tmp |
-| -z | --zk_quorum | Yes | Zookeeper Quorum URL (zk1:port,zk2:port,...) |
+| Short Code | Long Code | Is Required? | Description |
+|------------|---------------------|--------------|-----------------------------------------------------------------------------------------------------|
+| -h | | No | Generate the help screen/set of options |
+| -g | --geo_url | No | GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz |
+| -r | --remote_dir | No | HDFS directory to land formatted GeoIP City file - defaults to /apps/metron/geo/\<epoch millis\>/ |
+| -ra | --remote_asn_dir | No | HDFS directory to land formatted GeoIP ASN file - defaults to /apps/metron/asn/\<epoch millis\>/ |
+| -re | --retries | No | Number of GeoLite2 database download retries, after an initial failure. |
+| -t | --tmp_dir | No | Directory for landing the temporary GeoIP data - defaults to /tmp |
+| -z | --zk_quorum | Yes | Zookeeper Quorum URL (zk1:port,zk2:port,...) |
### Flatfile Summarizer
diff --git a/metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java b/metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/MaxmindDbEnrichmentLoader.java
similarity index 67%
rename from metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
rename to metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/MaxmindDbEnrichmentLoader.java
index fc89b89..78c78c8 100644
--- a/metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoader.java
+++ b/metron-platform/metron-data-management/src/main/java/org/apache/metron/dataloads/nonbulk/geo/MaxmindDbEnrichmentLoader.java
@@ -30,7 +30,8 @@ import org.apache.hadoop.util.GenericOptionsParser;
import org.apache.metron.common.configuration.ConfigurationsUtils;
import org.apache.metron.common.utils.CompressionStrategies;
import org.apache.metron.common.utils.JSONUtils;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import javax.annotation.Nullable;
import java.io.ByteArrayInputStream;
@@ -42,58 +43,77 @@ import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.util.Map;
-public class GeoEnrichmentLoader {
+public class MaxmindDbEnrichmentLoader {
private static final String DEFAULT_RETRIES = "2";
+ private static final String GEO_CITY_URL_DEFAULT = "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz";
+ private static final String ASN_URL_DEFAULT = "http://geolite.maxmind.com/download/geoip/database/GeoLite2-ASN.tar.gz";
private static abstract class OptionHandler implements Function<String, Option> {
}
public enum GeoEnrichmentOptions {
- HELP("h", new GeoEnrichmentLoader.OptionHandler() {
- @Nullable
+ HELP("h", new MaxmindDbEnrichmentLoader.OptionHandler() {
@Override
public Option apply(@Nullable String s) {
return new Option(s, "help", false, "Generate Help screen");
}
- }), GEO_URL("g", new GeoEnrichmentLoader.OptionHandler() {
- @Nullable
+ }),
+ ASN_URL("a", new MaxmindDbEnrichmentLoader.OptionHandler() {
@Override
public Option apply(@Nullable String s) {
- Option o = new Option(s, "geo_url", true, "GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz");
+ Option o = new Option(s, "asn_url", true, "GeoIP URL - " + ASN_URL_DEFAULT);
+ o.setArgName("ASN_URL");
+ o.setRequired(false);
+ return o;
+ }
+ }),
+ GEO_URL("g", new MaxmindDbEnrichmentLoader.OptionHandler() {
+ @Override
+ public Option apply(@Nullable String s) {
+ Option o = new Option(s, "geo_url", true, "GeoIP URL - defaults to " + GEO_CITY_URL_DEFAULT);
o.setArgName("GEO_URL");
o.setRequired(false);
return o;
}
- }), REMOTE_DIR("r", new GeoEnrichmentLoader.OptionHandler() {
- @Nullable
+ }),
+ REMOTE_GEO_DIR("r", new MaxmindDbEnrichmentLoader.OptionHandler() {
+ @Override
+ public Option apply(@Nullable String s) {
+ Option o = new Option(s, "remote_dir", true, "HDFS directory to land formatted GeoLite2 City file - defaults to /apps/metron/geo/<epoch millis>/");
+ o.setArgName("REMOTE_DIR");
+ o.setRequired(false);
+ return o;
+ }
+ }),
+ REMOTE_ASN_DIR("ra", new MaxmindDbEnrichmentLoader.OptionHandler() {
@Override
public Option apply(@Nullable String s) {
- Option o = new Option(s, "remote_dir", true, "HDFS directory to land formatted GeoIP file - defaults to /apps/metron/geo/<epoch millis>/");
+ Option o = new Option(s, "remote_asn_dir", true, "HDFS directory to land formatted GeoLite2 ASN file - defaults to /apps/metron/asn/<epoch millis>/");
o.setArgName("REMOTE_DIR");
o.setRequired(false);
return o;
}
- }), RETRIES("re", new GeoEnrichmentLoader.OptionHandler() {
- @Nullable
+ }),
+ RETRIES("re", new MaxmindDbEnrichmentLoader.OptionHandler() {
@Override
public Option apply(@Nullable String s) {
- Option o = new Option(s, "retries", true, "Number of geo db download retries, after an initial failure.");
+ Option o = new Option(s, "retries", true, "Number of GeoLite2 database download retries, after an initial failure.");
o.setArgName("RETRIES");
o.setRequired(false);
return o;
}
- }), TMP_DIR("t", new GeoEnrichmentLoader.OptionHandler() {
- @Nullable
+ }),
+ TMP_DIR("t", new MaxmindDbEnrichmentLoader.OptionHandler() {
@Override
public Option apply(@Nullable String s) {
- Option o = new Option(s, "tmp_dir", true, "Directory for landing the temporary GeoIP data - defaults to /tmp");
+ Option o = new Option(s, "tmp_dir", true, "Directory for landing the temporary GeoLite2 data - defaults to /tmp");
o.setArgName("TMP_DIR");
o.setRequired(false);
return o;
}
- }), ZK_QUORUM("z", new GeoEnrichmentLoader.OptionHandler() {
- @Nullable
+ }),
+ ZK_QUORUM("z", new MaxmindDbEnrichmentLoader.OptionHandler() {
@Override
public Option apply(@Nullable String s) {
Option o = new Option(s, "zk_quorum", true, "Zookeeper Quorum URL (zk1:port,zk2:port,...)");
@@ -105,7 +125,7 @@ public class GeoEnrichmentLoader {
Option option;
String shortCode;
- GeoEnrichmentOptions(String shortCode, GeoEnrichmentLoader.OptionHandler optionHandler) {
+ GeoEnrichmentOptions(String shortCode, MaxmindDbEnrichmentLoader.OptionHandler optionHandler) {
this.shortCode = shortCode;
this.option = optionHandler.apply(shortCode);
}
@@ -126,7 +146,7 @@ public class GeoEnrichmentLoader {
try {
CommandLine cli = parser.parse(getOptions(), args);
- if (GeoEnrichmentLoader.GeoEnrichmentOptions.HELP.has(cli)) {
+ if (MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.HELP.has(cli)) {
printHelp();
System.exit(0);
}
@@ -142,49 +162,65 @@ public class GeoEnrichmentLoader {
public static void printHelp() {
HelpFormatter formatter = new HelpFormatter();
- formatter.printHelp("GeoEnrichmentLoader", getOptions());
+ formatter.printHelp("MaxmindDbEnrichmentLoader", getOptions());
}
public static Options getOptions() {
Options ret = new Options();
- for (GeoEnrichmentLoader.GeoEnrichmentOptions o : GeoEnrichmentLoader.GeoEnrichmentOptions.values()) {
+ for (MaxmindDbEnrichmentLoader.GeoEnrichmentOptions o : MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.values()) {
ret.addOption(o.option);
}
return ret;
}
}
- protected void loadGeoIpDatabase(CommandLine cli) throws IOException {
+ protected void loadGeoLiteDatabase(CommandLine cli) throws IOException {
// Retrieve the database file
System.out.println("Retrieving GeoLite2 archive");
- String url = GeoEnrichmentOptions.GEO_URL.get(cli, "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz");
+
+ String geo_url = GeoEnrichmentOptions.GEO_URL.get(cli, GEO_CITY_URL_DEFAULT);
+ String asn_url = GeoEnrichmentOptions.ASN_URL.get(cli, ASN_URL_DEFAULT);
+
String tmpDir = GeoEnrichmentOptions.TMP_DIR.get(cli, "/tmp") + "/"; // Make sure there's a file separator at the end
int numRetries = Integer.parseInt(GeoEnrichmentOptions.RETRIES.get(cli, DEFAULT_RETRIES));
File localGeoFile = null;
+ File localAsnFile = null;
try {
- localGeoFile = downloadGeoFile(url, tmpDir, numRetries);
+ localGeoFile = downloadGeoFile(geo_url, tmpDir, numRetries);
+ localAsnFile = downloadGeoFile(asn_url, tmpDir, numRetries);
} catch (IllegalStateException ies) {
System.err.println("Failed to download geo db file. Aborting");
System.exit(5);
}
// Want to delete the tar in event of failure
localGeoFile.deleteOnExit();
+ localAsnFile.deleteOnExit();
System.out.println("GeoIP files downloaded successfully");
- // Push the file to HDFS and update Configs to ensure clients get new view
+ // Push the Geo file to HDFS and update Configs to ensure clients get new view
String zookeeper = GeoEnrichmentOptions.ZK_QUORUM.get(cli);
long millis = LocalDateTime.now().toInstant(ZoneOffset.UTC).toEpochMilli();
- String hdfsLoc = GeoEnrichmentOptions.REMOTE_DIR.get(cli, "/apps/metron/geo/" + millis);
- System.out.println("Putting GeoLite2 file into HDFS at: " + hdfsLoc);
+ String hdfsGeoLoc = GeoEnrichmentOptions.REMOTE_GEO_DIR.get(cli, "/apps/metron/geo/" + millis);
+ System.out.println("Putting GeoLite City file into HDFS at: " + hdfsGeoLoc);
- // Put into HDFS
+ // Put Geo into HDFS
Path srcPath = new Path(localGeoFile.getAbsolutePath());
- Path dstPath = new Path(hdfsLoc);
+ Path dstPath = new Path(hdfsGeoLoc);
+ putDbFile(srcPath, dstPath);
+ pushConfig(srcPath, dstPath, GeoLiteCityDatabase.GEO_HDFS_FILE, zookeeper);
+
+ // Push the ASN file to HDFS and update Configs to ensure clients get new view
+ String hdfsAsnLoc = GeoEnrichmentOptions.REMOTE_ASN_DIR.get(cli, "/apps/metron/asn/" + millis);
+ System.out.println("Putting ASN file into HDFS at: " + hdfsAsnLoc);
+
+ // Put ASN into HDFS
+ srcPath = new Path(localAsnFile.getAbsolutePath());
+ dstPath = new Path(hdfsAsnLoc);
putDbFile(srcPath, dstPath);
- pushConfig(srcPath, dstPath, zookeeper);
+ pushConfig(srcPath, dstPath, GeoLiteAsnDatabase.ASN_HDFS_FILE, zookeeper);
System.out.println("GeoLite2 file placement complete");
- System.out.println("Successfully created and updated new GeoIP information");
+ System.out.println("Successfully created and updated new GeoLite information");
}
protected File downloadGeoFile(String urlStr, String tmpDir, int numRetries) {
@@ -225,7 +261,7 @@ public class GeoEnrichmentLoader {
return localFile;
}
- protected void pushConfig(Path srcPath, Path dstPath, String zookeeper) {
+ protected void pushConfig(Path srcPath, Path dstPath, String configName, String zookeeper) {
System.out.println("Beginning update of global configs");
try (CuratorFramework client = ConfigurationsUtils.getClient(zookeeper)) {
client.start();
@@ -236,10 +272,10 @@ public class GeoEnrichmentLoader {
JSONUtils.MAP_SUPPLIER);
// Update the global config and push it back
- global.put(GeoLiteDatabase.GEO_HDFS_FILE, dstPath.toString() + "/" + srcPath.getName());
+ global.put(configName, dstPath.toString() + "/" + srcPath.getName());
ConfigurationsUtils.writeGlobalConfigToZookeeper(global, client);
} catch (Exception e) {
- System.err.println("Unable to load new GeoIP info into HDFS: " + e);
+ System.err.println("Unable to load new GeoLite2 config for " + configName + " into HDFS: " + e);
e.printStackTrace();
System.exit(2);
}
@@ -259,7 +295,7 @@ public class GeoEnrichmentLoader {
String[] otherArgs = new GenericOptionsParser(argv).getRemainingArgs();
CommandLine cli = GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
- GeoEnrichmentLoader loader = new GeoEnrichmentLoader();
- loader.loadGeoIpDatabase(cli);
+ MaxmindDbEnrichmentLoader loader = new MaxmindDbEnrichmentLoader();
+ loader.loadGeoLiteDatabase(cli);
}
}
diff --git a/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh b/metron-platform/metron-data-management/src/main/scripts/maxmind_enrichment_load.sh
similarity index 97%
rename from metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh
rename to metron-platform/metron-data-management/src/main/scripts/maxmind_enrichment_load.sh
index a2d5448..cf43b14 100644
--- a/metron-platform/metron-data-management/src/main/scripts/geo_enrichment_load.sh
+++ b/metron-platform/metron-data-management/src/main/scripts/maxmind_enrichment_load.sh
@@ -39,4 +39,4 @@ export METRON_VERSION=${project.version}
export METRON_HOME=/usr/metron/$METRON_VERSION
export DM_JAR=${project.artifactId}-$METRON_VERSION.jar
export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
-hadoop jar $METRON_HOME/lib/$DM_JAR org.apache.metron.dataloads.nonbulk.geo.GeoEnrichmentLoader "$@"
+hadoop jar $METRON_HOME/lib/$DM_JAR org.apache.metron.dataloads.nonbulk.geo.MaxmindDbEnrichmentLoader "$@"
diff --git a/metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoaderTest.java b/metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/nonbulk/geo/MaxmindDbEnrichmentLoaderTest.java
similarity index 53%
rename from metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoaderTest.java
rename to metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/nonbulk/geo/MaxmindDbEnrichmentLoaderTest.java
index 2babeee..126b28b 100644
--- a/metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/nonbulk/geo/GeoEnrichmentLoaderTest.java
+++ b/metron-platform/metron-data-management/src/test/java/org/apache/metron/dataloads/nonbulk/geo/MaxmindDbEnrichmentLoaderTest.java
@@ -35,10 +35,10 @@ import org.junit.Test;
import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
-public class GeoEnrichmentLoaderTest {
- private class MockGeoEnrichmentLoader extends GeoEnrichmentLoader {
+public class MaxmindDbEnrichmentLoaderTest {
+ private class MockMaxmindDbEnrichmentLoader extends MaxmindDbEnrichmentLoader {
@Override
- protected void pushConfig(Path srcPath, Path dstPath, String zookeeper) {
+ protected void pushConfig(Path srcPath, Path dstPath, String configName, String zookeeper) {
}
}
@@ -57,40 +57,61 @@ public class GeoEnrichmentLoaderTest {
@Test
public void testCommandLineShortOpts() throws Exception {
- String[] argv = {"-g testGeoUrl", "-r /test/remoteDir", "-t /test/tmpDir", "-z test:2181"};
+ String[] argv = {
+ "-g testGeoUrl",
+ "-a testAsnUrl",
+ "-r /test/remoteDirGeo",
+ "-ra", "/test/remoteDirAsn",
+ "-t /test/tmpDir",
+ "-z test:2181"
+ };
String[] otherArgs = new GenericOptionsParser(argv).getRemainingArgs();
- CommandLine cli = GeoEnrichmentLoader.GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
- Assert.assertEquals("testGeoUrl", GeoEnrichmentLoader.GeoEnrichmentOptions.GEO_URL.get(cli).trim());
- Assert.assertEquals("/test/remoteDir", GeoEnrichmentLoader.GeoEnrichmentOptions.REMOTE_DIR.get(cli).trim());
- Assert.assertEquals("/test/tmpDir", GeoEnrichmentLoader.GeoEnrichmentOptions.TMP_DIR.get(cli).trim());
- Assert.assertEquals("test:2181", GeoEnrichmentLoader.GeoEnrichmentOptions.ZK_QUORUM.get(cli).trim());
+ CommandLine cli = MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
+ Assert.assertEquals("testGeoUrl", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.GEO_URL.get(cli).trim());
+ Assert.assertEquals("testAsnUrl", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.ASN_URL.get(cli).trim());
+ Assert.assertEquals("/test/remoteDirGeo", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.REMOTE_GEO_DIR.get(cli).trim());
+ Assert.assertEquals("/test/remoteDirAsn", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.REMOTE_ASN_DIR.get(cli).trim());
+ Assert.assertEquals("/test/tmpDir", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.TMP_DIR.get(cli).trim());
+ Assert.assertEquals("test:2181", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.ZK_QUORUM.get(cli).trim());
}
@Test
public void testCommandLineLongOpts() throws Exception {
- String[] argv = {"--geo_url", "testGeoUrl", "--remote_dir", "/test/remoteDir", "--tmp_dir", "/test/tmpDir", "--zk_quorum", "test:2181"};
+ String[] argv = {
+ "--geo_url", "testGeoUrl",
+ "--remote_dir", "/test/remoteDir",
+ "-ra", "/test/remoteDir",
+ "--tmp_dir", "/test/tmpDir",
+ "--zk_quorum", "test:2181"
+ };
String[] otherArgs = new GenericOptionsParser(argv).getRemainingArgs();
- CommandLine cli = GeoEnrichmentLoader.GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
- Assert.assertEquals("testGeoUrl", GeoEnrichmentLoader.GeoEnrichmentOptions.GEO_URL.get(cli).trim());
- Assert.assertEquals("/test/remoteDir", GeoEnrichmentLoader.GeoEnrichmentOptions.REMOTE_DIR.get(cli).trim());
- Assert.assertEquals("/test/tmpDir", GeoEnrichmentLoader.GeoEnrichmentOptions.TMP_DIR.get(cli).trim());
- Assert.assertEquals("test:2181", GeoEnrichmentLoader.GeoEnrichmentOptions.ZK_QUORUM.get(cli).trim());
+ CommandLine cli = MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
+ Assert.assertEquals("testGeoUrl", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.GEO_URL.get(cli).trim());
+ Assert.assertEquals("/test/remoteDir", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.REMOTE_GEO_DIR.get(cli).trim());
+ Assert.assertEquals("/test/tmpDir", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.TMP_DIR.get(cli).trim());
+ Assert.assertEquals("test:2181", MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.ZK_QUORUM.get(cli).trim());
}
@Test
public void testLoadGeoIpDatabase() throws Exception {
- File dbPlainTextFile = new File(remoteDir.getAbsolutePath() + "/GeoEnrichmentLoaderTest.mmdb");
+ File dbPlainTextFile = new File(remoteDir.getAbsolutePath() + "/MaxmindDbEnrichmentLoaderTest.mmdb");
TestUtils.write(dbPlainTextFile, "hello world");
- File dbFile = new File(remoteDir.getAbsolutePath() + "/GeoEnrichmentLoaderTest.mmdb.gz");
+ File dbFile = new File(remoteDir.getAbsolutePath() + "/MaxmindDbEnrichmentLoaderTest.mmdb.gz");
CompressionStrategies.GZIP.compress(dbPlainTextFile, dbFile);
- String[] argv = {"--geo_url", "file://" + dbFile.getAbsolutePath(), "--remote_dir", remoteDir.getAbsolutePath(), "--tmp_dir", tmpDir.getAbsolutePath(), "--zk_quorum", "test:2181"};
+ String[] argv = {
+ "--geo_url", "file://" + dbFile.getAbsolutePath(),
+ "--remote_dir", remoteDir.getAbsolutePath(),
+ "--remote_asn_dir", remoteDir.getAbsolutePath(),
+ "--tmp_dir", tmpDir.getAbsolutePath(),
+ "--zk_quorum", "test:2181"
+ };
String[] otherArgs = new GenericOptionsParser(argv).getRemainingArgs();
- CommandLine cli = GeoEnrichmentLoader.GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
+ CommandLine cli = MaxmindDbEnrichmentLoader.GeoEnrichmentOptions.parse(new PosixParser(), otherArgs);
- GeoEnrichmentLoader loader = new MockGeoEnrichmentLoader();
- loader.loadGeoIpDatabase(cli);
+ MaxmindDbEnrichmentLoader loader = new MockMaxmindDbEnrichmentLoader();
+ loader.loadGeoLiteDatabase(cli);
Configuration config = new Configuration();
FileSystem fs = FileSystem.get(config);
assertTrue(fs.exists(new Path(remoteDir + "/" + dbFile.getName())));
@@ -101,14 +122,14 @@ public class GeoEnrichmentLoaderTest {
@Test
public void loader_throws_exception_on_bad_gzip_file() throws Exception {
- File dbFile = new File(remoteDir.getAbsolutePath() + "/GeoEnrichmentLoaderTest.mmdb");
+ File dbFile = new File(remoteDir.getAbsolutePath() + "/MaxmindDbEnrichmentLoaderTest.mmdb");
dbFile.createNewFile();
String geoUrl = "file://" + dbFile.getAbsolutePath();
int numRetries = 2;
exception.expect(IllegalStateException.class);
exception.expectMessage("Unable to download geo enrichment database.");
- GeoEnrichmentLoader loader = new MockGeoEnrichmentLoader();
+ MaxmindDbEnrichmentLoader loader = new MockMaxmindDbEnrichmentLoader();
loader.downloadGeoFile(geoUrl, tmpDir.getAbsolutePath(), numRetries);
}
diff --git a/metron-platform/metron-enrichment/pom.xml b/metron-platform/metron-enrichment/pom.xml
index 7429384..9f6e993 100644
--- a/metron-platform/metron-enrichment/pom.xml
+++ b/metron-platform/metron-enrichment/pom.xml
@@ -29,7 +29,7 @@
<slf4j.version>1.7.7</slf4j.version>
<storm.hdfs.version>0.1.2</storm.hdfs.version>
<commons-compress.version>1.13</commons-compress.version>
- <geoip.version>2.8.0</geoip.version>
+ <geoip.version>2.12.0</geoip.version>
<guava.version>${global_hbase_guava_version}</guava.version>
</properties>
<dependencies>
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoAdapter.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoAdapter.java
index 2447b59..830395f 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoAdapter.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoAdapter.java
@@ -19,9 +19,9 @@ package org.apache.metron.enrichment.adapters.geo;
import java.io.Serializable;
import java.lang.invoke.MethodHandles;
-import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.enrichment.bolt.CacheKey;
import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
import org.json.simple.JSONObject;
@@ -40,11 +40,10 @@ public class GeoAdapter implements EnrichmentAdapter<CacheKey>, Serializable {
return value.getField();
}
- @SuppressWarnings("unchecked")
@Override
public JSONObject enrich(CacheKey value) {
JSONObject enriched = new JSONObject();
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(value.coerceValue(String.class));
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(value.coerceValue(String.class));
if(!result.isPresent()) {
return new JSONObject();
}
@@ -56,13 +55,13 @@ public class GeoAdapter implements EnrichmentAdapter<CacheKey>, Serializable {
@Override
public boolean initializeAdapter(Map<String, Object> config) {
- GeoLiteDatabase.INSTANCE.update((String)config.get(GeoLiteDatabase.GEO_HDFS_FILE));
+ GeoLiteCityDatabase.INSTANCE.update((String)config.get(GeoLiteCityDatabase.GEO_HDFS_FILE));
return true;
}
@Override
public void updateAdapter(Map<String, Object> config) {
- GeoLiteDatabase.INSTANCE.updateIfNecessary(config);
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(config);
}
@Override
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/MaxMindDatabase.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/MaxMindDatabase.java
new file mode 100644
index 0000000..88aebbc
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/MaxMindDatabase.java
@@ -0,0 +1,133 @@
+package org.apache.metron.enrichment.adapters.maxmind;/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.maxmind.geoip2.DatabaseReader;
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Map;
+import java.util.zip.GZIPInputStream;
+import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
+import org.apache.commons.compress.archivers.tar.TarArchiveInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface MaxMindDatabase {
+ Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ String EXTENSION_MMDB = ".mmdb";
+ String EXTENSION_TAR_GZ = ".tar.gz";
+ String EXTENSION_MMDB_GZ = ".mmdb.gz";
+
+ /**
+ * Retrieves the configuration key that holds the HDFS database file location
+ * @return The configuration key
+ */
+ String getHdfsFileConfig();
+
+ /**
+ * Retrieves the default HDFS database file location
+ * @return The HDFS database file location
+ */
+ String getHdfsFileDefault();
+
+ /**
+ * Locks any underlying resources to ensure they are smoothly updated without disruption.
+ * Any callers implementing an update() function should lock during the update to ensure uninterrupted querying.
+ */
+ void lockIfNecessary();
+
+ /**
+ * Unlocks any underlying resources to ensure the lock is released after an update.
+ * Any callers implementing an update() function should ensure they've unlocked post update.
+ */
+ void unlockIfNecessary();
+
+ /**
+ * Gets the appropriate database reader for the underlying database that's been loaded
+ * @return The DatabaseReader for the MaxMind database.
+ */
+ DatabaseReader getReader();
+ void setReader(DatabaseReader reader);
+
+ /**
+ * Updates the database file, if the configuration points to a new file.
+ * Implementations may need to be synchronized to avoid issues querying during updates.
+ *
+ * @param globalConfig The global configuration that will be used to determine if an update is necessary.
+ */
+ void updateIfNecessary(Map<String, Object> globalConfig);
+
+ /**
+ * Update the database being queried to one backed by the provided HDFS file.
+ * Access to the database should be guarded by read locks to avoid disruption while updates are occurring.
+ * @param hdfsFile The HDFS file path to be used for new queries.
+ */
+ default void update(String hdfsFile) {
+ // If nothing is set (or it's been unset, use the defaults)
+ if (hdfsFile == null || hdfsFile.isEmpty()) {
+ LOG.debug("Using default for {}: {}", getHdfsFileConfig(), getHdfsFileDefault());
+ hdfsFile = getHdfsFileDefault();
+ }
+
+ FileSystem fs = MaxMindDbUtilities.getFileSystem();
+
+ if (hdfsFile.endsWith(MaxMindDatabase.EXTENSION_MMDB)) {
+ lockIfNecessary();
+ try (BufferedInputStream is = new BufferedInputStream(fs.open(new Path(hdfsFile)))) {
+ setReader(MaxMindDbUtilities.readNewDatabase(getReader(), hdfsFile, is));
+ } catch (IOException e) {
+ MaxMindDbUtilities.handleDatabaseIOException(hdfsFile, e);
+ } finally {
+ unlockIfNecessary();
+ }
+ } else if (hdfsFile.endsWith(MaxMindDatabase.EXTENSION_MMDB_GZ)) {
+ lockIfNecessary();
+ try (GZIPInputStream is = new GZIPInputStream(fs.open(new Path(hdfsFile)))) {
+ setReader(MaxMindDbUtilities.readNewDatabase(getReader(), hdfsFile, is));
+ } catch (IOException e) {
+ MaxMindDbUtilities.handleDatabaseIOException(hdfsFile, e);
+ } finally {
+ unlockIfNecessary();
+ }
+ } else if (hdfsFile.endsWith(MaxMindDatabase.EXTENSION_TAR_GZ)) {
+ lockIfNecessary();
+ try (TarArchiveInputStream is = new TarArchiveInputStream(
+ new GZIPInputStream(fs.open(new Path(hdfsFile))))) {
+ // Need to find the mmdb entry.
+ TarArchiveEntry entry = is.getNextTarEntry();
+ while (entry != null) {
+ if (entry.isFile() && entry.getName().endsWith(MaxMindDatabase.EXTENSION_MMDB)) {
+ try(InputStream mmdb = new BufferedInputStream(is))
+ { // Read directly from tarInput
+ setReader(MaxMindDbUtilities.readNewDatabase(getReader(), hdfsFile, mmdb));
+ break; // Don't care about the other entries, leave immediately
+ }
+ }
+ entry = is.getNextTarEntry();
+ }
+ } catch (IOException e) {
+ MaxMindDbUtilities.handleDatabaseIOException(hdfsFile, e);
+ } finally {
+ unlockIfNecessary();
+ }
+ }
+ }
+}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/MaxMindDbUtilities.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/MaxMindDbUtilities.java
new file mode 100644
index 0000000..3b12de5
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/MaxMindDbUtilities.java
@@ -0,0 +1,127 @@
+package org.apache.metron.enrichment.adapters.maxmind;/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.maxmind.db.CHMCache;
+import com.maxmind.geoip2.DatabaseReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A utilities class for working with MaxMind GeoLite2 databases. In particular, when the DB is stored on HDFS.
+ */
+public enum MaxMindDbUtilities {
+ INSTANCE;
+
+ static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ static InetAddressValidator ipvalidator = new InetAddressValidator();
+
+ /**
+ * Determines if an IP is ineligible. In particular, this is used to filter before querying the database, as that requires a readlock to be set.
+ * @param ip The IP to be tested
+ * @return true if invalid, false otherwise
+ */
+ public static boolean invalidIp(String ip) {
+ LOG.trace("Called validateIp({})", ip);
+ InetAddress addr;
+ try {
+ addr = InetAddress.getByName(ip);
+ } catch (UnknownHostException e) {
+ LOG.warn("No result found for IP {}", ip, e);
+ return true;
+ }
+ if (isIneligibleAddress(ip, addr)) {
+ LOG.debug("IP ineligible for lookup {}", ip);
+ return true;
+ }
+ return false;
+ }
+
+ /**
+ * Determines if an address isn't eligible for getting appropriate results from the underlying database.
+ * @param ipStr The IP String
+ * @param addr The addr to be tested
+ * @return True if ineligible, false otherwise
+ */
+ public static boolean isIneligibleAddress(String ipStr, InetAddress addr) {
+ return addr.isAnyLocalAddress() || addr.isLoopbackAddress()
+ || addr.isSiteLocalAddress() || addr.isMulticastAddress()
+ || !ipvalidator.isValidInet4Address(ipStr);
+ }
+
+ /**
+ * Logs and rethrows an IOException in a common way across implementations.
+ * @param hdfsFile The hdfsFile we were trying to read from
+ * @param e The exception we saw
+ */
+ public static void handleDatabaseIOException(String hdfsFile, IOException e) {
+ LOG.error("Unable to open new database file {}", hdfsFile, e);
+ throw new IllegalStateException("Unable to update MaxMind database");
+ }
+
+ /**
+ * Reads a new Database from a given HDFS file
+ * @param reader The DatabaseReader to use to read the file
+ * @param hdfsFile The HDFS file to read
+ * @param is An InputStream for use with the reader
+ * @return The DatabaseReader that is set up with the new file
+ * @throws IOException If there is an issue reading the file.
+ */
+ public static DatabaseReader readNewDatabase(DatabaseReader reader, String hdfsFile, InputStream is) throws IOException {
+ LOG.info("Update to GeoIP data started with {}", hdfsFile);
+ // InputStream based DatabaseReaders are always in memory.
+ DatabaseReader newReader = new DatabaseReader.Builder(is).withCache(new CHMCache()).build();
+ // If we've never set a reader, don't close the old one
+ if (reader != null) {
+ reader.close();
+ }
+ LOG.info("Finished update to GeoIP data started with {}", hdfsFile);
+ return newReader;
+ }
+
+ /**
+ * Retrieves the FileSystem
+ * @return A FileSystem object used to retrieve the the database
+ */
+ public static FileSystem getFileSystem() {
+ FileSystem fs;
+ try {
+ fs = FileSystem.get(new Configuration());
+ } catch (IOException e) {
+ LOG.error("Unable to retrieve get HDFS FileSystem");
+ throw new IllegalStateException("Unable to get HDFS FileSystem");
+ }
+ return fs;
+ }
+
+ /**
+ * Converts null to empty string
+ * @param raw The raw object
+ * @return Empty string if null, or the String value if not
+ */
+ public static String convertNullToEmptyString(Object raw) {
+ return raw == null ? "" : String.valueOf(raw);
+ }
+}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/asn/GeoLiteAsnDatabase.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/asn/GeoLiteAsnDatabase.java
new file mode 100644
index 0000000..d37cb22
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/asn/GeoLiteAsnDatabase.java
@@ -0,0 +1,164 @@
+package org.apache.metron.enrichment.adapters.maxmind.asn;/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import com.maxmind.geoip2.DatabaseReader;
+import com.maxmind.geoip2.exception.AddressNotFoundException;
+import com.maxmind.geoip2.exception.GeoIp2Exception;
+import com.maxmind.geoip2.model.AsnResponse;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Function;
+import org.apache.metron.enrichment.adapters.maxmind.MaxMindDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.MaxMindDbUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manages querying and updating of an Autonomous System Number (ASN) database provided by MaxMind.
+ */
+public enum GeoLiteAsnDatabase implements MaxMindDatabase {
+ INSTANCE;
+
+ protected static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final String ASN_HDFS_FILE = "asn.hdfs.file";
+ public static final String ASN_HDFS_FILE_DEFAULT = "/apps/metron/asn/default/GeoLite2-ASN.tar.gz";
+
+ private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
+ private static final Lock readLock = lock.readLock();
+ private static final Lock writeLock = lock.writeLock();
+ private static volatile String hdfsLoc = ASN_HDFS_FILE_DEFAULT;
+ private static DatabaseReader reader = null;
+
+ public enum AsnProps {
+ NETWORK("network"),
+ ASN("autonomous_system_number"),
+ ASO("autonomous_system_organization");
+ Function<Map<String, Object>, Object> getter;
+ String simpleName;
+
+ AsnProps(String simpleName) {
+ this(simpleName, m -> m.get(simpleName));
+ }
+
+ AsnProps(String simpleName,
+ Function<Map<String, Object>, Object> getter
+ ) {
+ this.simpleName = simpleName;
+ this.getter = getter;
+ }
+
+ public String getSimpleName() {
+ return simpleName;
+ }
+
+ public Object get(Map<String, Object> map) {
+ return getter.apply(map);
+ }
+
+ public void set(Map<String, Object> map, Object val) {
+ map.put(simpleName, val);
+ }
+ }
+
+ @Override
+ public String getHdfsFileConfig() {
+ return ASN_HDFS_FILE;
+ }
+
+ @Override
+ public String getHdfsFileDefault() {
+ return ASN_HDFS_FILE_DEFAULT;
+ }
+
+ @Override
+ public void lockIfNecessary() {
+ writeLock.lock();
+ }
+
+ @Override
+ public void unlockIfNecessary() {
+ writeLock.unlock();
+ }
+
+ @Override
+ public DatabaseReader getReader() {
+ return reader;
+ }
+
+ @Override
+ public void setReader(DatabaseReader reader) {
+ GeoLiteAsnDatabase.reader = reader;
+ }
+
+ public synchronized void updateIfNecessary(Map<String, Object> globalConfig) {
+ // Reload database if necessary (file changes on HDFS)
+ LOG.trace("Determining if GeoLiteAsnDatabase update required");
+ String hdfsFile = ASN_HDFS_FILE_DEFAULT;
+ if (globalConfig != null) {
+ hdfsFile = (String) globalConfig.getOrDefault(ASN_HDFS_FILE, ASN_HDFS_FILE_DEFAULT);
+ }
+
+ // Always update if we don't have a DatabaseReader
+ if (reader == null || !hdfsLoc.equals(hdfsFile)) {
+ // Update
+ hdfsLoc = hdfsFile;
+ update(hdfsFile);
+ } else {
+ LOG.trace("Update to GeoLiteAsnDatabase unnecessary");
+ }
+ }
+
+ /**
+ * Retrieves the result fields based on the incoming IP address
+ * @param ip The IP to lookup in the database
+ * @return Optional.empty() if the IP address is invalid or not in the database.
+ */
+ public Optional<Map<String, Object>> get(String ip) {
+ if (MaxMindDbUtilities.invalidIp(ip)) {
+ return Optional.empty();
+ }
+
+ try {
+ readLock.lock();
+ InetAddress addr = InetAddress.getByName(ip);
+ AsnResponse asnResponse = reader.asn(addr);
+ HashMap<String, Object> asnInfo = new HashMap<>();
+ AsnProps.ASN.set(asnInfo, asnResponse.getAutonomousSystemNumber());
+ AsnProps.ASO
+ .set(asnInfo, MaxMindDbUtilities.convertNullToEmptyString(asnResponse.getAutonomousSystemOrganization()));
+ AsnProps.NETWORK
+ .set(asnInfo, MaxMindDbUtilities.convertNullToEmptyString(asnResponse.getIpAddress()));
+
+ return Optional.of(asnInfo);
+ } catch (UnknownHostException | AddressNotFoundException e) {
+ LOG.debug("No result found for IP {}", ip);
+ } catch (GeoIp2Exception | IOException e) {
+ LOG.warn("GeoLite2 ASN DB encountered an error", e);
+ } finally {
+ readLock.unlock();
+ }
+ return Optional.empty();
+ }
+}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabase.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java
similarity index 56%
rename from metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabase.java
rename to metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java
index f5d20f7..0be58d8 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabase.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabase.java
@@ -15,10 +15,9 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.metron.enrichment.adapters.geo;
+package org.apache.metron.enrichment.adapters.maxmind.geo;
import ch.hsr.geohash.WGS84Point;
-import com.maxmind.db.CHMCache;
import com.maxmind.geoip2.DatabaseReader;
import com.maxmind.geoip2.exception.AddressNotFoundException;
import com.maxmind.geoip2.exception.GeoIp2Exception;
@@ -37,29 +36,27 @@ import java.util.Optional;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.function.Function;
-import java.util.function.Supplier;
-import java.util.zip.GZIPInputStream;
-
-import org.apache.commons.lang3.StringUtils;
-import org.apache.commons.validator.routines.InetAddressValidator;
-import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
-import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.enrichment.adapters.maxmind.MaxMindDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.MaxMindDbUtilities;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public enum GeoLiteDatabase {
+/**
+ * Manages querying and updating of an GeoLite2 City database provided by MaxMind.
+ */
+public enum GeoLiteCityDatabase implements MaxMindDatabase {
INSTANCE;
protected static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static final String GEO_HDFS_FILE = "geo.hdfs.file";
- public static final String GEO_HDFS_FILE_DEFAULT = "/apps/metron/geo/default/GeoLite2-City.mmdb.gz";
+ public static final String GEO_HDFS_FILE_DEFAULT = "/apps/metron/geo/default/GeoLite2-City.tar.gz";
+ public static final String GEO_HDFS_FILE_DEFAULT_FALLBACK = "/apps/metron/geo/default/GeoLite2-City.mmdb.gz";
private static ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
private static final Lock readLock = lock.readLock();
private static final Lock writeLock = lock.writeLock();
- private static InetAddressValidator ipvalidator = new InetAddressValidator();
private static volatile String hdfsLoc = GEO_HDFS_FILE_DEFAULT;
private static DatabaseReader reader = null;
@@ -99,12 +96,43 @@ public enum GeoLiteDatabase {
}
}
+ @Override
+ public String getHdfsFileConfig() {
+ return GEO_HDFS_FILE;
+ }
+
+ @Override
+ public String getHdfsFileDefault() {
+ return GEO_HDFS_FILE_DEFAULT;
+ }
+
+ @Override
+ public void lockIfNecessary() {
+ writeLock.lock();
+ }
+
+ @Override
+ public void unlockIfNecessary() {
+ writeLock.unlock();
+ }
+
+ @Override
+ public DatabaseReader getReader() {
+ return reader;
+ }
+
+ @Override
+ public void setReader(DatabaseReader reader) {
+ GeoLiteCityDatabase.reader = reader;
+ }
+
public synchronized void updateIfNecessary(Map<String, Object> globalConfig) {
// Reload database if necessary (file changes on HDFS)
- LOG.trace("[Metron] Determining if GeoIpDatabase update required");
+ LOG.trace("Determining if GeoIpDatabase update required");
String hdfsFile = GEO_HDFS_FILE_DEFAULT;
if (globalConfig != null) {
hdfsFile = (String) globalConfig.getOrDefault(GEO_HDFS_FILE, GEO_HDFS_FILE_DEFAULT);
+ hdfsFile = determineHdfsDirWithFallback(globalConfig, hdfsFile, GEO_HDFS_FILE_DEFAULT_FALLBACK);
}
// Always update if we don't have a DatabaseReader
@@ -113,70 +141,49 @@ public enum GeoLiteDatabase {
hdfsLoc = hdfsFile;
update(hdfsFile);
} else {
- LOG.trace("[Metron] Update to GeoIpDatabase unnecessary");
+ LOG.trace("Update to GeoLiteCity2Database unnecessary");
}
}
- @SuppressWarnings("unchecked")
- public void update(String hdfsFile) {
- // If nothing is set (or it's been unset, use the defaults)
- if (hdfsFile == null || hdfsFile.isEmpty()) {
- LOG.debug("[Metron] Using default for {}: {}", GEO_HDFS_FILE, GEO_HDFS_FILE_DEFAULT);
- hdfsFile = GEO_HDFS_FILE_DEFAULT;
- }
-
- FileSystem fs;
- try {
- fs = FileSystem.get(new Configuration());
- } catch (IOException e) {
- LOG.error("[Metron] Unable to retrieve get HDFS FileSystem");
- throw new IllegalStateException("[Metron] Unable to get HDFS FileSystem");
- }
-
- try (GZIPInputStream gis = new GZIPInputStream(fs.open(new Path(hdfsFile)))) {
- writeLock.lock();
- LOG.info("[Metron] Update to GeoIP data started with {}", hdfsFile);
- // InputStream based DatabaseReaders are always in memory.
- DatabaseReader newReader = new DatabaseReader.Builder(gis).withCache(new CHMCache()).build();
- DatabaseReader oldReader = reader;
- reader = newReader;
- // If we've never set a reader, don't close the old one
- if (oldReader != null) {
- oldReader.close();
- }
- LOG.info("[Metron] Finished update to GeoIP data started with {}", hdfsFile);
- } catch (IOException e) {
- LOG.error("[Metron] Unable to open new database file {}", hdfsFile, e);
- throw new IllegalStateException("[Metron] Unable to update MaxMind database");
- } finally {
- // Don't unlock if the try failed
- if (lock.isWriteLocked()) {
- writeLock.unlock();
+ protected String determineHdfsDirWithFallback(Map<String, Object> globalConfig, String hdfsFile, String hdfsFallbackFile) {
+ // GeoLite2 City has the case where our new default isn't the old, but we want to fallback if needed.
+ // Only consider fallback if the user hasn't specified a location via config.
+ if (!globalConfig.containsKey(GEO_HDFS_FILE)) {
+ FileSystem fs = MaxMindDbUtilities.getFileSystem();
+ try {
+ // Want to fallback under two conditions here
+ // 1. The default file doesn't exist. If it wasn't in the global config, it has to be the default.
+ // 2. The fallback exists.
+ // Otherwise, we'll leave it as the base default (which will cause issues later, but ensures logging encourages use of new database).
+ // Note that hdfsFile will be GEO_HDFS_FILE_DEFAULT if we even made it here.
+ if (hdfsPathsExist(fs, hdfsFile, hdfsFallbackFile)) {
+ hdfsFile = hdfsFallbackFile;
+ }
+ } catch (IOException e) {
+ LOG.warn("Issue validating database HDFS fallback locations", e);
+ // Do nothing else
}
}
+ return hdfsFile;
}
- // Optional.empty means that we don't have any geo location in database.
- // Optional exists, but empty means local IP (valid, but no info will be in the DB)
- @SuppressWarnings("unchecked")
- public Optional<HashMap<String, String>> get(String ip) {
- // Call get every single time, returns current version. Updates behind the scenes.
- LOG.trace("[Metron] Called GeoIpDatabase.get({})", ip);
- InetAddress addr = null;
- try {
- addr = InetAddress.getByName(ip);
- } catch (UnknownHostException e) {
- LOG.warn("[Metron] No result found for IP {}", ip, e);
- return Optional.empty();
- }
- if (isIneligibleAddress(ip, addr)) {
- LOG.debug("[Metron] IP ineligible for GeoLite2 lookup {}", ip);
+ protected boolean hdfsPathsExist(FileSystem fs, String hdfsFile, String fallbackFile) throws IOException {
+ return !fs.exists(new Path(hdfsFile)) && fs.exists(new Path(fallbackFile));
+ }
+
+ /**
+ * Retrieves the result fields based on the incoming IP address
+ * @param ip The IP to lookup in the database
+ * @return Optional.empty() if the IP address is invalid or not in the database.
+ */
+ public Optional<Map<String, String>> get(String ip) {
+ if (MaxMindDbUtilities.invalidIp(ip)) {
return Optional.empty();
}
try {
readLock.lock();
- addr = InetAddress.getByName(ip);
+ InetAddress addr = InetAddress.getByName(ip);
CityResponse cityResponse = reader.city(addr);
HashMap<String, String> geoInfo = new HashMap<>();
@@ -185,18 +192,18 @@ public enum GeoLiteDatabase {
Postal postal = cityResponse.getPostal();
Location location = cityResponse.getLocation();
- GeoProps.LOC_ID.set(geoInfo, convertNullToEmptyString(city.getGeoNameId()));
- GeoProps.COUNTRY.set(geoInfo, convertNullToEmptyString(country.getIsoCode()));
- GeoProps.CITY.set(geoInfo, convertNullToEmptyString(city.getName()));
- GeoProps.POSTAL_CODE.set(geoInfo, convertNullToEmptyString(postal.getCode()));
- GeoProps.DMA_CODE.set(geoInfo, convertNullToEmptyString(location.getMetroCode()));
+ GeoProps.LOC_ID.set(geoInfo, MaxMindDbUtilities.convertNullToEmptyString(city.getGeoNameId()));
+ GeoProps.COUNTRY.set(geoInfo, MaxMindDbUtilities.convertNullToEmptyString(country.getIsoCode()));
+ GeoProps.CITY.set(geoInfo, MaxMindDbUtilities.convertNullToEmptyString(city.getName()));
+ GeoProps.POSTAL_CODE.set(geoInfo, MaxMindDbUtilities.convertNullToEmptyString(postal.getCode()));
+ GeoProps.DMA_CODE.set(geoInfo, MaxMindDbUtilities.convertNullToEmptyString(location.getMetroCode()));
Double latitudeRaw = location.getLatitude();
- String latitude = convertNullToEmptyString(latitudeRaw);
+ String latitude = MaxMindDbUtilities.convertNullToEmptyString(latitudeRaw);
GeoProps.LATITUDE.set(geoInfo, latitude);
Double longitudeRaw = location.getLongitude();
- String longitude = convertNullToEmptyString(longitudeRaw);
+ String longitude = MaxMindDbUtilities.convertNullToEmptyString(longitudeRaw);
GeoProps.LONGITUDE.set(geoInfo, longitude);
if (latitudeRaw == null || longitudeRaw == null) {
@@ -207,9 +214,9 @@ public enum GeoLiteDatabase {
return Optional.of(geoInfo);
} catch (UnknownHostException | AddressNotFoundException e) {
- LOG.debug("[Metron] No result found for IP {}", ip);
+ LOG.debug("No result found for IP {}", ip);
} catch (GeoIp2Exception | IOException e) {
- LOG.warn("[Metron] GeoLite2 DB encountered an error", e);
+ LOG.warn("GeoLite2 City DB encountered an error", e);
} finally {
readLock.unlock();
}
@@ -224,22 +231,12 @@ public enum GeoLiteDatabase {
}
try {
- double latD = Double.parseDouble(latitude.toString());
- double longD = Double.parseDouble(longitude.toString());
+ double latD = Double.parseDouble(latitude);
+ double longD = Double.parseDouble(longitude);
return Optional.of(new WGS84Point(latD, longD));
} catch (NumberFormatException nfe) {
LOG.warn(String.format("Invalid lat/long: %s/%s: %s", latitude, longitude, nfe.getMessage()), nfe);
return Optional.empty();
}
}
-
- protected String convertNullToEmptyString(Object raw) {
- return raw == null ? "" : String.valueOf(raw);
- }
-
- private boolean isIneligibleAddress(String ipStr, InetAddress addr) {
- return addr.isAnyLocalAddress() || addr.isLoopbackAddress()
- || addr.isSiteLocalAddress() || addr.isMulticastAddress()
- || !ipvalidator.isValidInet4Address(ipStr);
- }
}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/DistanceStrategies.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/DistanceStrategies.java
similarity index 97%
rename from metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/DistanceStrategies.java
rename to metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/DistanceStrategies.java
index 6af214e..d0dd7d2 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/DistanceStrategies.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/DistanceStrategies.java
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.metron.enrichment.adapters.geo.hash;
+package org.apache.metron.enrichment.adapters.maxmind.geo.hash;
import ch.hsr.geohash.WGS84Point;
import org.locationtech.spatial4j.distance.DistanceUtils;
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/DistanceStrategy.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/DistanceStrategy.java
similarity index 93%
rename from metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/DistanceStrategy.java
rename to metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/DistanceStrategy.java
index 0303986..9b9eac5 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/DistanceStrategy.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/DistanceStrategy.java
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.metron.enrichment.adapters.geo.hash;
+package org.apache.metron.enrichment.adapters.maxmind.geo.hash;
import ch.hsr.geohash.WGS84Point;
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/GeoHashUtil.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/GeoHashUtil.java
similarity index 97%
rename from metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/GeoHashUtil.java
rename to metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/GeoHashUtil.java
index 902eea3..10b9040 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/hash/GeoHashUtil.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/maxmind/geo/hash/GeoHashUtil.java
@@ -15,12 +15,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.metron.enrichment.adapters.geo.hash;
+package org.apache.metron.enrichment.adapters.maxmind.geo.hash;
import ch.hsr.geohash.GeoHash;
import ch.hsr.geohash.WGS84Point;
import com.google.common.collect.Iterables;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import java.util.AbstractMap;
import java.util.Map;
@@ -42,7 +42,7 @@ public enum GeoHashUtil {
}
public Optional<String> computeHash(Map<String, String> geoLoc, int precision) {
- Optional<WGS84Point> point = GeoLiteDatabase.INSTANCE.toPoint(geoLoc);
+ Optional<WGS84Point> point = GeoLiteCityDatabase.INSTANCE.toPoint(geoLoc);
if(point.isPresent()) {
return computeHash(point.get(), precision);
}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
index fcfa918..00c23ff 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBolt.java
@@ -25,7 +25,8 @@ import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
import org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.metron.common.utils.MessageUtils;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.enrichment.utils.ThreatIntelUtils;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.StellarFunctions;
@@ -73,7 +74,10 @@ public class ThreatIntelJoinBolt extends EnrichmentJoinBolt {
@Override
public void prepare(Map map, TopologyContext topologyContext) {
super.prepare(map, topologyContext);
- GeoLiteDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(GeoLiteDatabase.GEO_HDFS_FILE));
+ GeoLiteCityDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(
+ GeoLiteCityDatabase.GEO_HDFS_FILE));
+ GeoLiteAsnDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(
+ GeoLiteAsnDatabase.ASN_HDFS_FILE));
initializeStellar();
}
@@ -111,7 +115,7 @@ public class ThreatIntelJoinBolt extends EnrichmentJoinBolt {
public void reloadCallback(String name, ConfigurationType type) {
super.reloadCallback(name, type);
if(type == ConfigurationType.GLOBAL) {
- GeoLiteDatabase.INSTANCE.updateIfNecessary(getConfigurations().getGlobalConfig());
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(getConfigurations().getGlobalConfig());
}
}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
index c258fb0..c81251e 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java
@@ -27,7 +27,8 @@ import org.apache.metron.common.message.MessageGetters;
import org.apache.metron.common.performance.PerformanceLogger;
import org.apache.metron.common.utils.ErrorUtils;
import org.apache.metron.common.utils.MessageUtils;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.enrichment.configuration.Enrichment;
import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
import org.apache.metron.enrichment.parallel.EnrichmentContext;
@@ -356,7 +357,10 @@ public class UnifiedEnrichmentBolt extends ConfiguredEnrichmentBolt {
messageGetter = this.getterStrategy.get(messageFieldName);
enricher = new ParallelEnricher(enrichmentsByType, ConcurrencyContext.get(strategy), captureCacheStats);
perfLog = new PerformanceLogger(() -> getConfigurations().getGlobalConfig(), Perf.class.getName());
- GeoLiteDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(GeoLiteDatabase.GEO_HDFS_FILE));
+ GeoLiteCityDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(
+ GeoLiteCityDatabase.GEO_HDFS_FILE));
+ GeoLiteAsnDatabase.INSTANCE.update((String)getConfigurations().getGlobalConfig().get(
+ GeoLiteAsnDatabase.ASN_HDFS_FILE));
initializeStellar();
enrichmentContext = new EnrichmentContext(StellarFunctions.FUNCTION_RESOLVER(), stellarContext);
}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/AsnEnrichmentFunctions.java
similarity index 56%
copy from metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
copy to metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/AsnEnrichmentFunctions.java
index aa40250..7387b63 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/AsnEnrichmentFunctions.java
@@ -23,7 +23,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.ParseException;
import org.apache.metron.stellar.dsl.Stellar;
@@ -31,59 +31,62 @@ import org.apache.metron.stellar.dsl.StellarFunction;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public class GeoEnrichmentFunctions {
+public class AsnEnrichmentFunctions {
+
private static final Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- @Stellar(name="GET"
- ,namespace="GEO"
- ,description="Look up an IPV4 address and returns geographic information about it"
- ,params = {
- "ip - The IPV4 address to lookup",
- "fields - Optional list of GeoIP fields to grab. Options are locID, country, city, postalCode, dmaCode, latitude, longitude, location_point"
- }
- ,returns = "If a Single field is requested a string of the field, If multiple fields a map of string of the fields, and null otherwise"
+ @Stellar(name = "GET"
+ , namespace = "ASN"
+ , description = "Look up an IPV4 address and returns Autonomous System Number information about it"
+ , params = {
+ "ip - The IPV4 address to lookup",
+ "fields - Optional list of ASN fields to grab. Options are autonomous_system_organization, autonomous_system_number, network"
+ }
+ , returns = "If a single field is requested a string of the field, If multiple fields a map of string of the fields, and null otherwise"
)
- public static class GeoGet implements StellarFunction {
+ public static class AsnGet implements StellarFunction {
+
boolean initialized = false;
@Override
public Object apply(List<Object> args, Context context) throws ParseException {
- if(!initialized) {
+ if (!initialized) {
return null;
}
- if(args.size() > 2) {
- throw new IllegalArgumentException("GEO_GET received more arguments than expected: " + args.size());
+ if (args.size() > 2) {
+ throw new IllegalArgumentException(
+ "ASN_GET received more arguments than expected: " + args.size());
}
- if(args.size() == 1 && args.get(0) instanceof String) {
+ if (args.size() == 1 && args.get(0) instanceof String) {
// If no fields are provided, return everything
String ip = (String) args.get(0);
- if(ip == null || ip.trim().isEmpty()) {
+ if (ip == null || ip.trim().isEmpty()) {
+ LOG.debug("No IP provided, returning null");
return null;
}
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(ip);
- if(result.isPresent()) {
- return result.get();
- } else {
- return Collections.EMPTY_MAP;
- }
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(ip);
+ return result.orElse(Collections.EMPTY_MAP);
} else if (args.size() == 2 && args.get(1) instanceof List) {
// If fields are provided, return just those fields.
String ip = (String) args.get(0);
@SuppressWarnings("unchecked")
List<String> fields = (List) args.get(1);
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(ip);
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(ip);
// If only one field is requested, just return it directly
- if(fields.size() == 1 && result.isPresent()) {
+ if (fields.size() == 1 && result.isPresent()) {
+ if (!result.get().containsKey(fields.get(0))) {
+ return null;
+ }
return result.get().get(fields.get(0));
} else if (result.isPresent()) {
// If multiple fields are requested, return all of them
- HashMap<String, String> filteredInfo = new HashMap<>();
- for(String field : fields) {
- HashMap<String, String> geoInfo = result.get();
- filteredInfo.put(field, geoInfo.get(field));
+ Map<String, Object> filteredInfo = new HashMap<>();
+ for (String field : fields) {
+ Map<String, Object> asnInfo = result.get();
+ filteredInfo.put(field, asnInfo.get(field));
}
return filteredInfo;
}
@@ -94,15 +97,17 @@ public class GeoEnrichmentFunctions {
@Override
public void initialize(Context context) {
- LOG.info("Initializing GeoEnrichmentFunctions");
- Map<String, Object> config = getConfig(context);
- String hdfsDir = (String) config.get(GeoLiteDatabase.GEO_HDFS_FILE);
- GeoLiteDatabase.INSTANCE.update(hdfsDir);
- initialized = true;
+ LOG.info("Initializing AsnEnrichmentFunctions");
+ Map<String, Object> config = getConfig(context);
+ String hdfsDir = (String) config.get(GeoLiteAsnDatabase.ASN_HDFS_FILE);
+ GeoLiteAsnDatabase.INSTANCE.update(hdfsDir);
+ initialized = true;
}
+ @SuppressWarnings("unchecked")
private static Map<String, Object> getConfig(Context context) {
- return (Map<String, Object>) context.getCapability(Context.Capabilities.GLOBAL_CONFIG, false).orElse(new HashMap<>());
+ return (Map<String, Object>) context.getCapability(Context.Capabilities.GLOBAL_CONFIG, false)
+ .orElse(new HashMap<>());
}
@Override
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
index aa40250..8204ede 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctions.java
@@ -23,7 +23,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.ParseException;
import org.apache.metron.stellar.dsl.Stellar;
@@ -62,27 +62,23 @@ public class GeoEnrichmentFunctions {
return null;
}
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(ip);
- if(result.isPresent()) {
- return result.get();
- } else {
- return Collections.EMPTY_MAP;
- }
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(ip);
+ return result.orElse(Collections.emptyMap());
} else if (args.size() == 2 && args.get(1) instanceof List) {
// If fields are provided, return just those fields.
String ip = (String) args.get(0);
@SuppressWarnings("unchecked")
List<String> fields = (List) args.get(1);
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(ip);
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(ip);
// If only one field is requested, just return it directly
if(fields.size() == 1 && result.isPresent()) {
return result.get().get(fields.get(0));
} else if (result.isPresent()) {
// If multiple fields are requested, return all of them
- HashMap<String, String> filteredInfo = new HashMap<>();
+ Map<String, String> filteredInfo = new HashMap<>();
for(String field : fields) {
- HashMap<String, String> geoInfo = result.get();
+ Map<String, String> geoInfo = result.get();
filteredInfo.put(field, geoInfo.get(field));
}
return filteredInfo;
@@ -96,8 +92,8 @@ public class GeoEnrichmentFunctions {
public void initialize(Context context) {
LOG.info("Initializing GeoEnrichmentFunctions");
Map<String, Object> config = getConfig(context);
- String hdfsDir = (String) config.get(GeoLiteDatabase.GEO_HDFS_FILE);
- GeoLiteDatabase.INSTANCE.update(hdfsDir);
+ String hdfsDir = (String) config.get(GeoLiteCityDatabase.GEO_HDFS_FILE);
+ GeoLiteCityDatabase.INSTANCE.update(hdfsDir);
initialized = true;
}
diff --git a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoHashFunctions.java b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoHashFunctions.java
index a1e64c5..99c99cd 100644
--- a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoHashFunctions.java
+++ b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/GeoHashFunctions.java
@@ -18,10 +18,10 @@
package org.apache.metron.enrichment.stellar;
import ch.hsr.geohash.WGS84Point;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
-import org.apache.metron.enrichment.adapters.geo.hash.DistanceStrategies;
-import org.apache.metron.enrichment.adapters.geo.hash.DistanceStrategy;
-import org.apache.metron.enrichment.adapters.geo.hash.GeoHashUtil;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.hash.DistanceStrategies;
+import org.apache.metron.enrichment.adapters.maxmind.geo.hash.DistanceStrategy;
+import org.apache.metron.enrichment.adapters.maxmind.geo.hash.GeoHashUtil;
import org.apache.metron.stellar.common.utils.ConversionUtils;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.ParseException;
@@ -62,8 +62,8 @@ public class GeoHashFunctions {
Optional<WGS84Point> point = GeoHashUtil.INSTANCE.toPoint(hash);
if(point.isPresent()) {
Map<String, Object> ret = new HashMap<>();
- ret.put(GeoLiteDatabase.GeoProps.LONGITUDE.getSimpleName(), point.get().getLongitude());
- ret.put(GeoLiteDatabase.GeoProps.LATITUDE.getSimpleName(), point.get().getLatitude());
+ ret.put(GeoLiteCityDatabase.GeoProps.LONGITUDE.getSimpleName(), point.get().getLongitude());
+ ret.put(GeoLiteCityDatabase.GeoProps.LATITUDE.getSimpleName(), point.get().getLatitude());
return ret;
}
return null;
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabaseTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabaseTest.java
deleted file mode 100644
index 9a29e3d..0000000
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabaseTest.java
+++ /dev/null
@@ -1,190 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- * <p>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.metron.enrichment.adapters.geo;
-
-import com.google.common.collect.ImmutableMap;
-import org.adrianwalker.multilinestring.Multiline;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.test.utils.UnitTestHelper;
-import org.json.simple.JSONObject;
-import org.json.simple.parser.JSONParser;
-import org.json.simple.parser.ParseException;
-import org.junit.*;
-import org.junit.rules.TemporaryFolder;
-
-import java.io.File;
-import java.io.IOException;
-import java.util.HashMap;
-import java.util.Optional;
-
-public class GeoLiteDatabaseTest {
- private static Context context;
- private static File geoHdfsFile;
- private static File geoHdfsFile_update;
- private static final String IP_WITH_DMA = "81.2.69.192";
- private static final String IP_NO_DMA = "216.160.83.56";
-
- /**
- * {
- * "locID":"5803556",
- * "country":"US",
- * "city":"Milton",
- * "postalCode":"98354",
- * "latitude":"47.2513",
- * "longitude":"-122.3149",
- * "dmaCode":"819",
- * "location_point":"47.2513,-122.3149"
- * }
- */
- @Multiline
- private static String expectedNoDmaMessageString;
-
- private static JSONObject expectedNoDmaMessage;
-
- /**
- * {
- * "locID":"2643743",
- * "country":"GB",
- * "city":"London",
- * "postalCode":"",
- * "latitude":"51.5142",
- * "longitude":"-0.0931",
- * "dmaCode":"",
- * "location_point":"51.5142,-0.0931"
- * }
- */
- @Multiline
- private static String expectedDmaMessageString;
-
- private static JSONObject expectedDmaMessage;
-
- private static FileSystem fs;
-
- @Rule
- public TemporaryFolder testFolder = new TemporaryFolder();
-
- @BeforeClass
- public static void setupOnce() throws ParseException, IOException {
- JSONParser jsonParser = new JSONParser();
- expectedNoDmaMessage = (JSONObject) jsonParser.parse(expectedNoDmaMessageString);
- expectedDmaMessage = (JSONObject) jsonParser.parse(expectedDmaMessageString);
-
- String baseDir = UnitTestHelper.findDir("GeoLite");
- geoHdfsFile = new File(new File(baseDir), "GeoIP2-City-Test.mmdb.gz");
- geoHdfsFile_update = new File(new File(baseDir), "GeoIP2-City-Test-2.mmdb.gz");
-
- Configuration config = new Configuration();
- fs = FileSystem.get(config);
- }
-
- @Before
- public void setup() throws Exception {
- testFolder.create();
- context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG
- , () -> ImmutableMap.of(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath())
- )
- .build();
- }
-
- @Test
- public void testGetLocal() throws Exception {
- GeoLiteDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
-
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get("192.168.0.1");
- Assert.assertFalse("Local address result should be empty", result.isPresent());
- }
-
- @Test
- public void testExternalAddressNotFound() throws Exception {
- GeoLiteDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
-
- // the range 203.0.113.0/24 is assigned as "TEST-NET-3" and should never be locatable
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get("203.0.113.1");
- Assert.assertFalse("External address not found", result.isPresent());
- }
-
- @Test
- @SuppressWarnings("unchecked")
- public void testGetRemoteWithDma() throws Exception {
- GeoLiteDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
-
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(IP_WITH_DMA);
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedDmaMessage, result.get());
- }
-
- @Test
- @SuppressWarnings("unchecked")
- public void testGetRemoteNoDma() throws Exception {
- GeoLiteDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
-
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(IP_NO_DMA);
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
- }
-
- @Test
- @SuppressWarnings("unchecked")
- public void testMultipleUpdates() throws Exception {
- GeoLiteDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
- GeoLiteDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
-
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(IP_NO_DMA);
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
- }
-
- @Test
- @SuppressWarnings("unchecked")
- public void testUpdateIfNecessary() throws Exception {
- HashMap<String, Object> globalConfig = new HashMap<>();
- globalConfig.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
- GeoLiteDatabase.INSTANCE.updateIfNecessary(globalConfig);
-
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(IP_NO_DMA);
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
- }
-
-
- @Test
- @SuppressWarnings("unchecked")
- public void testMultipleUpdateIfNecessary() throws Exception {
- HashMap<String, Object> globalConfig = new HashMap<>();
- globalConfig.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
- GeoLiteDatabase.INSTANCE.updateIfNecessary(globalConfig);
- GeoLiteDatabase.INSTANCE.updateIfNecessary(globalConfig);
-
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(IP_NO_DMA);
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
- }
-
- @Test
- @SuppressWarnings("unchecked")
- public void testDifferingUpdateIfNecessary() throws Exception {
- HashMap<String, Object> globalConfig = new HashMap<>();
- globalConfig.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
- GeoLiteDatabase.INSTANCE.updateIfNecessary(globalConfig);
- Optional<HashMap<String, String>> result = GeoLiteDatabase.INSTANCE.get(IP_NO_DMA);
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
-
- globalConfig.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile_update.getAbsolutePath());
- GeoLiteDatabase.INSTANCE.updateIfNecessary(globalConfig);
- result = GeoLiteDatabase.INSTANCE.get(IP_NO_DMA);
-
- Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
- }
-}
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/asn/GeoLiteAsnDatabaseTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/asn/GeoLiteAsnDatabaseTest.java
new file mode 100644
index 0000000..e572b97
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/asn/GeoLiteAsnDatabaseTest.java
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.adapters.maxmind.asn;
+
+import static org.apache.metron.enrichment.adapters.maxmind.MaxMindDatabase.EXTENSION_TAR_GZ;
+
+import com.google.common.collect.ImmutableMap;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.commons.io.FileUtils;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.test.utils.UnitTestHelper;
+import org.json.simple.JSONObject;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class GeoLiteAsnDatabaseTest {
+
+ private static Context context;
+ private static File asnHdfsFile;
+ private static File asnHdfsFile_update;
+ private static final String IP_ADDR = "8.8.4.0";
+ private static final String GEO_ASN = "GeoLite2-ASN";
+ private static final String GEO_ASN_FILE_NAME = GEO_ASN + EXTENSION_TAR_GZ;
+ private static final String GEO_ASN_COPY_FILE_NAME = GEO_ASN + "-2" + EXTENSION_TAR_GZ;
+
+ private static JSONObject expectedAsnMessage = new JSONObject();
+
+ @Rule
+ public TemporaryFolder testFolder = new TemporaryFolder();
+
+ @BeforeClass
+ @SuppressWarnings("unchecked")
+ public static void setupOnce() throws IOException {
+ // Construct this explicitly here, otherwise it'll be a Long instead of Integer.
+ expectedAsnMessage.put("autonomous_system_organization", "Google LLC");
+ expectedAsnMessage.put("autonomous_system_number", 15169);
+ expectedAsnMessage.put("network", "8.8.4.0");
+
+ String baseDir = UnitTestHelper.findDir("GeoLite");
+ asnHdfsFile = new File(new File(baseDir), GEO_ASN_FILE_NAME);
+ asnHdfsFile_update = new File(new File(baseDir), GEO_ASN_COPY_FILE_NAME);
+ FileUtils.copyFile(asnHdfsFile, asnHdfsFile_update);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ FileUtils.deleteQuietly(asnHdfsFile_update);
+ }
+
+ @Before
+ public void setup() throws Exception {
+ testFolder.create();
+ context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG,
+ () -> ImmutableMap.of(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath())
+ ).build();
+ }
+
+ @Test
+ public void testGetLocal() {
+ GeoLiteAsnDatabase.INSTANCE.update(asnHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get("192.168.0.1");
+ Assert.assertFalse("Local address result should be empty", result.isPresent());
+ }
+
+ @Test
+ public void testExternalAddressNotFound() {
+ GeoLiteAsnDatabase.INSTANCE.update(asnHdfsFile.getAbsolutePath());
+
+ // the range 203.0.113.0/24 is assigned as "TEST-NET-3" and should never be locatable
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get("203.0.113.1");
+ Assert.assertFalse("External address not found", result.isPresent());
+ }
+
+ @Test
+ public void testGetRemote() {
+ GeoLiteAsnDatabase.INSTANCE.update(asnHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(IP_ADDR);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedAsnMessage,
+ result.get());
+ }
+
+ @Test
+ public void testMultipleUpdates() {
+ GeoLiteAsnDatabase.INSTANCE.update(asnHdfsFile.getAbsolutePath());
+ GeoLiteAsnDatabase.INSTANCE.update(asnHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(IP_ADDR);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedAsnMessage,
+ result.get());
+ }
+
+ @Test
+ public void testUpdateIfNecessary() {
+ HashMap<String, Object> globalConfig = new HashMap<>();
+ globalConfig.put(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath());
+ GeoLiteAsnDatabase.INSTANCE.updateIfNecessary(globalConfig);
+
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(IP_ADDR);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedAsnMessage,
+ result.get());
+ }
+
+ @Test
+ public void testMultipleUpdateIfNecessary() {
+ HashMap<String, Object> globalConfig = new HashMap<>();
+ globalConfig.put(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath());
+ GeoLiteAsnDatabase.INSTANCE.updateIfNecessary(globalConfig);
+ GeoLiteAsnDatabase.INSTANCE.updateIfNecessary(globalConfig);
+
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(IP_ADDR);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedAsnMessage,
+ result.get());
+ }
+
+ @Test
+ public void testDifferingUpdateIfNecessary() {
+ HashMap<String, Object> globalConfig = new HashMap<>();
+ globalConfig.put(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath());
+ GeoLiteAsnDatabase.INSTANCE.updateIfNecessary(globalConfig);
+ Optional<Map<String, Object>> result = GeoLiteAsnDatabase.INSTANCE.get(IP_ADDR);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedAsnMessage,
+ result.get());
+
+ globalConfig.put(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile_update.getAbsolutePath());
+ GeoLiteAsnDatabase.INSTANCE.updateIfNecessary(globalConfig);
+ result = GeoLiteAsnDatabase.INSTANCE.get(IP_ADDR);
+
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedAsnMessage,
+ result.get());
+ }
+}
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/geo/GeoAdapterTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoAdapterTest.java
similarity index 89%
rename from metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/geo/GeoAdapterTest.java
rename to metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoAdapterTest.java
index 8d2a7ec..4355f91 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/geo/GeoAdapterTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoAdapterTest.java
@@ -15,10 +15,11 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.metron.enrichment.adapters.geo;
+package org.apache.metron.enrichment.adapters.maxmind.geo;
import com.google.common.collect.ImmutableMap;
import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.enrichment.adapters.geo.GeoAdapter;
import org.apache.metron.enrichment.bolt.CacheKey;
import org.apache.metron.test.utils.UnitTestHelper;
import org.json.simple.JSONObject;
@@ -29,7 +30,6 @@ import org.junit.BeforeClass;
import org.junit.Test;
import java.io.File;
-import java.util.Map;
public class GeoAdapterTest {
private static final String IP = "216.160.83.56";
@@ -60,10 +60,10 @@ public class GeoAdapterTest {
expectedMessage = (JSONObject) jsonParser.parse(expectedMessageString);
String baseDir = UnitTestHelper.findDir("GeoLite");
- geoHdfsFile = new File(new File(baseDir), "GeoIP2-City-Test.mmdb.gz");
+ geoHdfsFile = new File(new File(baseDir), "GeoLite2-City.mmdb.gz");
geo = new GeoAdapter();
- geo.initializeAdapter(ImmutableMap.of(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath()));
+ geo.initializeAdapter(ImmutableMap.of(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath()));
}
@Test
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabaseTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabaseTest.java
new file mode 100644
index 0000000..3ce96d7
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/adapters/maxmind/geo/GeoLiteCityDatabaseTest.java
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.adapters.maxmind.geo;
+
+import static org.apache.metron.enrichment.adapters.maxmind.MaxMindDatabase.EXTENSION_MMDB_GZ;
+import static org.apache.metron.enrichment.adapters.maxmind.MaxMindDatabase.EXTENSION_TAR_GZ;
+
+import com.google.common.collect.ImmutableMap;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import org.adrianwalker.multilinestring.Multiline;
+import org.apache.commons.io.FileUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.test.utils.UnitTestHelper;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class GeoLiteCityDatabaseTest {
+
+ private static Context context;
+ private static File geoHdfsFile;
+ private static File geoHdfsFileTarGz;
+ private static File geoHdfsFile_update;
+ private static final String IP_WITH_DMA = "81.2.69.192";
+ private static final String IP_NO_DMA = "216.160.83.56";
+ private static final String GEO_CITY = "GeoLite2-City";
+ private static final String GEO_CITY_FILE_NAME = GEO_CITY + EXTENSION_MMDB_GZ;
+ private static final String GEO_CITY_COPY_FILE_NAME = GEO_CITY + "-2" + EXTENSION_MMDB_GZ;
+
+ /**
+ * {
+ * "locID":"5803556",
+ * "country":"US",
+ * "city":"Milton",
+ * "postalCode":"98354",
+ * "latitude":"47.2513",
+ * "longitude":"-122.3149",
+ * "dmaCode":"819",
+ * "location_point":"47.2513,-122.3149"
+ * }
+ */
+ @Multiline
+ private static String expectedNoDmaMessageString;
+ private static JSONObject expectedNoDmaMessage;
+
+ /**
+ * {
+ * "locID":"2643743",
+ * "country":"GB",
+ * "city":"London",
+ * "postalCode":"",
+ * "latitude":"51.5142",
+ * "longitude":"-0.0931",
+ * "dmaCode":"",
+ * "location_point":"51.5142,-0.0931"
+ * }
+ */
+ @Multiline
+ private static String expectedDmaMessageString;
+ private static JSONObject expectedDmaMessage;
+
+
+ /**
+ * {
+ * "locID":"2640894",
+ * "country":"GB",
+ * "city":"Orpington",
+ * "postalCode":"BR6",
+ * "latitude":"51.3581",
+ * "longitude":"0.1277",
+ * "dmaCode":"",
+ * "location_point":"51.3581,0.1277"
+ * }
+ */
+ @Multiline
+ private static String expectedMessageStringTarGz;
+ private static JSONObject expectedMessageTarGz;
+
+ private static FileSystem fs;
+
+ @Rule
+ public TemporaryFolder testFolder = new TemporaryFolder();
+
+ @BeforeClass
+ public static void setupOnce() throws ParseException, IOException {
+ JSONParser jsonParser = new JSONParser();
+ expectedNoDmaMessage = (JSONObject) jsonParser.parse(expectedNoDmaMessageString);
+ expectedDmaMessage = (JSONObject) jsonParser.parse(expectedDmaMessageString);
+ expectedMessageTarGz = (JSONObject) jsonParser.parse(expectedMessageStringTarGz);
+
+ String baseDir = UnitTestHelper.findDir("GeoLite");
+ geoHdfsFile = new File(new File(baseDir), GEO_CITY_FILE_NAME);
+ geoHdfsFile_update = new File(new File(baseDir), GEO_CITY_COPY_FILE_NAME);
+ FileUtils.copyFile(geoHdfsFile, geoHdfsFile_update);
+ geoHdfsFileTarGz = new File(new File(baseDir), GEO_CITY + EXTENSION_TAR_GZ);
+
+ Configuration config = new Configuration();
+ fs = FileSystem.get(config);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ FileUtils.deleteQuietly(geoHdfsFile_update);
+ }
+
+ @Before
+ public void setup() throws Exception {
+ testFolder.create();
+ context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG
+ , () -> ImmutableMap.of(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath())
+ )
+ .build();
+ }
+
+ @Test
+ public void testGetLocal() {
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get("192.168.0.1");
+ Assert.assertFalse("Local address result should be empty", result.isPresent());
+ }
+
+ @Test
+ public void testExternalAddressNotFound() {
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
+
+ // the range 203.0.113.0/24 is assigned as "TEST-NET-3" and should never be locatable
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get("203.0.113.1");
+ Assert.assertFalse("External address not found", result.isPresent());
+ }
+
+ @Test
+ public void testGetRemoteWithDma() {
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_WITH_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedDmaMessage, result.get());
+ }
+
+ @Test
+ public void testGetRemoteWithTarGzFile() {
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFileTarGz.getAbsolutePath());
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_WITH_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedMessageTarGz, result.get());
+ }
+
+ @Test
+ public void testGetRemoteNoDma() {
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_NO_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
+ }
+
+ @Test
+ public void testMultipleUpdates() {
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
+ GeoLiteCityDatabase.INSTANCE.update(geoHdfsFile.getAbsolutePath());
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_NO_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
+ }
+
+ @Test
+ public void testUpdateIfNecessary() {
+ HashMap<String, Object> globalConfig = new HashMap<>();
+ globalConfig.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(globalConfig);
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_NO_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
+ }
+
+
+ @Test
+ public void testMultipleUpdateIfNecessary() {
+ HashMap<String, Object> globalConfig = new HashMap<>();
+ globalConfig.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(globalConfig);
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(globalConfig);
+
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_NO_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
+ }
+
+ @Test
+ public void testDifferingUpdateIfNecessary() {
+ HashMap<String, Object> globalConfig = new HashMap<>();
+ globalConfig.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(globalConfig);
+ Optional<Map<String, String>> result = GeoLiteCityDatabase.INSTANCE.get(IP_NO_DMA);
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
+
+ globalConfig.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile_update.getAbsolutePath());
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(globalConfig);
+ result = GeoLiteCityDatabase.INSTANCE.get(IP_NO_DMA);
+
+ Assert.assertEquals("Remote Local IP should return result based on DB", expectedNoDmaMessage, result.get());
+ }
+
+ @Test
+ public void testFallbackUnnecessary() {
+ String fakeFile = "fakefile.geolitecitydbtest";
+ Map<String, Object> globalConfig = Collections.singletonMap(GeoLiteCityDatabase.GEO_HDFS_FILE, fakeFile);
+ Assert.assertEquals(
+ GeoLiteCityDatabase.INSTANCE.determineHdfsDirWithFallback(globalConfig, fakeFile, ""),
+ fakeFile);
+ }
+
+ @Test
+ public void testFallbackUnncessaryAlreadyDefault() {
+ String defaultFile = GeoLiteCityDatabase.GEO_HDFS_FILE_DEFAULT;
+ Map<String, Object> globalConfig = Collections.singletonMap(GeoLiteCityDatabase.GEO_HDFS_FILE, defaultFile);
+ Assert.assertEquals(
+ GeoLiteCityDatabase.INSTANCE.determineHdfsDirWithFallback(globalConfig, defaultFile, ""),
+ defaultFile);
+ }
+
+ @Test
+ public void testFallbackToDefault() {
+ String defaultFile = GeoLiteCityDatabase.GEO_HDFS_FILE_DEFAULT;
+ Assert.assertEquals(GeoLiteCityDatabase.INSTANCE.determineHdfsDirWithFallback(Collections.emptyMap(), defaultFile, "fallback"), defaultFile);
+ }
+
+ @Test
+ public void testFallbackToOldDefault() throws IOException {
+ String fakeFile = "fakefile.geolitecitydbtest";
+ File file = File.createTempFile( this.getClass().getSimpleName(), "testfile");
+ file.deleteOnExit();
+ String fileName = file.getAbsolutePath();
+ Assert.assertEquals(GeoLiteCityDatabase.INSTANCE.determineHdfsDirWithFallback(Collections.emptyMap(), fakeFile, fileName), fileName);
+ }
+}
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBoltTest.java
index 17a53f4..aa9768c 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBoltTest.java
@@ -17,7 +17,6 @@
*/
package org.apache.metron.enrichment.bolt;
-import com.google.common.cache.CacheLoader;
import com.google.common.collect.ImmutableMap;
import org.adrianwalker.multilinestring.Multiline;
import org.apache.log4j.Level;
@@ -26,7 +25,7 @@ import org.apache.metron.common.Constants;
import org.apache.metron.common.configuration.ConfigurationsUtils;
import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
import org.apache.metron.common.error.MetronError;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.enrichment.configuration.Enrichment;
import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
@@ -166,8 +165,8 @@ public class GenericEnrichmentBoltTest extends BaseEnrichmentBoltTest {
HashMap<String, Object> globalConfig = new HashMap<>();
String baseDir = UnitTestHelper.findDir("GeoLite");
- File geoHdfsFile = new File(new File(baseDir), "GeoIP2-City-Test.mmdb.gz");
- globalConfig.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ File geoHdfsFile = new File(new File(baseDir), "GeoLite2-City.mmdb.gz");
+ globalConfig.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
genericEnrichmentBolt.getConfigurations().updateGlobalConfig(globalConfig);
try {
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
index 62b2570..03e9558 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/bolt/ThreatIntelJoinBoltTest.java
@@ -23,7 +23,8 @@ import org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
import org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig;
import org.apache.metron.common.message.MessageGetStrategy;
import org.apache.metron.common.utils.JSONUtils;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.test.bolt.BaseEnrichmentBoltTest;
import org.apache.metron.test.utils.UnitTestHelper;
import org.apache.storm.tuple.Tuple;
@@ -177,8 +178,10 @@ public class ThreatIntelJoinBoltTest extends BaseEnrichmentBoltTest {
threatIntelJoinBolt.getConfigurations().updateSensorEnrichmentConfig(sensorType, enrichmentConfig);
HashMap<String, Object> globalConfig = new HashMap<>();
String baseDir = UnitTestHelper.findDir("GeoLite");
- File geoHdfsFile = new File(new File(baseDir), "GeoIP2-City-Test.mmdb.gz");
- globalConfig.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ File geoHdfsFile = new File(new File(baseDir), "GeoLite2-City.mmdb.gz");
+ globalConfig.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ File asnHdfsFile = new File(new File(baseDir), "GeoLite2-ASN.tar.gz");
+ globalConfig.put(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath());
threatIntelJoinBolt.getConfigurations().updateGlobalConfig(globalConfig);
threatIntelJoinBolt.withMaxCacheSize(100);
threatIntelJoinBolt.withMaxTimeRetain(10000);
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
index 188f18b..743c31d 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
@@ -39,7 +39,8 @@ import org.apache.commons.lang3.StringUtils;
import org.apache.metron.TestConstants;
import org.apache.metron.common.Constants;
import org.apache.metron.common.utils.JSONUtils;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
import org.apache.metron.enrichment.converter.EnrichmentHelper;
import org.apache.metron.enrichment.converter.EnrichmentKey;
import org.apache.metron.enrichment.converter.EnrichmentValue;
@@ -94,6 +95,7 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
private final List<byte[]> inputMessages = getInputMessages(sampleParsedPath);
private static File geoHdfsFile;
+ private static File asnHdfsFile;
protected String fluxPath() {
return "../metron-enrichment/src/main/flux/enrichment/remote-splitjoin.yaml";
@@ -118,7 +120,8 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
@BeforeClass
public static void setupOnce() throws ParseException {
String baseDir = UnitTestHelper.findDir("GeoLite");
- geoHdfsFile = new File(new File(baseDir), "GeoIP2-City-Test.mmdb.gz");
+ geoHdfsFile = new File(new File(baseDir), "GeoLite2-City.mmdb.gz");
+ asnHdfsFile = new File(new File(baseDir), "GeoLite2-ASN.tar.gz");
}
/**
@@ -188,7 +191,8 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
config.put(SimpleHBaseEnrichmentFunctions.ACCESS_TRACKER_TYPE_CONF, "PERSISTENT_BLOOM");
config.put(PersistentBloomTrackerCreator.Config.PERSISTENT_BLOOM_TABLE, trackerHBaseTableName);
config.put(PersistentBloomTrackerCreator.Config.PERSISTENT_BLOOM_CF, cf);
- config.put(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ config.put(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath());
+ config.put(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath());
globalConfigStr = JSONUtils.INSTANCE.toJSON(config, true);
}
ConfigUploadComponent configUploadComponent = new ConfigUploadComponent()
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/AsnEnrichmentFunctionsTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/AsnEnrichmentFunctionsTest.java
new file mode 100644
index 0000000..84bd3bb
--- /dev/null
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/AsnEnrichmentFunctionsTest.java
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.enrichment.stellar;
+
+import com.google.common.collect.ImmutableMap;
+import java.io.File;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.metron.enrichment.adapters.maxmind.asn.GeoLiteAsnDatabase;
+import org.apache.metron.stellar.common.StellarProcessor;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.DefaultVariableResolver;
+import org.apache.metron.stellar.dsl.StellarFunctions;
+import org.apache.metron.test.utils.UnitTestHelper;
+import org.json.simple.JSONObject;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class AsnEnrichmentFunctionsTest {
+
+ private static Context context;
+ private static File asnHdfsFile;
+
+ private static JSONObject expectedMessage = new JSONObject();
+ private static JSONObject expectedSubsetMessage = new JSONObject();
+
+ @BeforeClass
+ @SuppressWarnings("unchecked")
+ public static void setupOnce() {
+ // Construct this explicitly here, otherwise it'll be a Long instead of Integer.
+ expectedMessage.put("autonomous_system_organization", "Google LLC");
+ expectedMessage.put("autonomous_system_number", 15169);
+ expectedMessage.put("network", "8.8.4.0");
+
+ expectedSubsetMessage.put("autonomous_system_organization", "Google LLC");
+ expectedSubsetMessage.put("autonomous_system_number", 15169);
+
+ String baseDir = UnitTestHelper.findDir("GeoLite");
+ asnHdfsFile = new File(new File(baseDir), "GeoLite2-ASN.tar.gz");
+ }
+
+ @Before
+ public void setup() throws Exception {
+ context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG,
+ () -> ImmutableMap.of(GeoLiteAsnDatabase.ASN_HDFS_FILE, asnHdfsFile.getAbsolutePath())
+ ).build();
+ }
+
+ public Object run(String rule, Map<String, Object> variables) {
+ StellarProcessor processor = new StellarProcessor();
+ Assert.assertTrue(rule + " not valid.", processor.validate(rule, context));
+ return processor.parse(rule,
+ new DefaultVariableResolver(variables::get, variables::containsKey),
+ StellarFunctions.FUNCTION_RESOLVER(), context);
+ }
+
+ @Test
+ public void testMissingDb() {
+ context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG,
+ () -> ImmutableMap.of(GeoLiteAsnDatabase.ASN_HDFS_FILE, "./fakefile.mmdb")
+ ).build();
+ String stellar = "ASN_GET()";
+ try {
+ run(stellar, ImmutableMap.of());
+ } catch (Exception expected) {
+ Assert.assertTrue(expected.getMessage().contains("File fakefile.mmdb does not exist"));
+ }
+ }
+
+ @Test
+ public void testMissingDbDuringUpdate() {
+ String stellar = "ASN_GET()";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Null IP should return null", result);
+ try {
+ GeoLiteAsnDatabase.INSTANCE.updateIfNecessary(
+ Collections.singletonMap(GeoLiteAsnDatabase.ASN_HDFS_FILE, "./fakefile.mmdb"));
+ } catch (IllegalStateException e) {
+ // ignore it, the file doesn't exist
+ }
+ // Should still continue to query the old database, instead of dying.
+ result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Null IP should return null", result);
+ }
+
+ @Test
+ public void testGetEmpty() {
+ String stellar = "ASN_GET()";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Empty IP should return null", result);
+ }
+
+ @Test
+ public void testGetNull() {
+ String stellar = "ASN_GET(null)";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Null IP should return null", result);
+ }
+
+ @Test(expected = org.apache.metron.stellar.dsl.ParseException.class)
+ public void testGetUndefined() {
+ String stellar = "ASN_GET(undefined)";
+ run(stellar, ImmutableMap.of());
+ }
+
+ @Test
+ public void testGetEmptyString() {
+ String stellar = "ASN_GET(' ')";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Empty IP should return null", result);
+ }
+
+ @Test
+ public void testGetLocal() {
+ String stellar = "ASN_GET('192.168.0.1')";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertEquals("Local IP should return empty map", new HashMap<String, String>(), result);
+ }
+
+ @Test
+ public void testGetRemote() throws Exception {
+ String stellar = "ASN_GET('8.8.4.0')";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertEquals("Remote IP should return result based on DB", expectedMessage, result);
+ }
+
+ @Test
+ public void testGetRemoteSingleField() throws Exception {
+ String stellar = "ASN_GET('8.8.4.0', ['autonomous_system_organization'])";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertEquals("Remote IP should return country result based on DB", "Google LLC", result);
+ }
+
+ @Test
+ public void testGetRemoteSingleFieldInteger() throws Exception {
+ String stellar = "ASN_GET('8.8.4.0', ['autonomous_system_number'])";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertEquals("Remote IP should return country result based on DB", 15169, result);
+ }
+
+ @Test
+ public void testGetRemoteMultipleFields() throws Exception {
+ String stellar = "ASN_GET('8.8.4.0', ['autonomous_system_organization', 'autonomous_system_number'])";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertEquals("Remote IP should return country result based on DB", expectedSubsetMessage,
+ result);
+ }
+
+ @Test(expected = org.apache.metron.stellar.dsl.ParseException.class)
+ public void testGetTooManyParams() throws Exception {
+ String stellar = "ASN_GET('8.8.4.0', ['autonomous_system_organization', 'autonomous_system_number', 'network'], 'garbage')";
+ run(stellar, ImmutableMap.of());
+ }
+}
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctionsTest.java b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctionsTest.java
index 2c0e8f1..2682a1f 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctionsTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/stellar/GeoEnrichmentFunctionsTest.java
@@ -19,12 +19,16 @@
package org.apache.metron.enrichment.stellar;
import com.google.common.collect.ImmutableMap;
+import java.io.File;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
import org.adrianwalker.multilinestring.Multiline;
+import org.apache.metron.enrichment.adapters.maxmind.geo.GeoLiteCityDatabase;
+import org.apache.metron.stellar.common.StellarProcessor;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.DefaultVariableResolver;
import org.apache.metron.stellar.dsl.StellarFunctions;
-import org.apache.metron.stellar.common.StellarProcessor;
-import org.apache.metron.enrichment.adapters.geo.GeoLiteDatabase;
import org.apache.metron.test.utils.UnitTestHelper;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
@@ -34,10 +38,6 @@ import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
-import java.io.File;
-import java.util.HashMap;
-import java.util.Map;
-
public class GeoEnrichmentFunctionsTest {
private static Context context;
private static File geoHdfsFile;
@@ -80,71 +80,110 @@ public class GeoEnrichmentFunctionsTest {
expectedSubsetMessage = (JSONObject) jsonParser.parse(expectedSubsetString);
String baseDir = UnitTestHelper.findDir("GeoLite");
- geoHdfsFile = new File(new File(baseDir), "GeoIP2-City-Test.mmdb.gz");
+ geoHdfsFile = new File(new File(baseDir), "GeoLite2-City.mmdb.gz");
}
@Before
public void setup() throws Exception {
context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG
- , () -> ImmutableMap.of(GeoLiteDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath())
+ , () -> ImmutableMap.of(GeoLiteCityDatabase.GEO_HDFS_FILE, geoHdfsFile.getAbsolutePath())
)
.build();
}
- public Object run(String rule, Map<String, Object> variables) throws Exception {
+ public Object run(String rule, Map<String, Object> variables) {
StellarProcessor processor = new StellarProcessor();
Assert.assertTrue(rule + " not valid.", processor.validate(rule, context));
return processor.parse(rule, new DefaultVariableResolver(x -> variables.get(x),x -> variables.containsKey(x)), StellarFunctions.FUNCTION_RESOLVER(), context);
}
@Test
- public void testGetNull() throws Exception {
+ public void testMissingDb() {
+ context = new Context.Builder().with(Context.Capabilities.GLOBAL_CONFIG
+ , () -> ImmutableMap.of(GeoLiteCityDatabase.GEO_HDFS_FILE, "./fakefile.mmdb")
+ ).build();
+ String stellar = "GEO_GET()";
+ try {
+ run(stellar, ImmutableMap.of());
+ } catch (Exception expected) {
+ Assert.assertTrue(expected.getMessage().contains("File fakefile.mmdb does not exist"));
+ }
+ }
+
+ @Test
+ public void testMissingDbDuringUpdate() {
String stellar = "GEO_GET()";
Object result = run(stellar, ImmutableMap.of());
- Assert.assertEquals("Null IP should return null", null, result);
+ Assert.assertNull("Null IP should return null", result);
+ try {
+ GeoLiteCityDatabase.INSTANCE.updateIfNecessary(
+ Collections.singletonMap(GeoLiteCityDatabase.GEO_HDFS_FILE, "./fakefile.mmdb"));
+ } catch (IllegalStateException e) {
+ // ignore it, the file doesn't exist
+ }
+ // Should still continue to query the old database, instead of dying.
+ result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Null IP should return null", result);
+ }
+
+ @Test
+ public void testGetEmpty() {
+ String stellar = "GEO_GET()";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Empty IP should return null", result);
+ }
+
+ @Test
+ public void testGetNull() {
+ String stellar = "GEO_GET(null)";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Null IP should return null", result);
+ }
+
+ @Test(expected = org.apache.metron.stellar.dsl.ParseException.class)
+ public void testGetUndefined() {
+ String stellar = "GEO_GET(undefined)";
+ Object result = run(stellar, ImmutableMap.of());
+ Assert.assertNull("Null IP should return null", result);
}
@Test
- public void testGetEmptyString() throws Exception {
+ public void testGetEmptyString() {
String stellar = "GEO_GET(' ')";
Object result = run(stellar, ImmutableMap.of());
- Assert.assertEquals("Empty IP should return null", null, result);
+ Assert.assertNull("Empty IP should return null", result);
}
@Test
- public void testGetLocal() throws Exception {
+ public void testGetLocal() {
String stellar = "GEO_GET('192.168.0.1')";
Object result = run(stellar, ImmutableMap.of());
Assert.assertEquals("Local IP should return empty map", new HashMap<String, String>(), result);
}
@Test
- @SuppressWarnings("unchecked")
- public void testGetRemote() throws Exception {
+ public void testGetRemote() {
String stellar = "GEO_GET('216.160.83.56')";
Object result = run(stellar, ImmutableMap.of());
Assert.assertEquals("Remote IP should return result based on DB", expectedMessage, result);
}
@Test
- @SuppressWarnings("unchecked")
- public void testGetRemoteSingleField() throws Exception {
+ public void testGetRemoteSingleField() {
String stellar = "GEO_GET('216.160.83.56', ['country'])";
Object result = run(stellar, ImmutableMap.of());
Assert.assertEquals("Remote IP should return country result based on DB", "US", result);
}
@Test
- @SuppressWarnings("unchecked")
- public void testGetRemoteMultipleFields() throws Exception {
+ public void testGetRemoteMultipleFields() {
String stellar = "GEO_GET('216.160.83.56', ['country', 'city', 'dmaCode', 'location_point'])";
Object result = run(stellar, ImmutableMap.of());
Assert.assertEquals("Remote IP should return country result based on DB", expectedSubsetMessage, result);
}
@Test(expected=org.apache.metron.stellar.dsl.ParseException.class)
- @SuppressWarnings("unchecked")
- public void testGetTooManyParams() throws Exception {
+ public void testGetTooManyParams() {
String stellar = "GEO_GET('216.160.83.56', ['country', 'city', 'dmaCode', 'location_point'], 'garbage')";
run(stellar, ImmutableMap.of());
}
diff --git a/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoIP2-City-Test-2.mmdb.gz b/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoIP2-City-Test-2.mmdb.gz
deleted file mode 100644
index 406c656..0000000
Binary files a/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoIP2-City-Test-2.mmdb.gz and /dev/null differ
diff --git a/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-ASN.tar.gz b/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-ASN.tar.gz
new file mode 100644
index 0000000..de4b05f
Binary files /dev/null and b/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-ASN.tar.gz differ
diff --git a/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoIP2-City-Test.mmdb.gz b/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-City.mmdb.gz
similarity index 100%
rename from metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoIP2-City-Test.mmdb.gz
rename to metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-City.mmdb.gz
diff --git a/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-City.tar.gz b/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-City.tar.gz
new file mode 100644
index 0000000..e4ae862
Binary files /dev/null and b/metron-platform/metron-enrichment/src/test/resources/GeoLite/GeoLite2-City.tar.gz differ
diff --git a/metron-stellar/stellar-common/README.md b/metron-stellar/stellar-common/README.md
index f4a49ef..333c2b5 100644
--- a/metron-stellar/stellar-common/README.md
+++ b/metron-stellar/stellar-common/README.md
@@ -330,6 +330,13 @@ Where:
* additionalsuffix - Optional - Additional string suffix that is a valid terminator.
* Returns: A new String if prefix was prepended, the same string otherwise.
+### `ASN_GET`
+* Description: Look up an IPV4 address and returns Autonomous System Number information about it
+* Input:
+ * ip - The IPV4 address to lookup
+ * fields - Optional list of ASN fields to grab. Options are network, autonomous_system_number, autonomous_system_organization.
+* Returns: If a Single field is requested a string of the field, If multiple fields a map of string of the fields, and null otherwise
+
### `BLOOM_ADD`
* Description: Adds an element to the bloom filter passed in
* Input: