You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nsinha17 <gi...@git.apache.org> on 2016/09/28 09:20:39 UTC
[GitHub] incubator-metron pull request #282: Metron 430: Ability to load specific Geo...
GitHub user nsinha17 opened a pull request:
https://github.com/apache/incubator-metron/pull/282
Metron 430: Ability to load specific GeoIP data to mysql server
With the current implementation, the latest geoip data gets loaded to MySQl db as the url in geoip.yml is made to point to :- http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.tar.xz
```
name: Download GeoIP databases
unarchive:
src: http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.tar.xz
dest: /tmp/geoip
copy: no
creates: /tmp/geoip/*/GeoLiteCity-Blocks.csv
```
However, we would like to provide the user with the ability to specify the geodata that he wishes to load to the mysql and build metron project.
The user can specify the URL or the filename directly that should be picked up for loading the geo data.
**Test Scenario 1 - Specifying URL hosting Geolite DB - Build Defaults**
1. Edit `roles/mysql_server/defaults/main.yml` file and specify the Geolite DB URL for `geodata_file` parameter. Note that this points to the Maxmind Geolite DB by default. You may leave this parameter unedited if a latest copy of the Maxmind Geolite DB is desired.
2. Now run the deployment code and the specified geodata would be populated in the mysql server.
**Test Scenario 2 - Specifying Geolite DB file manually**
1. Copy the required Geolite db file to `roles/mysql_server/files/`
2. Edit the `roles/mysql_server/defaults/main.yml` file and provide the name of the file to the parameter `geodata_file`
3. Now run the deployment code and the specified geodata would be populated in the mysql server.
Note :- A sample Geolite DB file called `geodata.tar.xz` is included in the `roles/mysql_server/files/` directory along with this pull request for test purposes.
**Testing Done**
- Tested with single and multi node deployments on openstack
- Tested both scenarios 1 and 2 above.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nsinha17/incubator-metron METRON-430
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/282.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #282
----
commit f29c0e760b5125abff325f5c1b6f8cef1eb10fc1
Author: nsinha17 <ns...@hortonworks.com>
Date: 2016-09-21T11:35:26Z
METRON-430: Enhancement done to load specific GeoIP data to mysql server
commit d32b16c84000b9cac0dbe6fd82dff5ae1252f5e9
Author: nsinha17 <ns...@hortonworks.com>
Date: 2016-09-23T06:30:54Z
Merge branch 'master' of https://github.com/apache/incubator-metron into METRON-430
commit 9516d00d310b2ac26d1cc8e516285ff4b74ffbc6
Author: nsinha17 <ns...@hortonworks.com>
Date: 2016-09-28T07:05:10Z
METRON-430 Modified mysql_server role to accept user specified geoip data
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #282: Metron 430: Ability to load specific Geo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/282#discussion_r81011621
--- Diff: metron-deployment/roles/mysql_server/tasks/geoip.yml ---
@@ -15,12 +15,21 @@
# limitations under the License.
#
---
-- name: Download GeoIP databases
+- name: Download GeoIP databases from specified URL
unarchive:
- src: http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.tar.xz
+ src: "{{ geodata_file }}"
dest: /tmp/geoip
copy: no
creates: /tmp/geoip/*/GeoLiteCity-Blocks.csv
+ when: '"{{ geodata_file }}".find("http://") != -1'
--- End diff --
With your original approach, you can set `geodata: 'skip'` (anything other than http://) and it completely skips loading the geodata. That is a nice option to have.
I think this still works with my suggested approach if you set the property like this; `geodata: ~`. Warning: I haven't tested this and Ansible makes these things a little painful.
However it gets done, I think it would be nice to...
* Use HTTPS, HTTP, and FTP URLs
* Not load any geodata, if so desired
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #282: Metron 430: Ability to load specific Geo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/282
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #282: Metron 430: Ability to load specific GeoIP data...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/282
+1 by inspection. Thanks for the contribution!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #282: Metron 430: Ability to load specific Geo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/282#discussion_r81008117
--- Diff: metron-deployment/roles/mysql_server/tasks/geoip.yml ---
@@ -15,12 +15,21 @@
# limitations under the License.
#
---
-- name: Download GeoIP databases
+- name: Download GeoIP databases from specified URL
unarchive:
- src: http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.tar.xz
+ src: "{{ geodata_file }}"
dest: /tmp/geoip
copy: no
creates: /tmp/geoip/*/GeoLiteCity-Blocks.csv
+ when: '"{{ geodata_file }}".find("http://") != -1'
--- End diff --
A better way to do this check might be the following. This approach is a little simpler and doesn't preclude the use of FTP or HTTPS.
```
when: geodata_file is defined
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #282: Metron 430: Ability to load specific GeoIP data...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/282
Neha and I just talked through this. The other issue that we discovered is that the PR has a canned tarball containing geoip data from Maxmind. Since this is proprietary data we cannot host it in Metron.
We agreed that we just want to cover two use cases:
1. The user wants to download the latest data from MaxMind. This is the default behavior.
2. The user wants to use data that is hosted at an alternative URL. The alternative URL could be HTTP, HTTPS, or FTP. This might occur when running Metron on an isolated/secure network or a test environment.
To cover these two use cases, we won't need the `metron-deployment/roles/mysql_server/files/geodata.tar.xz` tarball.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #282: Metron 430: Ability to load specific Geo...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/282#discussion_r81008161
--- Diff: metron-deployment/roles/mysql_server/tasks/geoip.yml ---
@@ -15,12 +15,21 @@
# limitations under the License.
#
---
-- name: Download GeoIP databases
+- name: Download GeoIP databases from specified URL
unarchive:
- src: http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.tar.xz
+ src: "{{ geodata_file }}"
dest: /tmp/geoip
copy: no
creates: /tmp/geoip/*/GeoLiteCity-Blocks.csv
+ when: '"{{ geodata_file }}".find("http://") != -1'
+
+- name: Copy GeoIP database file from files directory
+ unarchive:
+ src: "{{ geodata_file }}"
+ dest: /tmp/geoip
+ copy: yes
+ creates: /tmp/geoip/*/GeoLiteCity-Blocks.csv
+ when: '"{{ geodata_file }}".find("http://") == -1'
--- End diff --
Same here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #282: Metron 430: Ability to load specific GeoIP data...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/282
+1 looks good pending travis build
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---