You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/02/02 21:56:58 UTC
[GitHub] incubator-metron pull request #437: METRON-692: Update Upgrading.md for 0.3....
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/437
METRON-692: Update Upgrading.md for 0.3.0 -> 0.3.1
We need a document to describe the manual upgrade issues from one release to the other.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-692
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #437
----
commit c217b6291725fc9226a6690d2d09a72a5fde9b23
Author: cstella <ce...@gmail.com>
Date: 2017-02-02T21:55:10Z
Creating upgrade document
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #437: METRON-692: Update Upgrading.md for 0.3.0 -> 0....
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/437
Everything I know of is in here. +1. Thanks for putting this together.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #437: METRON-692: Update Upgrading.md for 0.3.0 -> 0....
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/437
@cestella Let me know if there's anythign to change or adjust that you see in here. I'm not sure if you can get the raw version in GitHub like in Jira, so ping me if this isn't grabbable in the current form and I'll get it to you differently.
### [Migrate Geo Enrichment outside of MySQL](https://issues.apache.org/jira/browse/METRON-283)
#### Description
As of 0.3.0, a MySQL database was used for storage and retrieval of GeoIP information during enrichment.
As of 0.3.1, the MySQL database is removed in favor of using MaxMind's binary GeoIP files and stored on HDFS
After initial setup, this change is transparent and existing enrichment definitions will run as-is.
#### Migration
While new installs will not require any additional steps, in an existing install a script must be run to retrieve and load the initial data.
The shell script `geo_enrichment_load.sh` will retrieve MaxMind GeoLite2 data and load data into HDFS, and update the configuration to point to this data.
In most cases the following usage will grab the data appropriately:
```
$METRON_HOME/bin/geo_enrichment_load.sh -z <zk_server>:<zk_port>
```
Additional options, including changing the source file location (which can be a file:// location if the GeoIP data is already downloaded), are available with the
`-h` flag and are also detailed in the metron-data-management README.me file.
One caveat is that this script will NOT update on disk config files. It is recommended to retrieve the configuration using
```
$METRON_HOME/bin/zk_load_configs.sh -z <zk_server>:<zk_port> -m DUMP
```
The new config will be `geo.hdfs.file` in the global section of the configuration. Append this key-value into the global.json in the config directory. A PUSH is unnecessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #437: METRON-692: Update Upgrading.md for 0.3....
Posted by cestella <gi...@git.apache.org>.
GitHub user cestella reopened a pull request:
https://github.com/apache/incubator-metron/pull/437
METRON-692: Update Upgrading.md for 0.3.0 -> 0.3.1
We need a document to describe the manual upgrade issues from one release to the other.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-692
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #437
----
commit c217b6291725fc9226a6690d2d09a72a5fde9b23
Author: cstella <ce...@gmail.com>
Date: 2017-02-02T21:55:10Z
Creating upgrade document
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #437: METRON-692: Update Upgrading.md for 0.3....
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/437
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #437: METRON-692: Update Upgrading.md for 0.3....
Posted by cestella <gi...@git.apache.org>.
Github user cestella closed the pull request at:
https://github.com/apache/incubator-metron/pull/437
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #437: METRON-692: Update Upgrading.md for 0.3.0 -> 0....
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/437
@justinleet Could you write up a description that fits the template in the doc and I'll cut and paste into the doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #437: METRON-692: Update Upgrading.md for 0.3.0 -> 0....
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/incubator-metron/pull/437
While the migration from MySQL is transparent once data is in place, it requires running geo_enrichment_load.sh the first time to populate the GeoIP data into HDFS (or alternatively manually uploading the binary database file into HDFS and updating the global config to have geo.hdfs.file point to it). By default, the script just needs to be run as
`geo_enrichment_load.sh -z <zk_server:zk_port>`
I would also mention that if there is a MySQL instance dedicated to Metron's Geo data, it can be removed (AFTER appropriate testing).
You want me to do a write-up for that portion, or would you rather grab it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #437: METRON-692: Update Upgrading.md for 0.3.0 -> 0....
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/437
I looked through all the 0.3.1 tickets and don't see any others that we definitely need to call out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---