You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2017/10/05 20:12:48 UTC
[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...
GitHub user cestella opened a pull request:
https://github.com/apache/metron/pull/791
METRON-1235: Document the properties pulled from the global configuration
## Contributor Comments
Our global config is very flexible and multiple parts of the system pull configuration from it. Also, Ambari manages bits and pieces of that configuration. It's imperative to make precise which properties are managed in the global config, what they're for and whether ambari manages them.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron global_config_doc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/791.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #791
----
commit e28e24749f93fc39298523ece695b1993a781fef
Author: cstella <ce...@gmail.com>
Date: 2017-10-05T20:08:28Z
Added more detailed documentation around global configs.
----
---
[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/791
I'm good with where it's at now. The color is good to have. It's really nice to have this well documented.
I'm +1, pending Travis. Although I'll be mildly impressed if you break Travis with doc changes.
---
[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/791
One more config I can think of that should be added.
`geo.hdfs.file` is also in the global config. It's in Ambari as `geoip_url`.
It looks like a small blurb should probably be added to metron-enrichment. Something along the lines of:
## GeoIP
Metron supports enrichment of IP information using [GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The location of the file is managed in the global config.
## Properties
### `geo.hdfs.file`
The location on HDFS of the GeoLite2 database file to use for GeoIP lookups.
---
[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/791
This is excellent. I really like the table layout. +1 pending the merge conflicts and Travis.
---
[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/791
---
[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/791#discussion_r143058984
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper.
There are two types of configurations at the moment, `global` and
`sensor` specific.
+
## Global Configuration
-See the "[Global Configuration](../metron-common)" section.
+There are a few enrichments which have independent configurations, such
+as from the global config.
+
+Also, see the "[Global Configuration](../metron-common)" section for
+more discussion of the global config.
+
+### GeoIP
+Metron supports enrichment of IP information using
+[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The
+location of the file is managed in the global config.
+
+#### `geo.hdfs.file`
+
+The location on HDFS of the GeoLite2 database file to use for GeoIP
+lookups. This file will be localized on the storm supervisors running
+the topology and used from there. If this file changes, a topology
--- End diff --
Regardless, I take your point, I'll make that clear.
---
[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/791#discussion_r143058876
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper.
There are two types of configurations at the moment, `global` and
`sensor` specific.
+
## Global Configuration
-See the "[Global Configuration](../metron-common)" section.
+There are a few enrichments which have independent configurations, such
+as from the global config.
+
+Also, see the "[Global Configuration](../metron-common)" section for
+more discussion of the global config.
+
+### GeoIP
+Metron supports enrichment of IP information using
+[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The
+location of the file is managed in the global config.
+
+#### `geo.hdfs.file`
+
+The location on HDFS of the GeoLite2 database file to use for GeoIP
+lookups. This file will be localized on the storm supervisors running
+the topology and used from there. If this file changes, a topology
--- End diff --
What if the file was updated? So, same filename, different contents.
---
[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/791#discussion_r143059336
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper.
There are two types of configurations at the moment, `global` and
`sensor` specific.
+
## Global Configuration
-See the "[Global Configuration](../metron-common)" section.
+There are a few enrichments which have independent configurations, such
+as from the global config.
+
+Also, see the "[Global Configuration](../metron-common)" section for
+more discussion of the global config.
+
+### GeoIP
+Metron supports enrichment of IP information using
+[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The
+location of the file is managed in the global config.
+
+#### `geo.hdfs.file`
+
+The location on HDFS of the GeoLite2 database file to use for GeoIP
+lookups. This file will be localized on the storm supervisors running
+the topology and used from there. If this file changes, a topology
--- End diff --
There, clarified. Is that right?
---
[GitHub] metron pull request #791: METRON-1235: Document the properties pulled from t...
Posted by justinleet <gi...@git.apache.org>.
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/791#discussion_r143058611
--- Diff: metron-platform/metron-enrichment/README.md ---
@@ -25,9 +25,26 @@ defined by JSON documents stored in zookeeper.
There are two types of configurations at the moment, `global` and
`sensor` specific.
+
## Global Configuration
-See the "[Global Configuration](../metron-common)" section.
+There are a few enrichments which have independent configurations, such
+as from the global config.
+
+Also, see the "[Global Configuration](../metron-common)" section for
+more discussion of the global config.
+
+### GeoIP
+Metron supports enrichment of IP information using
+[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The
+location of the file is managed in the global config.
+
+#### `geo.hdfs.file`
+
+The location on HDFS of the GeoLite2 database file to use for GeoIP
+lookups. This file will be localized on the storm supervisors running
+the topology and used from there. If this file changes, a topology
--- End diff --
Topology restart shouldn't be required. In the event of a global config change `GenericEnrichmentBolt.reloadCallback` calls `GeoAdapter.updateAdapter` which delegates to `GeoLiteDatabase.INSTANCE.updateIfNecessary`.
---
[GitHub] metron issue #791: METRON-1235: Document the properties pulled from the glob...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/791
Thanks @justinleet I added and provided some color; let me know what you think!
---