You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/11/01 15:01:07 UTC
[jira] [Commented] (METRON-529) Allow Configurable Periods for
Profiler Client
[ https://issues.apache.org/jira/browse/METRON-529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625637#comment-15625637 ]
ASF GitHub Bot commented on METRON-529:
---------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/334
METRON-529 Allow Configurable Periods for Profiler Client
## [METRON-529](https://issues.apache.org/jira/browse/METRON-529)
By default, the Profiler creates Profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the client API is expecting 15 minute periods, it will not be able to read data generated by a Profiler that has been configured with a 1 hour period.
The period duration can be configured in the Profiler by altering the Profiler topology's static properties file. The Stellar Client API currently provides no means to configure the period duration and defaults also to 15 minutes. This means that the Stellar Client API can only read profiles with a period duration of 15 minutes.
This PR addresses this limitation so that the Profiler Client can be configured to read profile data no matter what configuration settings were used to generate that data.
### Testing
I tested this change in the "Quick Dev" environment. I used the Profiler to create a Profile using a 1 minute period. I then confirmed that I was unable to read the data using the Profiler client with the default client configuration. I then changed the client configuration to a 1 minute duration. I was then able to read the profile data as expected.
#### Steps
1. Ensure the Snort sensor and the Snort parser are running. Stop all other topologies and sensors.
2. Edit `/usr/metron/0.2.1BETA/config/profiler.properties` and change the period duration to 1 minute. Also, point the Profiler at the `enrichments` topic to read data directly from the Snort Parser.
```
profiler.input.topic=enrichments
profiler.period.duration=1
profiler.period.duration.units=MINUTES
```
3. Follow the Profiler's [Getting Started](https://github.com/apache/incubator-metron/tree/master/metron-analytics/metron-profiler#getting-started) instructions to start generating profile data.
4. Configure the client to read data with a 1 minute period. To do so, edit `/usr/metron/0.2.1BETA/config/zookeeper/global.properties`.
```
{
"es.clustername": "metron",
"es.ip": "node1",
"es.port": "9300",
"es.date.format": "yyyy.MM.dd.HH",
"profiler.client.period.duration": "1",
"profiler.client.period.duration.units": "MINUTES"
}
```
5. Push the previous global configuration change to Zookeeper.
```
bin/zk_load_configs.sh -m PUSH -i config/zookeeper/ -z node1:2181
```
6. Open up the Stellar shell and use the Profile client to read the data.
```
[root@node1 0.2.1BETA]# bin/stellar -z node1:2181
Stellar, Go!
Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH, profiler.client.hbase.table=profiler, profiler.client.column.family=P, profiler.client.hbase.table.provider=org.apache.metron.hbase.HTableProvider, profiler.client.period.duration=1, profiler.client.period.duration.units=MINUTES, profiler.client.salt.divisor=1000}
[Stellar]>>> PROFILE_GET('test','192.168.138.158', 2, 'DAYS')
Functions loaded, you may refer to functions now...
[161.0]
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-529
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/334.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #334
----
commit a902c2a7a6f11ea5131fc84be39b1f8b4913dece
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-11-01T13:42:02Z
METRON-529 Allow Configurable Periods for Profiler Client
----
> Allow Configurable Periods for Profiler Client
> ----------------------------------------------
>
> Key: METRON-529
> URL: https://issues.apache.org/jira/browse/METRON-529
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> By default, the Profiler creates Profiles with a period duration of 15 minutes. This means that data is accumulated, summarized and flushed every 15 minutes. The Client API must also have knowledge of this duration to correctly retrieve the profile data. If the client API is expecting 15 minute periods, it will not be able to read data generated by a Profiler that has been configured with a 1 hour period.
> The period duration can be configured in the Profiler by altering the Profiler topology's static properties file. The Stellar Client API currently provides no means to configure the period duration and defaults also to 15 minutes. This means that the Stellar Client API can only read profiles with a period duration of 15 minutes. This is a known limitation that has not yet been addressed.
> https://github.com/apache/incubator-metron/blob/e31705029fd7518bd03c22eece0c2a8eb1187abf/metron-analytics/metron-profiler-client/README.md#period-duration
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)