You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/08/13 23:38:00 UTC
[jira] [Commented] (METRON-739) Create Local Profile Runner
[ https://issues.apache.org/jira/browse/METRON-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125086#comment-16125086 ]
ASF GitHub Bot commented on METRON-739:
---------------------------------------
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/693
METRON-739 Create Local Profile Runner
Troubleshooting issues when programming against a live stream of data can be difficult. This is the primary reason that drove the creation of the REPL. Similarly it would be useful to have a means to run a profile locally to troubleshoot, validate, or debug a profile. Now you can, right in the REPL!
Take it for a test spin.
1. Create your profiles. For this example, In the editor copy/paste in the basic "Hello, World" profile.
```
[Stellar]>>> conf := SHELL_EDIT()
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
```
1. Initialize the Profiler.
```
[Stellar]>>> profiler := PROFILER_INIT(conf)
[Stellar]>>> profiler
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
```
1. Create some telemetry. In the editor, copy/paste in the message JSON.
```
[Stellar]>>> message := SHELL_EDIT()
[Stellar]>>> message
{
"ip_src_addr": "10.0.0.1",
"protocol": "HTTPS",
"length": "10",
"bytes_in": "234"
}
```
1. Apply some telemetry messages to your profiles.
```
[Stellar]>>> PROFILER_APPLY(message, profiler)
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
[Stellar]>>> PROFILER_APPLY(message, profiler)
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
[Stellar]>>> PROFILER_APPLY(message, profiler)
org.apache.metron.profiler.StandAloneProfiler@4f8ef473
```
1. Flush the profiles, to see what it has calculated. The result is a list of profile measurements. Each value is a map containing detailed information about the profile measurements that were taken.
```
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[Stellar]>>> values
[{period={duration=900000, period=1669628, start=1502665200000, end=1502666100000},
profile=hello-world, groups=[], value=3, entity=10.0.0.1}]
```
I have added unit tests to cover all of the functionality, have tested the new Stellar functions in the REPL and have also completed a basic regression test for the Profiler itself.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
```
mvn -q clean integration-test install && build_utils/verify_licenses.sh
```
- [ ] Have you written or updated unit tests and or integration tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:
```
cd site-book
mvn site
```
#### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-739
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/693.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #693
----
commit c4cf920288d70d43709f951f811217b652f861f6
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-08-10T20:51:24Z
METRON-739 Create Local Profile Runner
commit 2c67a7f202aaa67e47341ea0041da7412823ea96
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-08-13T18:46:10Z
Added tests for Stellar functions
commit 50aab50e91bc3fece0c97da493926eeac63cc871
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-08-13T19:16:15Z
Merge remote-tracking branch 'apache/master' into METRON-739
commit b2ddf706a971404278d5a189874429e33a72527e
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-08-13T19:39:37Z
Added license headers
commit 50d3290ac5778f2f0d0868659f84365a77b91bec
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-08-13T22:23:09Z
Fixed defaults when no global config, also better docs
commit 8c48d9b666ca7f04b7475e9dd55d4a28c01eb67e
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-08-13T23:31:26Z
Returning details about profile measurements
----
> Create Local Profile Runner
> ---------------------------
>
> Key: METRON-739
> URL: https://issues.apache.org/jira/browse/METRON-739
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Assignee: Nick Allen
>
> Troubleshooting issues when programming against a live stream of data can be difficult. This is the primary reason that drove the creation of the REPL.
> Similarly it would be useful to have a means to run a profile 'locally' to troubleshoot, validate, or debug the profile. I would want to somehow be able to throw canned data at the Profile message-by-message and interrogate the current state of the Profile. I would also need to be able to trigger a flush event and interrogate the result.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)