You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by nickwallen <gi...@git.apache.org> on 2018/03/16 16:03:51 UTC
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/967
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
WARNING: This change is dependent on #965 . The diff will make much more sense once #965 is reviewed.
Using the 'result/triage' expression allows you to send profile data to Kafka. This allows you to leverage the Threat Triage functionality against data coming out of the Profiler.
If there is no 'result/triage' expression, then nothing should be sent to Kafka. Currently, a message containing some data, but no actual profile value, is sent to Kafka.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1494
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/967.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #967
----
----
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/967
Still looks fine - tests now passing with 977, lgtm. +1
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
@mmiklavc This is ready to go if you don't mind taking another look at this. This is not much different than what you have already reviewed.
Since your original review I added another set of unit tests and a series of debug logging which helped me track down the problem.
I updated the description to outline exactly what has been changed here. If I can clarify anything, let me know.
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
Of course, now the integration tests don't want to fail.
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/967
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/967
My +1 still stands. Tests are good, readable.
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/967
...Except for the test timeout in Travis
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:
https://github.com/apache/metron/pull/967
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
Using the 'result/triage' expression allows you to send profile data to Kafka. This allows you to leverage the Threat Triage functionality against data coming out of the Profiler.
If there is no 'result/triage' expression, then nothing should be sent to Kafka. Currently, a message containing some data, but no actual profile value, is sent to Kafka.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1494
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/967.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #967
----
commit 64343bc0d99880ac8bb17137a9226c3f44417da7
Author: Nick Allen <ni...@...>
Date: 2018-02-13T14:52:54Z
METRON-590 Enable Use of Event Time in Profiler
commit 3f615715d759ee8af7ec873f6bef2f20c07bc35f
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:03:00Z
Updated README with more details on event time.
commit de754649e1f3177c4e3b209fe9495826e4b76e86
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:12:03Z
Fixed the property name in the integration tests
commit 36923c0f841f6cc24f17794b696c1e22881806c7
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:53:01Z
Fixed a few items I missed in rename to 'window lag'
commit 1e8b76b9819e983f6a649183fd3a89e7c5ab1817
Author: Nick Allen <ni...@...>
Date: 2018-03-16T16:01:46Z
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
commit d78ba493620a4e902c7d6b0dc6719b1c9d3c0bca
Author: Nick Allen <ni...@...>
Date: 2018-03-16T17:26:05Z
Cleaning up imports
commit ab75a1d7d887351bfcbadb4221ad10938e3998ee
Author: Nick Allen <ni...@...>
Date: 2018-03-16T19:59:23Z
Fixed comment
commit 5de7a6235acf642e8aeae79df627b436d4d3d87d
Author: Nick Allen <ni...@...>
Date: 2018-03-20T21:31:03Z
Merge branch 'master' into METRON-1494
commit edbe2adf98f8ebef159d7dee167208208707c636
Author: Nick Allen <ni...@...>
Date: 2018-03-21T18:25:13Z
Merge remote-tracking branch 'apache/master' into METRON-1494
commit 9aa09f4cdb9a6e1947606f0b7823d85823121cf7
Author: Nick Allen <ni...@...>
Date: 2018-03-21T18:41:55Z
Added tests for this fix
commit 6be13340255d1f9f606765d2f87ad0744cea07ec
Author: Nick Allen <ni...@...>
Date: 2018-03-21T20:28:08Z
Making sure the timeout for all profiler integration tests is the same
commit 044c0a72db9dffda67243678f515a0a195fcf3a4
Author: Nick Allen <ni...@...>
Date: 2018-03-21T20:30:29Z
Fix alignment
commit cf9ce2604ad4de3a3b5957606253b5d43921094c
Author: Nick Allen <ni...@...>
Date: 2018-03-21T22:05:09Z
Added unit tests for HBaseEmitter and additional logging
commit 19f4998d978b32ee5f522c223dadd959f6ffc3d0
Author: Nick Allen <ni...@...>
Date: 2018-03-21T22:05:41Z
Setting Profiler logs to DEBUG
commit e8c203a297939d4c0b32aa55e5efb42f81b00e14
Author: Nick Allen <ni...@...>
Date: 2018-03-22T16:13:58Z
Added additional logging to track down intermittent integration test failure
commit 6faee6a4d58e9d2b65083422ce67a7e6f4e684c2
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:13:48Z
This log statement is helpful
commit e613836ffc585d38cc59ef6c4859d843df413918
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:14:06Z
Fixed comment
commit d3465b9a36a971a1a347a445a67f96a0e1affed1
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:14:22Z
Increase logging for Storm's windowing functionality
----
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
I am not sure what is going on. Those tests have never failed when run under my own Github account.
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
This should go in after #977
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
Reopen to trigger CI again
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:
https://github.com/apache/metron/pull/967
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
Using the 'result/triage' expression allows you to send profile data to Kafka. This allows you to leverage the Threat Triage functionality against data coming out of the Profiler.
If there is no 'result/triage' expression, then nothing should be sent to Kafka. Currently, a message containing some data, but no actual profile value, is sent to Kafka.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1494
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/967.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #967
----
commit 64343bc0d99880ac8bb17137a9226c3f44417da7
Author: Nick Allen <ni...@...>
Date: 2018-02-13T14:52:54Z
METRON-590 Enable Use of Event Time in Profiler
commit 3f615715d759ee8af7ec873f6bef2f20c07bc35f
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:03:00Z
Updated README with more details on event time.
commit de754649e1f3177c4e3b209fe9495826e4b76e86
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:12:03Z
Fixed the property name in the integration tests
commit 36923c0f841f6cc24f17794b696c1e22881806c7
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:53:01Z
Fixed a few items I missed in rename to 'window lag'
commit 1e8b76b9819e983f6a649183fd3a89e7c5ab1817
Author: Nick Allen <ni...@...>
Date: 2018-03-16T16:01:46Z
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
commit d78ba493620a4e902c7d6b0dc6719b1c9d3c0bca
Author: Nick Allen <ni...@...>
Date: 2018-03-16T17:26:05Z
Cleaning up imports
commit ab75a1d7d887351bfcbadb4221ad10938e3998ee
Author: Nick Allen <ni...@...>
Date: 2018-03-16T19:59:23Z
Fixed comment
commit 5de7a6235acf642e8aeae79df627b436d4d3d87d
Author: Nick Allen <ni...@...>
Date: 2018-03-20T21:31:03Z
Merge branch 'master' into METRON-1494
commit edbe2adf98f8ebef159d7dee167208208707c636
Author: Nick Allen <ni...@...>
Date: 2018-03-21T18:25:13Z
Merge remote-tracking branch 'apache/master' into METRON-1494
commit 9aa09f4cdb9a6e1947606f0b7823d85823121cf7
Author: Nick Allen <ni...@...>
Date: 2018-03-21T18:41:55Z
Added tests for this fix
commit 6be13340255d1f9f606765d2f87ad0744cea07ec
Author: Nick Allen <ni...@...>
Date: 2018-03-21T20:28:08Z
Making sure the timeout for all profiler integration tests is the same
commit 044c0a72db9dffda67243678f515a0a195fcf3a4
Author: Nick Allen <ni...@...>
Date: 2018-03-21T20:30:29Z
Fix alignment
commit cf9ce2604ad4de3a3b5957606253b5d43921094c
Author: Nick Allen <ni...@...>
Date: 2018-03-21T22:05:09Z
Added unit tests for HBaseEmitter and additional logging
commit 19f4998d978b32ee5f522c223dadd959f6ffc3d0
Author: Nick Allen <ni...@...>
Date: 2018-03-21T22:05:41Z
Setting Profiler logs to DEBUG
commit e8c203a297939d4c0b32aa55e5efb42f81b00e14
Author: Nick Allen <ni...@...>
Date: 2018-03-22T16:13:58Z
Added additional logging to track down intermittent integration test failure
commit 6faee6a4d58e9d2b65083422ce67a7e6f4e684c2
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:13:48Z
This log statement is helpful
commit e613836ffc585d38cc59ef6c4859d843df413918
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:14:06Z
Fixed comment
commit d3465b9a36a971a1a347a445a67f96a0e1affed1
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:14:22Z
Increase logging for Storm's windowing functionality
----
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
Ok fixed that. Had to pull that out my local stash.
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/metron/pull/967
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:
https://github.com/apache/metron/pull/967
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
Using the 'result/triage' expression allows you to send profile data to Kafka. This allows you to leverage the Threat Triage functionality against data coming out of the Profiler.
If there is no 'result/triage' expression, then nothing should be sent to Kafka. Currently, a message containing some data, but no actual profile value, is sent to Kafka.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1494
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/967.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #967
----
commit 64343bc0d99880ac8bb17137a9226c3f44417da7
Author: Nick Allen <ni...@...>
Date: 2018-02-13T14:52:54Z
METRON-590 Enable Use of Event Time in Profiler
commit 3f615715d759ee8af7ec873f6bef2f20c07bc35f
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:03:00Z
Updated README with more details on event time.
commit de754649e1f3177c4e3b209fe9495826e4b76e86
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:12:03Z
Fixed the property name in the integration tests
commit 36923c0f841f6cc24f17794b696c1e22881806c7
Author: Nick Allen <ni...@...>
Date: 2018-03-15T21:53:01Z
Fixed a few items I missed in rename to 'window lag'
commit 1e8b76b9819e983f6a649183fd3a89e7c5ab1817
Author: Nick Allen <ni...@...>
Date: 2018-03-16T16:01:46Z
METRON-1494 Profiler Emits Messages to Kafka When Not Needed
commit d78ba493620a4e902c7d6b0dc6719b1c9d3c0bca
Author: Nick Allen <ni...@...>
Date: 2018-03-16T17:26:05Z
Cleaning up imports
commit ab75a1d7d887351bfcbadb4221ad10938e3998ee
Author: Nick Allen <ni...@...>
Date: 2018-03-16T19:59:23Z
Fixed comment
commit 5de7a6235acf642e8aeae79df627b436d4d3d87d
Author: Nick Allen <ni...@...>
Date: 2018-03-20T21:31:03Z
Merge branch 'master' into METRON-1494
commit edbe2adf98f8ebef159d7dee167208208707c636
Author: Nick Allen <ni...@...>
Date: 2018-03-21T18:25:13Z
Merge remote-tracking branch 'apache/master' into METRON-1494
commit 9aa09f4cdb9a6e1947606f0b7823d85823121cf7
Author: Nick Allen <ni...@...>
Date: 2018-03-21T18:41:55Z
Added tests for this fix
commit 6be13340255d1f9f606765d2f87ad0744cea07ec
Author: Nick Allen <ni...@...>
Date: 2018-03-21T20:28:08Z
Making sure the timeout for all profiler integration tests is the same
commit 044c0a72db9dffda67243678f515a0a195fcf3a4
Author: Nick Allen <ni...@...>
Date: 2018-03-21T20:30:29Z
Fix alignment
commit cf9ce2604ad4de3a3b5957606253b5d43921094c
Author: Nick Allen <ni...@...>
Date: 2018-03-21T22:05:09Z
Added unit tests for HBaseEmitter and additional logging
commit 19f4998d978b32ee5f522c223dadd959f6ffc3d0
Author: Nick Allen <ni...@...>
Date: 2018-03-21T22:05:41Z
Setting Profiler logs to DEBUG
commit e8c203a297939d4c0b32aa55e5efb42f81b00e14
Author: Nick Allen <ni...@...>
Date: 2018-03-22T16:13:58Z
Added additional logging to track down intermittent integration test failure
commit 6faee6a4d58e9d2b65083422ce67a7e6f4e684c2
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:13:48Z
This log statement is helpful
commit e613836ffc585d38cc59ef6c4859d843df413918
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:14:06Z
Fixed comment
commit d3465b9a36a971a1a347a445a67f96a0e1affed1
Author: Nick Allen <ni...@...>
Date: 2018-03-22T17:14:22Z
Increase logging for Storm's windowing functionality
----
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/967#discussion_r179923960
--- Diff: metron-analytics/metron-profiler/src/main/java/org/apache/metron/profiler/bolt/KafkaEmitter.java ---
@@ -58,19 +61,48 @@ public void declareOutputFields(OutputFieldsDeclarer declarer) {
@Override
public void emit(ProfileMeasurement measurement, OutputCollector collector) {
- JSONObject message = new JSONObject();
- message.put("profile", measurement.getDefinition().getProfile());
- message.put("entity", measurement.getEntity());
- message.put("period", measurement.getPeriod().getPeriod());
- message.put("period.start", measurement.getPeriod().getStartTimeMillis());
- message.put("period.end", measurement.getPeriod().getEndTimeMillis());
- message.put("timestamp", System.currentTimeMillis());
- message.put("source.type", sourceType);
- message.put("is_alert", "true");
+ // only need to emit, if there are triage values
+ Map<String, Object> triageValues = measurement.getTriageValues();
+ if(MapUtils.isNotEmpty(triageValues)) {
--- End diff --
This is the core of the fix. We only need to emit if there are triage values.
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
The problem is somewhere in Storm's windowing functionality. The time that it initially recognizes is too far in the future and causes it to mark the messages sent in as expired. This only occurs intermittently.
Here you can see test messages generated with the last timestamp being 1521756035817.
```
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for message with timestamp=1521756035759
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for message with timestamp=1521756035802
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for message with timestamp=1521756035806
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message; profile=example2, entity=10.0.0.2, timestamp=1521756035807
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for message with timestamp=1521756035807
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message; profile=example2, entity=10.0.0.2, timestamp=1521756035808
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for message with timestamp=1521756035808
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message; profile=example2, entity=10.0.0.2, timestamp=1521756035813
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for message with timestamp=1521756035813
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message; profile=example2, entity=10.0.0.3, timestamp=1521756035814
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for message with timestamp=1521756035814
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message; profile=example2, entity=10.0.0.3, timestamp=1521756035816
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for message with timestamp=1521756035816
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for message; profile=example2, entity=10.0.0.3, timestamp=1521756035817
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for message with timestamp=1521756035817
```
The first timestamp that Storm recognizes is 1521756041122, which is 5.3 seconds ahead of the latest timestamp in the data.
```
2018-03-22 22:00:41 DEBUG WindowManager:189 - Scan events, eviction policy TimeEvictionPolicy{windowLength=5000, referenceTime=1521756041122}
```
Storm then marks these messages as expired and the Profiler never sees them.
```
2018-03-22 22:00:41 DEBUG WindowManager:212 - [6] events expired from window.
2018-03-22 22:00:41 DEBUG WindowManager:214 - invoking windowLifecycleListener.onExpiry
2018-03-22 22:00:41 DEBUG WindowManager:147 - No events in the window, skipping onActivation
```
Epic test failure.
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by mmiklavc <gi...@git.apache.org>.
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/967
+1 via inspection
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
Trying to get the integration test to fail again.
---
[GitHub] metron issue #967: METRON-1494 Profiler Emits Messages to Kafka When Not Nee...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
Crank. I just noticed that the unit tests that I wrote for this didn't get added to this branch. Let me correct that. I had added some unit tests specifically for this.
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/metron/pull/967
---
[GitHub] metron pull request #967: METRON-1494 Profiler Emits Messages to Kafka When ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/metron/pull/967
---