You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2017/09/07 01:05:37 UTC
[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/736
METRON-1162 Apply Live Messages to the Profile Debugger
I want to be able to use `PROFILER_APPLY` with live messages from a Metron cluster. For example, I would like to be able to grab 10 messages from my production Metron and then apply them in my debugger. This would be tremendously helpful for iterating between the debugger and a live cluster when troubleshooting.
```
p := PROFILER_INIT(conf)
msgs := KAFKA_GET("indexing", 10)
PROFILER_APPLY(msgs, p)
PROFILER_FLUSH(p)
```
The `PROFILER_APPLY` function does not currently accept a list of messages. It accepts a String that is a JSON list, but not a Stellar list. For `PROFILER_APPLY` to play happy with `KAFKA_GET` (and other potential uses) it needs to accept a List.
This PR makes `PROFILER_APPLY` rather flexible in the types of arguments that it accepts.
* String containing one JSON message
* String containing an array of JSON messages
* List containing a JSONObject
* List containing a String of one JSON message
* List containing a String that is an array of JSON messages
### Testing
This can be tested by launching the REPL and running the following.
Create a Profiler definition.
```
[Stellar]>>> conf := SHELL_EDIT()
[Stellar]>>> conf
{
"profiles": [
{
"profile": "hello-world",
"onlyif": "exists(ip_src_addr)",
"foreach": "ip_src_addr",
"init": { "count": "0" },
"update": { "count": "count + 1" },
"result": "count"
}
]
}
```
Create a sample message.
```
[Stellar]>>> msg := SHELL_EDIT()
[Stellar]>>> msg
{
"ip_src_addr": "10.0.0.1",
"protocol": "HTTPS",
"length": "10",
"bytes_in": "234"
}
```
Initialize the Profiler.
```
[Stellar]>>> p := PROFILER_INIT(conf)
[Stellar]>>> p
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
```
Apply 3 messages at a time by passing in a list.
```
[Stellar]>>> PROFILER_APPLY([msg, msg, msg], p)
Profiler{1 profile(s), 3 messages(s), 3 route(s)}
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1162
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/736.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #736
----
commit 1cd5794eb7cadbd9923a27d6fe5e63c63e75ccb6
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-09-06T20:39:01Z
METRON-1160 Blueprint configuration validation failed: Missing required properties
commit b42f27635f640f8495364af6fc01dedc7c762d94
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-09-07T00:55:04Z
METRON-1162 Apply Live Messages to the Profile Debugger
commit 21005351eb000ac0c85fe5a1702b850ec25ca092
Author: Nick Allen <ni...@nickallen.org>
Date: 2017-09-07T01:03:48Z
Revert "METRON-1160 Blueprint configuration validation failed: Missing required properties"
This reverts commit 1cd5794eb7cadbd9923a27d6fe5e63c63e75ccb6.
----
---
[GitHub] metron issue #736: METRON-1162 Apply Live Messages to the Profile Debugger
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/736
+1 by inspection, lgtm!
---
[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/736#discussion_r137808595
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java ---
@@ -131,50 +133,99 @@ public boolean isInitialized() {
@Override
public Object apply(List<Object> args, Context context) throws ParseException {
- // user must provide the message as a string
- String arg0 = Util.getArg(0, String.class, args);
- if(arg0 == null) {
- throw new IllegalArgumentException(format("expected string, found null"));
+ // the use can pass in one or more messages in a few different forms
+ Object arg0 = Util.getArg(0, Object.class, args);
+ List<JSONObject> messages = getMessages(arg0);
+
+ // user must provide the stand alone profiler
+ StandAloneProfiler profiler = Util.getArg(1, StandAloneProfiler.class, args);
+ try {
+ for (JSONObject message : messages) {
+ profiler.apply(message);
+ }
+
+ } catch (ExecutionException e) {
+ throw new IllegalArgumentException(format("Failed to apply message; error=%s", e.getMessage()), e);
+ }
+
+ return profiler;
+ }
+
+ /**
+ * Gets a message or messages from the function arguments.
+ *
+ * @param arg The function argument containing the message(s).
+ * @return A list of messages
+ */
+ private List<JSONObject> getMessages(Object arg) {
+ List<JSONObject> messages;
+
+ if (arg instanceof String) {
+ messages = getMessagesFromString((String) arg);
+
+ } else if (arg instanceof List) {
+ messages = getMessagesFromList((List<String>) arg);
+
+ } else if (arg instanceof JSONObject) {
+ messages = Collections.singletonList((JSONObject) arg);
+
+ } else {
+ throw new IllegalArgumentException(format("invalid message: found '%s', expected String, List, or JSONObject",
+ ClassUtils.getShortClassName(arg, "null")));
}
- // there could be one or more messages
+ return messages;
+ }
+
+ /**
+ * Gets a message or messages from a List
+ *
+ * @param listOfStrings The function argument is a List of Strings.
+ * @return A list of messages.
+ */
+ private List<JSONObject> getMessagesFromList(List<String> listOfStrings) {
--- End diff --
Just a very small nit, but could we make that an iterable instead of a List since that's all you really need? We now have SETs in stellar too and I think we may end up creating lazy iterable transformation functions too. (e.g. `PROFILE_APPLY( LAZY_MAP( [ msg1, msg2 ], m -> REGEX_APPLY(...., m))`
---
[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/736
---
[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/736#discussion_r137822891
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java ---
@@ -131,50 +133,99 @@ public boolean isInitialized() {
@Override
public Object apply(List<Object> args, Context context) throws ParseException {
- // user must provide the message as a string
- String arg0 = Util.getArg(0, String.class, args);
- if(arg0 == null) {
- throw new IllegalArgumentException(format("expected string, found null"));
+ // the use can pass in one or more messages in a few different forms
+ Object arg0 = Util.getArg(0, Object.class, args);
+ List<JSONObject> messages = getMessages(arg0);
+
+ // user must provide the stand alone profiler
+ StandAloneProfiler profiler = Util.getArg(1, StandAloneProfiler.class, args);
+ try {
+ for (JSONObject message : messages) {
+ profiler.apply(message);
+ }
+
+ } catch (ExecutionException e) {
+ throw new IllegalArgumentException(format("Failed to apply message; error=%s", e.getMessage()), e);
+ }
+
+ return profiler;
+ }
+
+ /**
+ * Gets a message or messages from the function arguments.
+ *
+ * @param arg The function argument containing the message(s).
+ * @return A list of messages
+ */
+ private List<JSONObject> getMessages(Object arg) {
+ List<JSONObject> messages;
+
+ if (arg instanceof String) {
+ messages = getMessagesFromString((String) arg);
+
+ } else if (arg instanceof List) {
+ messages = getMessagesFromList((List<String>) arg);
+
+ } else if (arg instanceof JSONObject) {
+ messages = Collections.singletonList((JSONObject) arg);
+
+ } else {
+ throw new IllegalArgumentException(format("invalid message: found '%s', expected String, List, or JSONObject",
+ ClassUtils.getShortClassName(arg, "null")));
}
- // there could be one or more messages
+ return messages;
+ }
+
+ /**
+ * Gets a message or messages from a List
+ *
+ * @param listOfStrings The function argument is a List of Strings.
+ * @return A list of messages.
+ */
+ private List<JSONObject> getMessagesFromList(List<String> listOfStrings) {
--- End diff --
Yes, that's better. Will do.
---
[GitHub] metron pull request #736: METRON-1162 Apply Live Messages to the Profile Deb...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/736#discussion_r137825517
--- Diff: metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java ---
@@ -131,50 +133,99 @@ public boolean isInitialized() {
@Override
public Object apply(List<Object> args, Context context) throws ParseException {
- // user must provide the message as a string
- String arg0 = Util.getArg(0, String.class, args);
- if(arg0 == null) {
- throw new IllegalArgumentException(format("expected string, found null"));
+ // the use can pass in one or more messages in a few different forms
+ Object arg0 = Util.getArg(0, Object.class, args);
+ List<JSONObject> messages = getMessages(arg0);
+
+ // user must provide the stand alone profiler
+ StandAloneProfiler profiler = Util.getArg(1, StandAloneProfiler.class, args);
+ try {
+ for (JSONObject message : messages) {
+ profiler.apply(message);
+ }
+
+ } catch (ExecutionException e) {
+ throw new IllegalArgumentException(format("Failed to apply message; error=%s", e.getMessage()), e);
+ }
+
+ return profiler;
+ }
+
+ /**
+ * Gets a message or messages from the function arguments.
+ *
+ * @param arg The function argument containing the message(s).
+ * @return A list of messages
+ */
+ private List<JSONObject> getMessages(Object arg) {
+ List<JSONObject> messages;
+
+ if (arg instanceof String) {
+ messages = getMessagesFromString((String) arg);
+
+ } else if (arg instanceof List) {
+ messages = getMessagesFromList((List<String>) arg);
+
+ } else if (arg instanceof JSONObject) {
+ messages = Collections.singletonList((JSONObject) arg);
+
+ } else {
+ throw new IllegalArgumentException(format("invalid message: found '%s', expected String, List, or JSONObject",
+ ClassUtils.getShortClassName(arg, "null")));
}
- // there could be one or more messages
+ return messages;
+ }
+
+ /**
+ * Gets a message or messages from a List
+ *
+ * @param listOfStrings The function argument is a List of Strings.
+ * @return A list of messages.
+ */
+ private List<JSONObject> getMessagesFromList(List<String> listOfStrings) {
--- End diff --
Done and done.
---