You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by sirsean <gi...@git.apache.org> on 2016/01/15 23:34:52 UTC
[GitHub] incubator-metron pull request: Bro Parser Improvements
GitHub user sirsean opened a pull request:
https://github.com/apache/incubator-metron/pull/11
Bro Parser Improvements
The old Bro parser required that the data came wrapped in a key that
specified the protocol. But not every Bro produces data in that format;
now the parser supports that format AND a format that is flatter and the
protocol is just another attribute in the message.
This also contains some minor code cleanup in the Bro parser.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sirsean/incubator-metron bro-parser
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/11.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11
----
commit 41e159f61d5e68066024493e50079f3d1ba1302b
Author: Sean Schulte <si...@gmail.com>
Date: 2016-01-15T22:32:42Z
Bro Parser Improvements
The old Bro parser required that the data came wrapped in a key that
specified the protocol. But not every Bro produces data in that format;
now the parser supports that format AND a format that is flatter and the
protocol is just another attribute in the message.
This also contains some minor code cleanup in the Bro parser.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Bro Parser Improvements
Posted by ehashman <gi...@git.apache.org>.
Github user ehashman commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/11#discussion_r50460904
--- Diff: metron-streaming/Metron-MessageParsers/src/main/java/org/apache/metron/parsing/parsers/BasicBroParser.java ---
@@ -22,130 +22,122 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.apache.metron.tldextractor.BasicTldExtractor;
-
@SuppressWarnings("serial")
public class BasicBroParser extends AbstractParser {
- protected static final Logger _LOG = LoggerFactory
- .getLogger(BasicBroParser.class);
- private JSONCleaner cleaner = new JSONCleaner();
- private BasicTldExtractor tldex = new BasicTldExtractor();
-
- @SuppressWarnings("unchecked")
- public JSONObject parse(byte[] msg) {
-
- _LOG.trace("[Metron] Starting to parse incoming message");
-
- String raw_message = null;
-
- try {
-
- raw_message = new String(msg, "UTF-8");
- _LOG.trace("[Metron] Received message: " + raw_message);
-
- JSONObject cleaned_message = cleaner.Clean(raw_message);
- _LOG.debug("[Metron] Cleaned message: " + raw_message);
-
- if (cleaned_message == null || cleaned_message.isEmpty())
- throw new Exception("Unable to clean message: " + raw_message);
-
- String key = cleaned_message.keySet().iterator().next().toString();
-
- if (key == null)
- throw new Exception("Unable to retrieve key for message: "
- + raw_message);
-
- JSONObject payload = (JSONObject) cleaned_message.get(key);
-
- String originalString = " |";
- for (Object k : payload.keySet()) {
- originalString = originalString + " " + k.toString() + ":"
- + payload.get(k).toString();
- }
- originalString = key.toUpperCase() + originalString;
- payload.put("original_string", originalString);
-
- if (payload == null)
- throw new Exception("Unable to retrieve payload for message: "
- + raw_message);
-
- if (payload.containsKey("ts")) {
- String ts = payload.remove("ts").toString();
- payload.put("timestamp", ts);
- _LOG.trace("[Metron] Added ts to: " + payload);
- }
-
- if (payload.containsKey("id.orig_h")) {
- String source_ip = payload.remove("id.orig_h").toString();
- payload.put("ip_src_addr", source_ip);
- _LOG.trace("[Metron] Added ip_src_addr to: " + payload);
- } else if (payload.containsKey("tx_hosts")) {
- JSONArray txHosts = (JSONArray) payload.remove("tx_hosts");
- if (txHosts != null && !txHosts.isEmpty()) {
- payload.put("ip_src_addr", txHosts.get(0));
- _LOG.trace("[Metron] Added ip_src_addr to: " + payload);
- }
- }
-
- if (payload.containsKey("id.resp_h")) {
- String source_ip = payload.remove("id.resp_h").toString();
- payload.put("ip_dst_addr", source_ip);
- _LOG.trace("[Metron] Added ip_dst_addr to: " + payload);
- } else if (payload.containsKey("rx_hosts")) {
- JSONArray rxHosts = (JSONArray) payload.remove("rx_hosts");
- if (rxHosts != null && !rxHosts.isEmpty()) {
- payload.put("ip_dst_addr", rxHosts.get(0));
- _LOG.trace("[Metron] Added ip_dst_addr to: " + payload);
- }
- }
-
- if (payload.containsKey("id.orig_p")) {
- String source_port = payload.remove("id.orig_p").toString();
- payload.put("ip_src_port", source_port);
- _LOG.trace("[Metron] Added ip_src_port to: " + payload);
- }
- if (payload.containsKey("id.resp_p")) {
- String dest_port = payload.remove("id.resp_p").toString();
- payload.put("ip_dst_port", dest_port);
- _LOG.trace("[Metron] Added ip_dst_port to: " + payload);
- }
-
-// if (payload.containsKey("host")) {
-//
-// String host = payload.get("host").toString().trim();
-// String tld = tldex.extractTLD(host);
-//
-// payload.put("tld", tld);
-// _LOG.trace("[Metron] Added tld to: " + payload);
-//
-// }
-// if (payload.containsKey("query")) {
-// String host = payload.get("query").toString();
-// String[] parts = host.split("\\.");
-// int length = parts.length;
-// if (length >= 2) {
-// payload.put("tld", parts[length - 2] + "."
-// + parts[length - 1]);
-// _LOG.trace("[Metron] Added tld to: " + payload);
-// }
-// }
-
- _LOG.trace("[Metron] Inner message: " + payload);
-
- payload.put("protocol", key);
- _LOG.debug("[Metron] Returning parsed message: " + payload);
-
- return payload;
-
- } catch (Exception e) {
-
- _LOG.error("Unable to Parse Message: " + raw_message);
- e.printStackTrace();
- return null;
- }
-
- }
-
-
+ protected static final Logger _LOG = LoggerFactory
+ .getLogger(BasicBroParser.class);
+ private JSONCleaner cleaner = new JSONCleaner();
+
+ @SuppressWarnings("unchecked")
+ public JSONObject parse(byte[] msg) {
+
+ _LOG.trace("[Metron] Starting to parse incoming message");
+
+ String rawMessage = null;
+
+ try {
+ rawMessage = new String(msg, "UTF-8");
+ _LOG.trace("[Metron] Received message: " + rawMessage);
+
+ JSONObject cleanedMessage = cleaner.Clean(rawMessage);
+ _LOG.debug("[Metron] Cleaned message: " + cleanedMessage);
+
+ if (cleanedMessage == null || cleanedMessage.isEmpty()) {
+ throw new Exception("Unable to clean message: " + rawMessage);
+ }
+
+ String key;
+ JSONObject payload;
+ if (cleanedMessage.containsKey("type")) {
+ key = cleanedMessage.get("type").toString();
+ payload = cleanedMessage;
+ } else {
+ key = cleanedMessage.keySet().iterator().next().toString();
+
+ if (key == null) {
+ throw new Exception("Unable to retrieve key for message: "
+ + rawMessage);
+ }
+
+ payload = (JSONObject) cleanedMessage.get(key);
+ }
+
+ if (payload == null) {
+ throw new Exception("Unable to retrieve payload for message: "
+ + rawMessage);
+ }
+
+ String originalString = key.toUpperCase() + " |";
+ for (Object k : payload.keySet()) {
+ String value = payload.get(k).toString();
+ originalString += " " + k.toString() + ":" + value;
+ }
+ payload.put("original_string", originalString);
+
+ replaceKey(payload, "timestamp", new String[]{ "ts" });
+
+ if (payload.containsKey("timestamp")) {
+ try {
+ long timestamp = Long.parseLong(payload.get("timestamp").toString());
+ payload.put("timestamp", timestamp);
+ } catch (NumberFormatException nfe) {
+ _LOG.error(String.format("[Metron] tiemstamp is invalid: %s", payload.get("timestamp")));
--- End diff --
Typo: `tiemstamp -> timestamp`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Bro Parser Improvements
Posted by ehashman <gi...@git.apache.org>.
Github user ehashman commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/11#discussion_r50458006
--- Diff: metron-streaming/Metron-MessageParsers/src/test/java/org/apache/metron/parsing/test/BasicBroParserTest.java ---
@@ -29,6 +29,27 @@ public BasicBroParserTest() throws Exception {
jsonParser = new JSONParser();
}
+ public void testUnwrappedBroMessage() throws Exception {
--- End diff --
Should this be `ParseException`, like the rest of the tests here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Bro Parser Improvements
Posted by ehashman <gi...@git.apache.org>.
Github user ehashman commented on the pull request:
https://github.com/apache/incubator-metron/pull/11#issuecomment-173705999
Note for other reviewers: the way GitHub flowed this is not ideal. Try a `git diff -w` for easier reading.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Bro Parser Improvements
Posted by sirsean <gi...@git.apache.org>.
Github user sirsean commented on the pull request:
https://github.com/apache/incubator-metron/pull/11#issuecomment-173709064
@ehashman Thanks for the review, I fixed those things.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Bro Parser Improvements
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/11
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: Bro Parser Improvements
Posted by ehashman <gi...@git.apache.org>.
Github user ehashman commented on the pull request:
https://github.com/apache/incubator-metron/pull/11#issuecomment-173705778
Other than comments above, LGTM! +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---