You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by DomenicPuzio <gi...@git.apache.org> on 2016/05/10 19:11:14 UTC
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
GitHub user DomenicPuzio opened a pull request:
https://github.com/apache/incubator-metron/pull/115
[METRON-150] Adding WebSphere parser, unit tests, and integration tests
This PR is for [METRON-150](https://issues.apache.org/jira/browse/METRON-150). It satisfies the specified requirements of the WebSphere parser, passes unit tests, and passes integration tests. The parser uses Grok to pull out the fields present in all message types and uses Java to handle message-specific parsing.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DomenicPuzio/incubator-metron master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/115.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #115
----
commit eafc6c9b2e05c2f51922b5454efc367915cf5f6d
Author: xxl072 <xx...@captitalone.com>
Date: 2016-05-10T18:46:24Z
Adding WebSphere parser, unit tests, and integration tests to close METRON-150
commit a4e3f8cd98d45fe38d5cf35269e60ba59a117796
Author: xxl072 <xx...@captitalone.com>
Date: 2016-05-10T19:01:13Z
Cleaning up comments and whitespace
commit 85bb7584f50b06159341bdc6e7091c7cf1b14405
Author: xxl072 <xx...@captitalone.com>
Date: 2016-05-10T19:04:07Z
Adding Apache license information to classes
commit 92ab5134570387d3519bf6fad3e500d709850d13
Author: xxl072 <xx...@captitalone.com>
Date: 2016-05-10T19:06:17Z
Adding Apache license to one more file for METRON-150
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by DomenicPuzio <gi...@git.apache.org>.
Github user DomenicPuzio commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-219095558
@merrimanr and @cestella, thank you SO MUCH for the help! I really appreciate it! I will certainly follow the steps to test on vagrant for future parsers.
I'm very excited to be a contributor, and I'm already working on my next parser!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218770707
Looks good to me. Nice job @DomenicPuzio. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/115#discussion_r62781383
--- Diff: metron-platform/metron-integration-test/src/main/resources/sample/config/sensors/websphere.json ---
@@ -0,0 +1,22 @@
+{
+ "index": "websphere",
+ "batchSize": 1,
+ "enrichmentFieldMap":
--- End diff --
So, this structure got refactored a bit as of the last PR. Check out the example [here](https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#example) The upshot is that instead of `enrichmentFieldMap` and `threatIntelFieldMap` there are `enrichment` and `threatIntel` sections and `fieldMap` is in both.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by DomenicPuzio <gi...@git.apache.org>.
Github user DomenicPuzio commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218462405
@cestella, thank you so much for the feedback! I will get to work on fixing the 'websphere.json' file that was causing some failed tests. I agree that a negative case in the test class is a really good call.
@merrimanr, I agree that there is a good amount of overlap between this parser and the GrokParser class. There was some custom behavior needed to handle different log types, so I used GrokAsaParser (which also extends BasicParser) as a model. Off the top of my head, there are several sources that have different log types within a single source - Infoblox DNS, Checkpoint Firewall, Big IP VPN - so we should determine a strategy to address those. I like your idea of making the GrokParser more flexible so that we can easily extend it and add custom behavior for sources like WebSphere. Is this something that I should work to address in this PR or in a future refactoring? Either way, I am more than happy to work on that!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218772767
Great Job! Have you tested this on the full-dev-vagrant at all?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218603385
@DomenicPuzio, I think we should address it now since there are likely more Grok parsers being contributed soon. I submitted a pull request against your branch with an example.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/115#discussion_r62843136
--- Diff: metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/websphere/GrokWebSphereParserTest.java ---
@@ -0,0 +1,115 @@
+/**
+* Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.websphere;
+
+import static org.junit.Assert.assertEquals;
+import java.util.List;
+import org.json.simple.JSONObject;
+import org.junit.Test;
+
+public class GrokWebSphereParserTest {
+
+ @Test
+ public void testParseLoginLine() throws Exception {
+
+ //Set up parser, parse message
+ GrokWebSphereParser parser = new GrokWebSphereParser();
--- End diff --
We probably want a negative case for each of these scenarios (login, logout, rbm and other). What happens when you try to parse malformed lines or an empty line..don't want it to crash the topology.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218452417
Thanks so much for the contribution. This is going to be great when it gets in!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by DomenicPuzio <gi...@git.apache.org>.
Github user DomenicPuzio commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218661286
@merrimanr, thank you so much for the feedback and assistance. I have incorporated your changes! I have also added negative test cases (trying to parse empty or malformed lines) to help ensure that the topology will not crash when attempting to parse incorrectly formatted messages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-219048777
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-219048736
Ok, tested this. Sorry it took so long (vagrant drama).
Ok, so this is what I did to test this.
Spin up the full-dev-vagrant:
1. Do a build by running `mvn clean integration-test` from the
`metron-platform` directory
2. From the `metron-deployment/vagrant/full-dev-platform` directory run `vagrant up`
* Note: If you already have a vagrant machine running and you just want to
redeploy code, you should be able to do `vagrant provision --tags enrichment`
Log into the full-dev-vagrant and do the following:
* Optionally, stop some things to give myself enough memory headroom:
* Kill every running parser topology via the [storm UI](http://node1:8744/index.html) to proceed (that VM is pretty packed atm).
* Stop pcap replay via `/etc/init.d/pcap-replay stop`
* Kill bro via `/usr/local/bin/broctl` and type `stop` and then `exit` at the broctl prompt
* Marvel at the lack of fans running on your laptop
* Create the kafka topic for the sensor: `/usr/hdp/current/kafka-broker/bin//kafka-topics.sh --zookeeper localhost:2181 --create --topic websphere --partitions 1 --replication-factor 1`
* Create text file with sample data in it called `WebsphereOutput.txt`. This is essentially just [WebsphereOutput.txt](https://raw.githubusercontent.com/DomenicPuzio/incubator-metron/af50623dcb764fda0281b3657c0f40c993c958f3/metron-platform/metron-integration-test/src/main/resources/sample/data/SampleInput/WebsphereOutput.txt)
* Start the parser topology using `/usr/metron/0.1BETA/bin/start_parser_topology.sh`, in this case `/usr/metron/0.1BETA/bin/start_parser_topology.sh websphere`
* Add your data to the kafka queue you created earlier via `cat WebsphereOutput.txt | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic websphere`
* Wait a minute and then check the elasticsearch head plugin (if you haven't installed it, you can via `/usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head`).
* Browse to the `Browser` tab and click on the index named `${sensor_name}_index_${DATE}` (in my case: `websphere_index_2016.05.13.13`
* You should see some messages. Click on each one and inspect the JSON to ensure it looks right.
For future parsers, this is a good template for acceptance testing. You
shouldn't have to go through a full maven provisioning each time if you
keep the VM around and just run `vagrant provision -tags enrichment`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/115
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request: [METRON-150] Adding WebSphere parse...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the pull request:
https://github.com/apache/incubator-metron/pull/115#issuecomment-218266475
@DomenicPuzio, thanks this looks pretty good. Are you able to leverage the Grok parser that already exists (org.apache.metron.parsers.GrokParser) or is there some custom behavior needed? I see a lot of overlap between the custom Java Grok parser you wrote and what's currently there. If the current Grok parser is not flexible enough I think we should either update it to make it more flexible or extend it to provide custom behavior (rather than writing a new one). Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---