You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by cestella <gi...@git.apache.org> on 2016/08/28 00:45:05 UTC
[GitHub] incubator-metron pull request #234: METRON-397: Add a stellar function to in...
GitHub user cestella opened a pull request:
https://github.com/apache/incubator-metron/pull/234
METRON-397: Add a stellar function to interact with the HBase enrichment table
We currently can only do simple HBase enrichment and threat intel via specialized adapters. With the advent of stellar and the incorporation as an Enrichment adapter, it would be nice to be able to :
* Retrieve the map of values stored in HBase given an enrichment type and indicator
* Determine existence of a given enrichment type and indicator
To that end, two Stellar functions were added:
* `ENRICHMENT_GET`
* `ENRICHMENT_EXISTS`
This PR relies upon METRON-396, so the PR commit will become much more readable after it is committed.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cestella/incubator-metron METRON-397
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/234.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #234
----
commit 0c0aec60ba56be64ed882072b6986c39ae9cb497
Author: cstella <ce...@gmail.com>
Date: 2016-08-28T00:41:36Z
METRON-397: Annotations based function resolver.
commit d77b048194b76809d43f5e985eda6d52347e835d
Author: cstella <ce...@gmail.com>
Date: 2016-08-28T00:27:49Z
Updating integration tests and implementation.
commit a068e1e0b52fdd18f0fc386308b5adafacc098ae
Author: cstella <ce...@gmail.com>
Date: 2016-08-28T00:30:49Z
Updating profiler.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/234
The AccessTracker allows us to track which keys in the hbase table are accessed. This is so that pruning an be done based on most current usage. It is used in the exisiting HBase Enrichment/ThreatIntel bolt, so it's being continued to be used here. There are two access trackers currently setup, one is a noop, which is the default. The other one is to capture the key being accessed in a bloom filter and write the filter out to HBase periodically. The `threatintel_bulk_pruner.sh` uses this table for pruning infrequently used rows.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/234
@nickwallen Thanks for the feedback. I'm going to disagree heartily that we should spend much time constructing javadocs for an API that is extremely fluid and changing. To put it frankly, we are in a state of extreme code churn as we grow the platform and those kinds of docs tend to go stale very quickly. I would rather see more focus on external documentation on the portions of the stack which are stable.
At this point in our maturity, we expect a certain ability to grok code by reading it with the judicious help of the integration tests, unit tests and a debugger. This is precisely why I spent so much time constructing an integration test harness, so we could trace code to better understand it.
TL;DR: Your position around javadocs and general code documentation is not committer consensus and may not (is not IMO) the best approach for this position in our maturity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/234
What is an AccessTracker? There's no docs and on first pass I don't understand how that relates to what you have described in METRON-397.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #234: METRON-397: Add a stellar function to in...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/234
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/234
Maybe I am missing them, but I don't see unit tests for the functionality.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/234
@nickwallen This is tested as part of the enrichment integration tests, but I will add a unit test for the specific function as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by merrimanr <gi...@git.apache.org>.
Github user merrimanr commented on the issue:
https://github.com/apache/incubator-metron/pull/234
+1. Ran unit and integration tests and everything looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #234: METRON-397: Add a stellar function to interact ...
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/234
Putting exactly this sort of description as javadoc for every class that you create would be very much appreciated.
I feel I spend too much effort tracing through Metron code and guessing as to what it is doing. Effort that would be avoided with a few well-placed comments and javadocs.
We need to minimize barriers for contributors. Its very difficult for others to contribute if we don't spend significant effort to document and comment that code that we are creating. I think in general this is one thing that we need to significantly improve upon on with Metron.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---