You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/04/01 16:41:25 UTC
[jira] [Commented] (METRON-93) Generalize the HBase threat intel infrastructure to support enrichments

    [ https://issues.apache.org/jira/browse/METRON-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221780#comment-15221780 ] 

ASF GitHub Bot commented on METRON-93:
--------------------------------------

GitHub user cestella opened a pull request:

    https://github.com/apache/incubator-metron/pull/66

    METRON-93: Generalize the HBase threat intel infrastructure to support enrichments

    As it stands, the threat intel infrastructure is awkward. Namely, different threat intelligence sources must be pushed into separate hbase tables (malicious_ips separate form malicious_hosts, for instance). We'd rather have one table where the type is brought into the rowkey. Since this infrastructure is generalized, also add a simple hbase enrichment adapter.
    Furthermore, the configuration for a new enrichment should be added to zookeeper as part of the data load.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cestella/incubator-metron MET-129

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-metron/pull/66.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #66
    
----
commit 3e55257e092c0c5651d7b3b8c69d0d7ccd568fc7
Author: cstella <ce...@gmail.com>
Date:   2016-03-21T18:50:38Z

    Refactoring extractor framework to not use ThreatIntel anymore

commit 225ab51498c9bb978246a0ce1449d0dc97bbed97
Author: cstella <ce...@gmail.com>
Date:   2016-03-21T20:43:19Z

    Updating dataloads to not use threatintelkey/value anymore.

commit 72b3b0ec9814902c21774a7b0f486ee5af19d97a
Author: cstella <ce...@gmail.com>
Date:   2016-03-22T14:21:30Z

    Refactoring out the threat intel adapter stuff.

commit 6e4501d101ac41b407867c85ec76de0d31bc34b2
Author: cstella <ce...@gmail.com>
Date:   2016-03-23T18:13:24Z

    Updating test yaml.

commit 72eea03e35412f8720c0612776ced4709cca9697
Author: cstella <ce...@gmail.com>
Date:   2016-03-24T15:44:51Z

    Updating other adapters to use the new scheme.

commit b75c38a6343164aa050dba713f14322f861d6d78
Author: cstella <ce...@gmail.com>
Date:   2016-03-24T16:21:22Z

    Updating configuredbolt to not load configs unnecessarily.

commit 998f69e9d20396a55ccfeec07238b6b22fa386cf
Author: cstella <ce...@gmail.com>
Date:   2016-03-24T19:07:39Z

    updating

commit 3233e517b128de24dbfa5c2e9e58dcf32c1c876d
Author: cstella <ce...@gmail.com>
Date:   2016-03-25T14:07:04Z

    fixing utilities to update zookeeper.

commit 15bfed855815931fd1ed068c47506b5a039a5bfe
Author: cstella <ce...@gmail.com>
Date:   2016-03-28T17:32:12Z

    updated ansible

commit 0c58f2c32891e251c354ab23b09cc3db29ece817
Author: cstella <ce...@gmail.com>
Date:   2016-03-28T17:36:48Z

    Merge branch 'master' into MET-129

commit 19e3d94b5b2ee08498eee5b76d8e1479caa1c3ed
Author: cstella <ce...@gmail.com>
Date:   2016-03-28T18:57:42Z

    Whoops.

commit 00beb07eee99fc18e1d6f10530fa1c848889c3dc
Author: cstella <ce...@gmail.com>
Date:   2016-03-29T16:58:20Z

    Added test.

commit 2551af972e026add7eb4fecc5506907841456625
Author: cstella <ce...@gmail.com>
Date:   2016-03-29T18:16:13Z

    Unfolding the data in hbase before it goes into the index.

commit 0b0a62c2eca8ee1d8a72a6602e46c75200923ac0
Author: cstella <ce...@gmail.com>
Date:   2016-03-30T14:20:06Z

    Updating bolt.

commit 79dcbfff208aefd2bbfac09482dcaa393db39171
Author: cstella <ce...@gmail.com>
Date:   2016-03-30T21:02:52Z

    Updating to support multiget, etc.

commit d079c5d6df3248c68d1dd5304e1e257f7899a2c3
Author: cstella <ce...@gmail.com>
Date:   2016-03-31T19:01:24Z

    updating ansible.

commit 520cbf7350fdc43d09e769530d343d7aad5ee815
Author: cstella <ce...@gmail.com>
Date:   2016-04-01T14:07:29Z

    adding shell script

commit b4fc71e0a18ba9f2b897c3b01e16093903ed100a
Author: cstella <ce...@gmail.com>
Date:   2016-04-01T14:37:49Z

    Merging from master

----


> Generalize the HBase threat intel infrastructure to support enrichments
> -----------------------------------------------------------------------
>
>                 Key: METRON-93
>                 URL: https://issues.apache.org/jira/browse/METRON-93
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Casey Stella
>            Assignee: Casey Stella
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> As it stands, the threat intel infrastructure is awkward.  Namely, different threat intelligence sources must be pushed into separate hbase tables (malicious_ips separate form malicious_hosts, for instance).  We'd rather have one table where the type is brought into the rowkey.  Since this infrastructure is generalized, also add a simple hbase enrichment adapter.
> Furthermore, the configuration for a new enrichment should be added to zookeeper as part of the data load.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)