You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Ali Nazemian <al...@gmail.com> on 2017/04/02 14:26:48 UTC
Metron capability to enrich a list granularity
Hi all,
I was wondering how I can achieve the following use case in the current
version of Metron?
I want to have attributes in the Metron JSON object that are an array. For
example, if a threat is impacting multiple users, they are all contained in
an attribute (e.g. user_id:[id1, id2, id3]). Now if I want to enrich the
event with data that requires the user_id as a key in enrichment stored in
HBASE, how would I do this?
Cheers,
Ali
Re: Metron capability to enrich a list granularity
Posted by Ali Nazemian <al...@gmail.com>.
Hi James,
I understand the concern about nested document and the corresponding
complexity, but the complexity of nested documents does not add any
complexity to having Multi-value objects as a list of attributes. Am I
missing something?
Cheers,
Ali
On Thu, Apr 6, 2017 at 6:35 AM, James Sirota <js...@apache.org> wrote:
> That is correct, nested Json structures introduced a high level of
> complexity for Solr. We did this work a while ago, but if I remember
> correctly we had to have child documents for nested pieces so for one
> ingest of a complex Json you could end up with multiple documents indexed.
> Template definition and management became more difficult as well and there
> were also some gotchas on the query side. For those reasons it became much
> simpler to just flatten everything.
>
>
> 04.04.2017, 05:46, "Nick Allen" <ni...@nickallen.org>:
>
> I have no knowledge of specifics. I was not involved with the original
> problem. I just know that I had to flatten the output from threat triage
> based on these concerns as part of PR #438 [1].
>
>
> [1] https://github.com/apache/incubator-metron/pull/438.
>
> On Mon, Apr 3, 2017 at 8:11 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> Thanks, Nick.
>
> Can you give me more information on what the problem with Solr indexing
> was at the first place? I've got some experience with Solr so I might be
> able to help to fix that situation.
>
> Regards,
> Ali
>
> On Mon, Apr 3, 2017 at 11:55 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> Up to this point, we have been making the assumption that we need to
> "flatten" complex data types like lists and maps before they get indexed.
> For example, a list like this...
>
> {
> users: [ mary, alice, bob ]
> }
>
>
> is flattened and ends up looking like this...
>
> {
> users.0: mary,
> users.1: alice,
> users.2: bob
> }
>
>
> The goal of the JIRA that I referenced is to make each indexer responsible
> for transforming the message in whatever way necessary to correctly index
> the data. This way enrichments and transformations that occur upstream
> don't have to worry about this.
>
> I *think* the specific issue is that Solr indexing may not work with
> complex data types like lists and maps in some scenarios. I *think*
> Elasticsearch indexing may be fine. Others may have more insight, but this
> is what I remember. It is probably worth the effort to validate this in
> your environment and see if any problems arise. It should be fairly simple
> to validate.
>
>
>
>
>
> On Sun, Apr 2, 2017 at 10:50 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> Thank you very much, Nick. I was not aware of the fact that Metron does
> not support the multi-value attribute. So, in this case, I need to have a
> Stellar function to deal with splitting data and mapping to enrichment CF.
> Is that correct?
>
> Regards,
> Ali
>
> On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> You could use the programmatic enrichment functions to do this. For
> instance, say you wanted to look-up the impacted users in a company
> 'phonebook' to get more information.
>
> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
> "cf")
>
> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
> "cf")
>
> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), "tb",
> "cf")
>
>
> Also note that there is an open JIRA to ensure that all of the index
> destinations can handle complex types in the message JSON. This may or may
> not impact your use case, but something to keep in mind.
>
> https://issues.apache.org/jira/browse/METRON-735
>
>
>
>
>
> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> Hi all,
>
>
> I was wondering how I can achieve the following use case in the current
> version of Metron?
>
>
>
> I want to have attributes in the Metron JSON object that are an array.
> For example, if a threat is impacting multiple users, they are all
> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
> to enrich the event with data that requires the user_id as a key in
> enrichment stored in HBASE, how would I do this?
>
>
> Cheers,
> Ali
>
>
>
>
>
> --
> A.Nazemian
>
>
>
>
>
> --
> A.Nazemian
>
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
>
--
A.Nazemian
Re: Metron capability to enrich a list granularity
Posted by Nick Allen <ni...@nickallen.org>.
I have no knowledge of specifics. I was not involved with the original
problem. I just know that I had to flatten the output from threat triage
based on these concerns as part of PR #438 [1].
[1] https://github.com/apache/incubator-metron/pull/438.
On Mon, Apr 3, 2017 at 8:11 PM, Ali Nazemian <al...@gmail.com> wrote:
> Thanks, Nick.
>
> Can you give me more information on what the problem with Solr indexing
> was at the first place? I've got some experience with Solr so I might be
> able to help to fix that situation.
>
> Regards,
> Ali
>
> On Mon, Apr 3, 2017 at 11:55 PM, Nick Allen <ni...@nickallen.org> wrote:
>
>> Up to this point, we have been making the assumption that we need to
>> "flatten" complex data types like lists and maps before they get indexed.
>> For example, a list like this...
>>
>> {
>> users: [ mary, alice, bob ]
>> }
>>
>>
>> is flattened and ends up looking like this...
>>
>> {
>> users.0: mary,
>> users.1: alice,
>> users.2: bob
>> }
>>
>>
>> The goal of the JIRA that I referenced is to make each indexer
>> responsible for transforming the message in whatever way necessary to
>> correctly index the data. This way enrichments and transformations that
>> occur upstream don't have to worry about this.
>>
>> I *think* the specific issue is that Solr indexing may not work with
>> complex data types like lists and maps in some scenarios. I *think*
>> Elasticsearch indexing may be fine. Others may have more insight, but this
>> is what I remember. It is probably worth the effort to validate this in
>> your environment and see if any problems arise. It should be fairly simple
>> to validate.
>>
>>
>>
>>
>>
>> On Sun, Apr 2, 2017 at 10:50 PM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>>> Thank you very much, Nick. I was not aware of the fact that Metron does
>>> not support the multi-value attribute. So, in this case, I need to have a
>>> Stellar function to deal with splitting data and mapping to enrichment CF.
>>> Is that correct?
>>>
>>> Regards,
>>> Ali
>>>
>>> On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <ni...@nickallen.org> wrote:
>>>
>>>> You could use the programmatic enrichment functions to do this. For
>>>> instance, say you wanted to look-up the impacted users in a company
>>>> 'phonebook' to get more information.
>>>>
>>>> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
>>>> "cf")
>>>>
>>>> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
>>>> "cf")
>>>>
>>>> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2),
>>>> "tb", "cf")
>>>>
>>>>
>>>> Also note that there is an open JIRA to ensure that all of the index
>>>> destinations can handle complex types in the message JSON. This may or may
>>>> not impact your use case, but something to keep in mind.
>>>>
>>>> https://issues.apache.org/jira/browse/METRON-735
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>>
>>>>> I was wondering how I can achieve the following use case in the
>>>>> current version of Metron?
>>>>>
>>>>>
>>>>>
>>>>> I want to have attributes in the Metron JSON object that are an
>>>>> array. For example, if a threat is impacting multiple users, they are all
>>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>>>> to enrich the event with data that requires the user_id as a key in
>>>>> enrichment stored in HBASE, how would I do this?
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Ali
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> A.Nazemian
>>>
>>
>>
>
>
> --
> A.Nazemian
>
Re: Metron capability to enrich a list granularity
Posted by Ali Nazemian <al...@gmail.com>.
Thanks, Nick.
Can you give me more information on what the problem with Solr indexing was
at the first place? I've got some experience with Solr so I might be able
to help to fix that situation.
Regards,
Ali
On Mon, Apr 3, 2017 at 11:55 PM, Nick Allen <ni...@nickallen.org> wrote:
> Up to this point, we have been making the assumption that we need to
> "flatten" complex data types like lists and maps before they get indexed.
> For example, a list like this...
>
> {
> users: [ mary, alice, bob ]
> }
>
>
> is flattened and ends up looking like this...
>
> {
> users.0: mary,
> users.1: alice,
> users.2: bob
> }
>
>
> The goal of the JIRA that I referenced is to make each indexer responsible
> for transforming the message in whatever way necessary to correctly index
> the data. This way enrichments and transformations that occur upstream
> don't have to worry about this.
>
> I *think* the specific issue is that Solr indexing may not work with
> complex data types like lists and maps in some scenarios. I *think*
> Elasticsearch indexing may be fine. Others may have more insight, but this
> is what I remember. It is probably worth the effort to validate this in
> your environment and see if any problems arise. It should be fairly simple
> to validate.
>
>
>
>
>
> On Sun, Apr 2, 2017 at 10:50 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
>> Thank you very much, Nick. I was not aware of the fact that Metron does
>> not support the multi-value attribute. So, in this case, I need to have a
>> Stellar function to deal with splitting data and mapping to enrichment CF.
>> Is that correct?
>>
>> Regards,
>> Ali
>>
>> On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <ni...@nickallen.org> wrote:
>>
>>> You could use the programmatic enrichment functions to do this. For
>>> instance, say you wanted to look-up the impacted users in a company
>>> 'phonebook' to get more information.
>>>
>>> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
>>> "cf")
>>>
>>> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
>>> "cf")
>>>
>>> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), "tb",
>>> "cf")
>>>
>>>
>>> Also note that there is an open JIRA to ensure that all of the index
>>> destinations can handle complex types in the message JSON. This may or may
>>> not impact your use case, but something to keep in mind.
>>>
>>> https://issues.apache.org/jira/browse/METRON-735
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>> I was wondering how I can achieve the following use case in the current
>>>> version of Metron?
>>>>
>>>>
>>>>
>>>> I want to have attributes in the Metron JSON object that are an array.
>>>> For example, if a threat is impacting multiple users, they are all
>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>>> to enrich the event with data that requires the user_id as a key in
>>>> enrichment stored in HBASE, how would I do this?
>>>>
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>
>>>
>>
>>
>> --
>> A.Nazemian
>>
>
>
--
A.Nazemian
Re: Metron capability to enrich a list granularity
Posted by Nick Allen <ni...@nickallen.org>.
Up to this point, we have been making the assumption that we need to
"flatten" complex data types like lists and maps before they get indexed.
For example, a list like this...
{
users: [ mary, alice, bob ]
}
is flattened and ends up looking like this...
{
users.0: mary,
users.1: alice,
users.2: bob
}
The goal of the JIRA that I referenced is to make each indexer responsible
for transforming the message in whatever way necessary to correctly index
the data. This way enrichments and transformations that occur upstream
don't have to worry about this.
I *think* the specific issue is that Solr indexing may not work with
complex data types like lists and maps in some scenarios. I *think*
Elasticsearch indexing may be fine. Others may have more insight, but this
is what I remember. It is probably worth the effort to validate this in
your environment and see if any problems arise. It should be fairly simple
to validate.
On Sun, Apr 2, 2017 at 10:50 PM, Ali Nazemian <al...@gmail.com> wrote:
> Thank you very much, Nick. I was not aware of the fact that Metron does
> not support the multi-value attribute. So, in this case, I need to have a
> Stellar function to deal with splitting data and mapping to enrichment CF.
> Is that correct?
>
> Regards,
> Ali
>
> On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <ni...@nickallen.org> wrote:
>
>> You could use the programmatic enrichment functions to do this. For
>> instance, say you wanted to look-up the impacted users in a company
>> 'phonebook' to get more information.
>>
>> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
>> "cf")
>>
>> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
>> "cf")
>>
>> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), "tb",
>> "cf")
>>
>>
>> Also note that there is an open JIRA to ensure that all of the index
>> destinations can handle complex types in the message JSON. This may or may
>> not impact your use case, but something to keep in mind.
>>
>> https://issues.apache.org/jira/browse/METRON-735
>>
>>
>>
>>
>>
>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>>
>>> I was wondering how I can achieve the following use case in the current
>>> version of Metron?
>>>
>>>
>>>
>>> I want to have attributes in the Metron JSON object that are an array.
>>> For example, if a threat is impacting multiple users, they are all
>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>> to enrich the event with data that requires the user_id as a key in
>>> enrichment stored in HBASE, how would I do this?
>>>
>>>
>>> Cheers,
>>> Ali
>>>
>>
>>
>
>
> --
> A.Nazemian
>
Re: Metron capability to enrich a list granularity
Posted by Ali Nazemian <al...@gmail.com>.
Thank you very much, Nick. I was not aware of the fact that Metron does not
support the multi-value attribute. So, in this case, I need to have a
Stellar function to deal with splitting data and mapping to enrichment CF.
Is that correct?
Regards,
Ali
On Mon, Apr 3, 2017 at 6:31 AM, Nick Allen <ni...@nickallen.org> wrote:
> You could use the programmatic enrichment functions to do this. For
> instance, say you wanted to look-up the impacted users in a company
> 'phonebook' to get more information.
>
> 'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
> "cf")
>
> 'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
> "cf")
>
> "impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), "tb",
> "cf")
>
>
> Also note that there is an open JIRA to ensure that all of the index
> destinations can handle complex types in the message JSON. This may or may
> not impact your use case, but something to keep in mind.
>
> https://issues.apache.org/jira/browse/METRON-735
>
>
>
>
>
> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
>> Hi all,
>>
>>
>> I was wondering how I can achieve the following use case in the current
>> version of Metron?
>>
>>
>>
>> I want to have attributes in the Metron JSON object that are an array.
>> For example, if a threat is impacting multiple users, they are all
>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>> to enrich the event with data that requires the user_id as a key in
>> enrichment stored in HBASE, how would I do this?
>>
>>
>> Cheers,
>> Ali
>>
>
>
--
A.Nazemian
Re: Metron capability to enrich a list granularity
Posted by Nick Allen <ni...@nickallen.org>.
You could use the programmatic enrichment functions to do this. For
instance, say you wanted to look-up the impacted users in a company
'phonebook' to get more information.
'impacted-user-0": ENRICHMENT_GET(''phonebook", GET(user_ids, 0), "tb",
"cf")
'impacted-user-1": ENRICHMENT_GET(''phonebook", GET(user_ids, 1), "tb",
"cf")
"impacted-user-2": ENRICHMENT_GET(''phonebook", GET(user_ids, 2), "tb",
"cf")
Also note that there is an open JIRA to ensure that all of the index
destinations can handle complex types in the message JSON. This may or may
not impact your use case, but something to keep in mind.
https://issues.apache.org/jira/browse/METRON-735
On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com> wrote:
> Hi all,
>
>
> I was wondering how I can achieve the following use case in the current
> version of Metron?
>
>
>
> I want to have attributes in the Metron JSON object that are an array.
> For example, if a threat is impacting multiple users, they are all
> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
> to enrich the event with data that requires the user_id as a key in
> enrichment stored in HBASE, how would I do this?
>
>
> Cheers,
> Ali
>
Re: Metron capability to enrich a list granularity
Posted by Ali Nazemian <al...@gmail.com>.
Thanks, Casey.
Unfortunately, I haven't got any actual estimation yet.
On Fri, Apr 7, 2017 at 4:07 AM, Casey Stella <ce...@gmail.com> wrote:
> Ok, so yeah, you've hit upon a limitation currently. Right now, via
> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>
> - enrichment_type - The enrichment type
> - indicator - The string indicator to look up
> - hbase_table - The HBase Table to use
> - column_family - The Column Family to use
>
> Right now we only accept a string for the indicator (which likely would be
> your user_id). You'd probably like to call ENRICHMENT_GET for each id in
> the user_id variable. We can't quite do that yet. There has been some
> talk about a MAP function created where you can apply a stellar function
> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
> 'enrichments', 't')) which would return a list containing the output of
> ENRICHMENT_GET for each call.
>
> There is another, more immediate change that could be made for this
> specific case. We could enable ENRICHMENT_GET to take a list of indicators
> as the second argument.
>
> Sorry, that doesn't exactly solve your problem in the immediate-case, but
> it provides some context for future fixes. ;) I don't suppose you know the
> length of the list beforehand, right? Even the maximum size?
>
> Casey
>
>
> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
>> Hi all,
>>
>>
>> I was wondering how I can achieve the following use case in the current
>> version of Metron?
>>
>>
>>
>> I want to have attributes in the Metron JSON object that are an array.
>> For example, if a threat is impacting multiple users, they are all
>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>> to enrich the event with data that requires the user_id as a key in
>> enrichment stored in HBASE, how would I do this?
>>
>>
>> Cheers,
>> Ali
>>
>
>
--
A.Nazemian
Re: Metron capability to enrich a list granularity
Posted by Ali Nazemian <al...@gmail.com>.
Great. Thanks, Casey.
On Sat, Apr 8, 2017 at 4:11 AM, Casey Stella <ce...@gmail.com> wrote:
> So, a couple of things, after some review on the PR , the syntax here
> changed to more java-like and I realized I was doing things the hard way
> (this is what I get for doing things at 1AM).
>
> That example should be:
>
> - MAP(user_ids, user_id -> MAP_GET('login_time', ENRICHMENT_GET('et',
> user_id, 'enrichments', 't') ) )
>
> Casey
>
> On Fri, Apr 7, 2017 at 12:16 AM, Casey Stella <ce...@gmail.com> wrote:
>
>> METRON-831, PR @ https://github.com/apache/incubator-metron/pull/517
>>
>> Just so we're clear, let's assume the following:
>>
>> - Enrichment table called 'enrichments'
>> - Enrichment CF called 't'
>> - A message field called user_ids that is a list of user IDs
>> - The enrichment type for this HBase enrichment is 'et'
>> - The indicator in the HBase enrichment is a user ID
>> - You want a certain field out of the HBase enrichment data per user
>> ID. Let's call that field 'login_time'
>>
>> In order do that with METRON-831, you'd do the following to get the login
>> time fields for the list of users. For the sake of simplicity, I'll break
>> it into temporary variables:
>>
>> - enriched_users := MAP( user_ids, &( user_id : ENRICHMENT_GET('et',
>> user_id, 'enrichments', 't') ) )
>> - login_times := MAP(enriched_users, &( enrichment :
>> MAP_GET(enrichment, 'login_time') ) )
>> - MAP_GET here retrieves the value associated with the key
>> 'login_time', which is the name.
>>
>> Since you probably don't want intermediate values out there, you might
>> want to smash that into one big statement (we need a way to remove
>> temporary variables in stellar enrichments, btw):
>>
>> - MAP(MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
>> 'enrichments', 't') ) ), &( enrichment : MAP_GET(enrichment, 'login_time')
>> ) )
>>
>> On a side-note, it might be nice to have an optional arg to
>> ENRICHMENT_GET that lets you specify just the fields to return. That would
>> simplify the call to:
>>
>> - MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
>> 'enrichments', 't', ['login_time']) ) )
>>
>>
>> On Thu, Apr 6, 2017 at 8:10 PM, Casey Stella <ce...@gmail.com> wrote:
>>
>>> There'll be a JIRA and a PR tonight ;) It sprung from the keyboard.
>>> I've been waiting for a good reason for some time. heh
>>>
>>> On Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ot...@gmail.com>
>>> wrote:
>>>
>>>> Is there a Jira for the MAP Casey?
>>>>
>>>>
>>>> On April 6, 2017 at 14:07:15, Casey Stella (cestella@gmail.com) wrote:
>>>>
>>>> Ok, so yeah, you've hit upon a limitation currently. Right now, via
>>>> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>>>>
>>>> - enrichment_type - The enrichment type
>>>> - indicator - The string indicator to look up
>>>> - hbase_table - The HBase Table to use
>>>> - column_family - The Column Family to use
>>>>
>>>> Right now we only accept a string for the indicator (which likely would
>>>> be your user_id). You'd probably like to call ENRICHMENT_GET for each id
>>>> in the user_id variable. We can't quite do that yet. There has been some
>>>> talk about a MAP function created where you can apply a stellar function
>>>> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
>>>> 'enrichments', 't')) which would return a list containing the output of
>>>> ENRICHMENT_GET for each call.
>>>>
>>>> There is another, more immediate change that could be made for this
>>>> specific case. We could enable ENRICHMENT_GET to take a list of indicators
>>>> as the second argument.
>>>>
>>>> Sorry, that doesn't exactly solve your problem in the immediate-case,
>>>> but it provides some context for future fixes. ;) I don't suppose you know
>>>> the length of the list beforehand, right? Even the maximum size?
>>>>
>>>> Casey
>>>>
>>>>
>>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>>
>>>>> I was wondering how I can achieve the following use case in the
>>>>> current version of Metron?
>>>>>
>>>>>
>>>>>
>>>>> I want to have attributes in the Metron JSON object that are an
>>>>> array. For example, if a threat is impacting multiple users, they are all
>>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>>>> to enrich the event with data that requires the user_id as a key in
>>>>> enrichment stored in HBASE, how would I do this?
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Ali
>>>>>
>>>>
>>>>
>>>
>>
>
--
A.Nazemian
Re: Metron capability to enrich a list granularity
Posted by Casey Stella <ce...@gmail.com>.
So, a couple of things, after some review on the PR , the syntax here
changed to more java-like and I realized I was doing things the hard way
(this is what I get for doing things at 1AM).
That example should be:
- MAP(user_ids, user_id -> MAP_GET('login_time', ENRICHMENT_GET('et',
user_id, 'enrichments', 't') ) )
Casey
On Fri, Apr 7, 2017 at 12:16 AM, Casey Stella <ce...@gmail.com> wrote:
> METRON-831, PR @ https://github.com/apache/incubator-metron/pull/517
>
> Just so we're clear, let's assume the following:
>
> - Enrichment table called 'enrichments'
> - Enrichment CF called 't'
> - A message field called user_ids that is a list of user IDs
> - The enrichment type for this HBase enrichment is 'et'
> - The indicator in the HBase enrichment is a user ID
> - You want a certain field out of the HBase enrichment data per user
> ID. Let's call that field 'login_time'
>
> In order do that with METRON-831, you'd do the following to get the login
> time fields for the list of users. For the sake of simplicity, I'll break
> it into temporary variables:
>
> - enriched_users := MAP( user_ids, &( user_id : ENRICHMENT_GET('et',
> user_id, 'enrichments', 't') ) )
> - login_times := MAP(enriched_users, &( enrichment :
> MAP_GET(enrichment, 'login_time') ) )
> - MAP_GET here retrieves the value associated with the key
> 'login_time', which is the name.
>
> Since you probably don't want intermediate values out there, you might
> want to smash that into one big statement (we need a way to remove
> temporary variables in stellar enrichments, btw):
>
> - MAP(MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
> 'enrichments', 't') ) ), &( enrichment : MAP_GET(enrichment, 'login_time')
> ) )
>
> On a side-note, it might be nice to have an optional arg to ENRICHMENT_GET
> that lets you specify just the fields to return. That would simplify the
> call to:
>
> - MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
> 'enrichments', 't', ['login_time']) ) )
>
>
> On Thu, Apr 6, 2017 at 8:10 PM, Casey Stella <ce...@gmail.com> wrote:
>
>> There'll be a JIRA and a PR tonight ;) It sprung from the keyboard. I've
>> been waiting for a good reason for some time. heh
>>
>> On Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Is there a Jira for the MAP Casey?
>>>
>>>
>>> On April 6, 2017 at 14:07:15, Casey Stella (cestella@gmail.com) wrote:
>>>
>>> Ok, so yeah, you've hit upon a limitation currently. Right now, via
>>> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>>>
>>> - enrichment_type - The enrichment type
>>> - indicator - The string indicator to look up
>>> - hbase_table - The HBase Table to use
>>> - column_family - The Column Family to use
>>>
>>> Right now we only accept a string for the indicator (which likely would
>>> be your user_id). You'd probably like to call ENRICHMENT_GET for each id
>>> in the user_id variable. We can't quite do that yet. There has been some
>>> talk about a MAP function created where you can apply a stellar function
>>> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
>>> 'enrichments', 't')) which would return a list containing the output of
>>> ENRICHMENT_GET for each call.
>>>
>>> There is another, more immediate change that could be made for this
>>> specific case. We could enable ENRICHMENT_GET to take a list of indicators
>>> as the second argument.
>>>
>>> Sorry, that doesn't exactly solve your problem in the immediate-case,
>>> but it provides some context for future fixes. ;) I don't suppose you know
>>> the length of the list beforehand, right? Even the maximum size?
>>>
>>> Casey
>>>
>>>
>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>> I was wondering how I can achieve the following use case in the current
>>>> version of Metron?
>>>>
>>>>
>>>>
>>>> I want to have attributes in the Metron JSON object that are an array.
>>>> For example, if a threat is impacting multiple users, they are all
>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>>> to enrich the event with data that requires the user_id as a key in
>>>> enrichment stored in HBASE, how would I do this?
>>>>
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>
>>>
>>
>
Re: Metron capability to enrich a list granularity
Posted by Ali Nazemian <al...@gmail.com>.
Great. Thanks, Casey. Neither Metron passes list attribute, nor we have a
clear understanding of the exact enrichment, so we need to wait a little
more...
On Fri, Apr 7, 2017 at 2:16 PM, Casey Stella <ce...@gmail.com> wrote:
> METRON-831, PR @ https://github.com/apache/incubator-metron/pull/517
>
> Just so we're clear, let's assume the following:
>
> - Enrichment table called 'enrichments'
> - Enrichment CF called 't'
> - A message field called user_ids that is a list of user IDs
> - The enrichment type for this HBase enrichment is 'et'
> - The indicator in the HBase enrichment is a user ID
> - You want a certain field out of the HBase enrichment data per user
> ID. Let's call that field 'login_time'
>
> In order do that with METRON-831, you'd do the following to get the login
> time fields for the list of users. For the sake of simplicity, I'll break
> it into temporary variables:
>
> - enriched_users := MAP( user_ids, &( user_id : ENRICHMENT_GET('et',
> user_id, 'enrichments', 't') ) )
> - login_times := MAP(enriched_users, &( enrichment :
> MAP_GET(enrichment, 'login_time') ) )
> - MAP_GET here retrieves the value associated with the key
> 'login_time', which is the name.
>
> Since you probably don't want intermediate values out there, you might
> want to smash that into one big statement (we need a way to remove
> temporary variables in stellar enrichments, btw):
>
> - MAP(MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
> 'enrichments', 't') ) ), &( enrichment : MAP_GET(enrichment, 'login_time')
> ) )
>
> On a side-note, it might be nice to have an optional arg to ENRICHMENT_GET
> that lets you specify just the fields to return. That would simplify the
> call to:
>
> - MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
> 'enrichments', 't', ['login_time']) ) )
>
>
> On Thu, Apr 6, 2017 at 8:10 PM, Casey Stella <ce...@gmail.com> wrote:
>
>> There'll be a JIRA and a PR tonight ;) It sprung from the keyboard. I've
>> been waiting for a good reason for some time. heh
>>
>> On Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ot...@gmail.com>
>> wrote:
>>
>>> Is there a Jira for the MAP Casey?
>>>
>>>
>>> On April 6, 2017 at 14:07:15, Casey Stella (cestella@gmail.com) wrote:
>>>
>>> Ok, so yeah, you've hit upon a limitation currently. Right now, via
>>> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>>>
>>> - enrichment_type - The enrichment type
>>> - indicator - The string indicator to look up
>>> - hbase_table - The HBase Table to use
>>> - column_family - The Column Family to use
>>>
>>> Right now we only accept a string for the indicator (which likely would
>>> be your user_id). You'd probably like to call ENRICHMENT_GET for each id
>>> in the user_id variable. We can't quite do that yet. There has been some
>>> talk about a MAP function created where you can apply a stellar function
>>> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
>>> 'enrichments', 't')) which would return a list containing the output of
>>> ENRICHMENT_GET for each call.
>>>
>>> There is another, more immediate change that could be made for this
>>> specific case. We could enable ENRICHMENT_GET to take a list of indicators
>>> as the second argument.
>>>
>>> Sorry, that doesn't exactly solve your problem in the immediate-case,
>>> but it provides some context for future fixes. ;) I don't suppose you know
>>> the length of the list beforehand, right? Even the maximum size?
>>>
>>> Casey
>>>
>>>
>>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>> I was wondering how I can achieve the following use case in the current
>>>> version of Metron?
>>>>
>>>>
>>>>
>>>> I want to have attributes in the Metron JSON object that are an array.
>>>> For example, if a threat is impacting multiple users, they are all
>>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>>> to enrich the event with data that requires the user_id as a key in
>>>> enrichment stored in HBASE, how would I do this?
>>>>
>>>>
>>>> Cheers,
>>>> Ali
>>>>
>>>
>>>
>>
>
--
A.Nazemian
Re: Metron capability to enrich a list granularity
Posted by Casey Stella <ce...@gmail.com>.
METRON-831, PR @ https://github.com/apache/incubator-metron/pull/517
Just so we're clear, let's assume the following:
- Enrichment table called 'enrichments'
- Enrichment CF called 't'
- A message field called user_ids that is a list of user IDs
- The enrichment type for this HBase enrichment is 'et'
- The indicator in the HBase enrichment is a user ID
- You want a certain field out of the HBase enrichment data per user
ID. Let's call that field 'login_time'
In order do that with METRON-831, you'd do the following to get the login
time fields for the list of users. For the sake of simplicity, I'll break
it into temporary variables:
- enriched_users := MAP( user_ids, &( user_id : ENRICHMENT_GET('et',
user_id, 'enrichments', 't') ) )
- login_times := MAP(enriched_users, &( enrichment : MAP_GET(enrichment,
'login_time') ) )
- MAP_GET here retrieves the value associated with the key
'login_time', which is the name.
Since you probably don't want intermediate values out there, you might want
to smash that into one big statement (we need a way to remove temporary
variables in stellar enrichments, btw):
- MAP(MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
'enrichments', 't') ) ), &( enrichment : MAP_GET(enrichment, 'login_time')
) )
On a side-note, it might be nice to have an optional arg to ENRICHMENT_GET
that lets you specify just the fields to return. That would simplify the
call to:
- MAP( user_ids, &( user_id : ENRICHMENT_GET('et', user_id,
'enrichments', 't', ['login_time']) ) )
On Thu, Apr 6, 2017 at 8:10 PM, Casey Stella <ce...@gmail.com> wrote:
> There'll be a JIRA and a PR tonight ;) It sprung from the keyboard. I've
> been waiting for a good reason for some time. heh
>
> On Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
>> Is there a Jira for the MAP Casey?
>>
>>
>> On April 6, 2017 at 14:07:15, Casey Stella (cestella@gmail.com) wrote:
>>
>> Ok, so yeah, you've hit upon a limitation currently. Right now, via
>> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>>
>> - enrichment_type - The enrichment type
>> - indicator - The string indicator to look up
>> - hbase_table - The HBase Table to use
>> - column_family - The Column Family to use
>>
>> Right now we only accept a string for the indicator (which likely would
>> be your user_id). You'd probably like to call ENRICHMENT_GET for each id
>> in the user_id variable. We can't quite do that yet. There has been some
>> talk about a MAP function created where you can apply a stellar function
>> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
>> 'enrichments', 't')) which would return a list containing the output of
>> ENRICHMENT_GET for each call.
>>
>> There is another, more immediate change that could be made for this
>> specific case. We could enable ENRICHMENT_GET to take a list of indicators
>> as the second argument.
>>
>> Sorry, that doesn't exactly solve your problem in the immediate-case, but
>> it provides some context for future fixes. ;) I don't suppose you know the
>> length of the list beforehand, right? Even the maximum size?
>>
>> Casey
>>
>>
>> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>>
>>> I was wondering how I can achieve the following use case in the current
>>> version of Metron?
>>>
>>>
>>>
>>> I want to have attributes in the Metron JSON object that are an array.
>>> For example, if a threat is impacting multiple users, they are all
>>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>>> to enrich the event with data that requires the user_id as a key in
>>> enrichment stored in HBASE, how would I do this?
>>>
>>>
>>> Cheers,
>>> Ali
>>>
>>
>>
>
Re: Metron capability to enrich a list granularity
Posted by Casey Stella <ce...@gmail.com>.
There'll be a JIRA and a PR tonight ;) It sprung from the keyboard. I've
been waiting for a good reason for some time. heh
On Thu, Apr 6, 2017 at 8:08 PM, Otto Fowler <ot...@gmail.com> wrote:
> Is there a Jira for the MAP Casey?
>
>
> On April 6, 2017 at 14:07:15, Casey Stella (cestella@gmail.com) wrote:
>
> Ok, so yeah, you've hit upon a limitation currently. Right now, via
> Stellar you can use ENRICHMENT_GET which takes the following parameters:
>
> - enrichment_type - The enrichment type
> - indicator - The string indicator to look up
> - hbase_table - The HBase Table to use
> - column_family - The Column Family to use
>
> Right now we only accept a string for the indicator (which likely would be
> your user_id). You'd probably like to call ENRICHMENT_GET for each id in
> the user_id variable. We can't quite do that yet. There has been some
> talk about a MAP function created where you can apply a stellar function
> across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
> 'enrichments', 't')) which would return a list containing the output of
> ENRICHMENT_GET for each call.
>
> There is another, more immediate change that could be made for this
> specific case. We could enable ENRICHMENT_GET to take a list of indicators
> as the second argument.
>
> Sorry, that doesn't exactly solve your problem in the immediate-case, but
> it provides some context for future fixes. ;) I don't suppose you know the
> length of the list beforehand, right? Even the maximum size?
>
> Casey
>
>
> On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com>
> wrote:
>
>> Hi all,
>>
>>
>> I was wondering how I can achieve the following use case in the current
>> version of Metron?
>>
>>
>>
>> I want to have attributes in the Metron JSON object that are an array.
>> For example, if a threat is impacting multiple users, they are all
>> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
>> to enrich the event with data that requires the user_id as a key in
>> enrichment stored in HBASE, how would I do this?
>>
>>
>> Cheers,
>> Ali
>>
>
>
Re: Metron capability to enrich a list granularity
Posted by Otto Fowler <ot...@gmail.com>.
Is there a Jira for the MAP Casey?
On April 6, 2017 at 14:07:15, Casey Stella (cestella@gmail.com) wrote:
Ok, so yeah, you've hit upon a limitation currently. Right now, via
Stellar you can use ENRICHMENT_GET which takes the following parameters:
- enrichment_type - The enrichment type
- indicator - The string indicator to look up
- hbase_table - The HBase Table to use
- column_family - The Column Family to use
Right now we only accept a string for the indicator (which likely would be
your user_id). You'd probably like to call ENRICHMENT_GET for each id in
the user_id variable. We can't quite do that yet. There has been some
talk about a MAP function created where you can apply a stellar function
across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
'enrichments', 't')) which would return a list containing the output of
ENRICHMENT_GET for each call.
There is another, more immediate change that could be made for this
specific case. We could enable ENRICHMENT_GET to take a list of indicators
as the second argument.
Sorry, that doesn't exactly solve your problem in the immediate-case, but
it provides some context for future fixes. ;) I don't suppose you know the
length of the list beforehand, right? Even the maximum size?
Casey
On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com> wrote:
> Hi all,
>
>
> I was wondering how I can achieve the following use case in the current
> version of Metron?
>
>
>
> I want to have attributes in the Metron JSON object that are an array.
> For example, if a threat is impacting multiple users, they are all
> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
> to enrich the event with data that requires the user_id as a key in
> enrichment stored in HBASE, how would I do this?
>
>
> Cheers,
> Ali
>
Re: Metron capability to enrich a list granularity
Posted by Casey Stella <ce...@gmail.com>.
Ok, so yeah, you've hit upon a limitation currently. Right now, via
Stellar you can use ENRICHMENT_GET which takes the following parameters:
- enrichment_type - The enrichment type
- indicator - The string indicator to look up
- hbase_table - The HBase Table to use
- column_family - The Column Family to use
Right now we only accept a string for the indicator (which likely would be
your user_id). You'd probably like to call ENRICHMENT_GET for each id in
the user_id variable. We can't quite do that yet. There has been some
talk about a MAP function created where you can apply a stellar function
across a list of values. i.e. MAP( user_id, @ENRICHMENT_GET('et', $,
'enrichments', 't')) which would return a list containing the output of
ENRICHMENT_GET for each call.
There is another, more immediate change that could be made for this
specific case. We could enable ENRICHMENT_GET to take a list of indicators
as the second argument.
Sorry, that doesn't exactly solve your problem in the immediate-case, but
it provides some context for future fixes. ;) I don't suppose you know the
length of the list beforehand, right? Even the maximum size?
Casey
On Sun, Apr 2, 2017 at 10:26 AM, Ali Nazemian <al...@gmail.com> wrote:
> Hi all,
>
>
> I was wondering how I can achieve the following use case in the current
> version of Metron?
>
>
>
> I want to have attributes in the Metron JSON object that are an array.
> For example, if a threat is impacting multiple users, they are all
> contained in an attribute (e.g. user_id:[id1, id2, id3]). Now if I want
> to enrich the event with data that requires the user_id as a key in
> enrichment stored in HBASE, how would I do this?
>
>
> Cheers,
> Ali
>