You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by "Rider,Jonathan" <Jo...@capitalone.com> on 2016/05/17 20:07:04 UTC
JSON Structure for Soltra Parser
Hello Metron Team,
I’m going to be working on a Soltra parser, but before I do, I would like to know the convention on how the json should be stored after being parsed. Soltra is an xml format with a lot of nested structure. Is there a standard for flattening all of the JSON objects or is having nested JSON acceptable?
Additionally, at Capital One, we are stripping out most of the data from each file and only keeping 5 or so of the ~30 fields that could be taken. I’m assuming we should keep all of the fields for this general parser and companies that use it can remove the fields to fit their needs. Is this assumption correct?
Thanks,
Jonathan Rider
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: JSON Structure for Soltra Parser
Posted by Casey Stella <ce...@gmail.com>.
So, we do require at the moment flattened JSON, rather than hierarchical
JSON. As for which fields to index, I'd say your instinct is right, it'd
be good to grab as many as can be argued might be valuable.
On Tue, May 17, 2016 at 4:07 PM, Rider,Jonathan <
Jonathan.Rider@capitalone.com> wrote:
> Hello Metron Team,
>
> I’m going to be working on a Soltra parser, but before I do, I would like
> to know the convention on how the json should be stored after being parsed.
> Soltra is an xml format with a lot of nested structure. Is there a standard
> for flattening all of the JSON objects or is having nested JSON acceptable?
>
> Additionally, at Capital One, we are stripping out most of the data from
> each file and only keeping 5 or so of the ~30 fields that could be taken.
> I’m assuming we should keep all of the fields for this general parser and
> companies that use it can remove the fields to fit their needs. Is this
> assumption correct?
>
> Thanks,
>
> Jonathan Rider
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
Re: JSON Structure for Soltra Parser
Posted by "Rider,Jonathan" <Jo...@capitalone.com>.
Yes, the data is STIX with Cybox objects inside of it. Unfortunately, it seems like we’re actually doing a lot of preprocessing on it before it actually reaches Metron, so the parser that we have would not be very useful to anyone else. So I most likely won’t be submitting a pull request for it.
On 5/17/16, 5:56 PM, "Nick Allen" <ni...@nickallen.org> wrote:
>Are you asking about gathering threat intel information from a Soltra Edge
>server? Is this data in STIX[1] and/or Cybox[2] format? If so, we've
>already done some work in this area.
>
>[1] https://stixproject.github.io
>[2] https://cyboxproject.github.io
>
>On Tue, May 17, 2016 at 4:07 PM, Rider,Jonathan <
>Jonathan.Rider@capitalone.com> wrote:
>
>> Hello Metron Team,
>>
>> I’m going to be working on a Soltra parser, but before I do, I would like
>> to know the convention on how the json should be stored after being parsed.
>> Soltra is an xml format with a lot of nested structure. Is there a standard
>> for flattening all of the JSON objects or is having nested JSON acceptable?
>>
>> Additionally, at Capital One, we are stripping out most of the data from
>> each file and only keeping 5 or so of the ~30 fields that could be taken.
>> I’m assuming we should keep all of the fields for this general parser and
>> companies that use it can remove the fields to fit their needs. Is this
>> assumption correct?
>>
>> Thanks,
>>
>> Jonathan Rider
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>
>
>--
>Nick Allen <ni...@nickallen.org>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: JSON Structure for Soltra Parser
Posted by Nick Allen <ni...@nickallen.org>.
Are you asking about gathering threat intel information from a Soltra Edge
server? Is this data in STIX[1] and/or Cybox[2] format? If so, we've
already done some work in this area.
[1] https://stixproject.github.io
[2] https://cyboxproject.github.io
On Tue, May 17, 2016 at 4:07 PM, Rider,Jonathan <
Jonathan.Rider@capitalone.com> wrote:
> Hello Metron Team,
>
> I’m going to be working on a Soltra parser, but before I do, I would like
> to know the convention on how the json should be stored after being parsed.
> Soltra is an xml format with a lot of nested structure. Is there a standard
> for flattening all of the JSON objects or is having nested JSON acceptable?
>
> Additionally, at Capital One, we are stripping out most of the data from
> each file and only keeping 5 or so of the ~30 fields that could be taken.
> I’m assuming we should keep all of the fields for this general parser and
> companies that use it can remove the fields to fit their needs. Is this
> assumption correct?
>
> Thanks,
>
> Jonathan Rider
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
--
Nick Allen <ni...@nickallen.org>