[jira] [Comment Edited] (METRON-477) Support lower fidelity retention of network traffic over time

["Nick Allen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536344#comment-15536344 ] 

Nick Allen edited comment on METRON-477 at 9/30/16 4:02 PM:
------------------------------------------------------------

> In addition, both the full pcap and truncated pcap should be able to be retrieved using the same method (CLI, UI, etc.).

Retrieving the data is also an important piece.  As a user I just want to say "get me data related to IP 1.1.1.1".  The system should be able to retrieve all data related to that IP across all of the buckets.  

The data itself will be in different forms across each Bucket.  The query for IP 1.1.1.1 would return a subset of the results as raw pcap, a subset as truncated pcap, and a subset as daily summaries.

To implement this, there might have to be some kind of metadata that moves with the data across each bucket.  It is this metadata that the query functionality would use to respond to a user's query.


was (Author: nickwallen):
> In addition, both the full pcap and truncated pcap should be able to be retrieved using the same method (CLI, UI, etc.).

Retrieving the data is also an important piece.  As a user I just want to say "get me data related to IP 1.1.1.1".  The system should be able to retrieve all data related to that IP across all of the buckets.  The data itself will be in different forms across each Bucket.  The query for IP 1.1.1.1 would return a subset of the results as raw pcap, a subset as truncated pcap, and a subset as daily summaries.

To implement this, there might have to be some kind of metadata that moves with the data across each bucket.  It is this metadata that the query functionality would use to respond to a user's query.

> Support lower fidelity retention of network traffic over time
> -------------------------------------------------------------
>
>                 Key: METRON-477
>                 URL: https://issues.apache.org/jira/browse/METRON-477
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Jon Zeolla
>
> Currently fastcapa supports full pcap capture.  I would like to see the ability to retain network traffic for longer periods of time but at increasing less fidelity.  
> For instance:
>  - Full PCAP is ingested and stored in bucket 1
>  - Transition "Full PCAP" to "Truncated PCAP" after bucket 1 hits X size, stored in bucket 2
>  - Transform the truncated PCAP into flows or daily summaries after bucket 2 hits X size, stored in bucket 3
> This system should be setup so that the transition jobs are highly configurable (as in sizes for each bucket, truncation cutoffs length, transition ordering, etc.).  In addition, both the full pcap and truncated pcap should be able to be retrieved using the same method (CLI, UI, etc.).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)