[jira] [Created] (METRON-195) Next Generation Metron UI

["George Vetticaden (JIRA)" <ji...@apache.org>]

George Vetticaden created METRON-195:
----------------------------------------

             Summary: Next Generation Metron UI
                 Key: METRON-195
                 URL: https://issues.apache.org/jira/browse/METRON-195
             Project: Metron
          Issue Type: Wish
            Reporter: George Vetticaden


Over the last few weeks, we have had the chance to interview over 20 different SOC/Security personnel from various organizations. These folks represented the following personas:
* Tier 1, 2 and 3 SOC Analyst
* SOC Manager
* Security Platform Architects
* SIEM Engineers
* Data Scientists

The interviews ranged from 10 minute to 2 hour conversations and some of the interview questions can be found on the following wiki page: https://cwiki.apache.org/confluence/display/METRON/SOC+Interview+Discussion+Guide


In addition to these questions, we had the opportunity to watch them use their current security tooling and also watch them use the current Metron based Kibana UI (for folks that had Metron deployed)

The user interviews are documented here:
https://cwiki.apache.org/confluence/display/METRON/User+Interviews
This provides us a valuable trove of requirements and insights about the challenges faced by SOC personnel.

While the current Metron UI based on Kibana 3/4 has some advantages including the ability to connect to different elastic indexes, dashboard and panel support, and rich support for different panel types,  it was clear  the current Metron UI lacks some key capabilities that most SOC personnel would require. Some key limitations identified based on these interviews included:
* Clunky unintuitive user interface. Someone has to to teach how to use Metron UI couple of times for a user to understand even how to start with it.
* All things around searching was very difficult to understand. This included things like
** Difference between search and filtering
** How to create a pinned/named query
** How to associate pinned queries to panels
** Search DSL was difficult to use
* No Authentication or RBAC support
* Alert Triaging not supported

In addition to these challenges, the following were key requirements that the SOC personnel deemed critical for Metron to have:
* Support for Alert Triaging. The ability for the user to group/search alerts and their related telemetries and create a case/ticket from it.
* Case Workflow Management
* Alert Queues that SOC analysts can pick up alerts to work on
* Authentication & Authorization/RBAC Support
* Make Search easier for a novice easier
* Faceting Support
* More robust pivoting functionality. 
* Help the advanced user write more complicated queries/searches
* More advanced visualization support including link analysis / Knowledge Graphs
* Curated/Filtered views of Alerts for SOC 1 Analysts
* KPIs  that provides accountability (e/g: average time spent on case, top 10 critical alerts, SLA violations..)
* Better/ more intelligent grouping of alerts into a "Meta-Alert"
* Drill down of Meta-Alert into the set of alerts the meta consists of. Drilling down on a raw alert to the the telemetry data sources
* Management UI Tooling around provisioning, monitoring and managing Metron.

Based on this data, some key design principles started to emerge that the Metron UI should adhere to:
# A Single Pane of Glass - One interface to monitor and act Telemetry feeds, Behavior anomalies, Alerts and Meta-Alerts, Metron system health, Investigate, slice and dice, hunt and seek, search & filter, Collaborate and share info and Identify and collect data
# Find Needles in the Haystack - Find that tiny detail that helps you create a better defense. Meta-Alerts are doorways to the data details  Meta-Alerts can contain hundreds of smaller related alerts. Search and Facet to dive deeper
# Same Data, Many Angles -  One data set can be visualized in many ways. Not only do we need to show many views of a data set out of the box. We must also allow for customization.
# It’s All Collaborative - I can share anything I see. I can preserve it for later. I can save it for myself or send to someone else. Once a collaboration begins, we can annotate and categorize data for future use and investigation. Oversight. All activity is audited. So what I do and the amount of time I spend doing it can be backtracked and replayed somehow.
# It Evolves - Off the shelve, the system will be powerful. However, it can grow and evolve with threat intelligence feeds, Behavior Modeling, App store style add-ons, and Learning Models. Learning Models use machine learning to predict anomalous behavior. It takes a pattern and alerts when similar patterns emerge. Behavior Modeling observers the average entity behavior and creates an expected range of behavior.  The system learns from what happens as well as what you feed it.
# A Tailored Fit - The system is personalized and customizable. Based on role, a user may get a specific interface. Based on experience, a user can modify interfaces to suit there personal preferences.
# I’m Never Lost - No matter how deep a user dives into the data, they always know where they are. I can move back through my diving to a previous result. The complexity of these systems can be immense. We need to design for interruption and the resuming of their place by providing some way-finding tools to guide users.
# It Teaches You - The system teaches novice users to become advanced users. As you use the system if gives you all the hints you need to become an expert. Complex things are made easy enough for anyone to be able to do.
# Think for Them - The system anticipates user needs and does the thinking for them. We put the burden of complexity on the software so that we simplify the user experience. Make the experience as human as possible, guide and help users achieve outcomes. Shorten the number clicks by decreasing complexity.

Hence, based on these challenges and requirements discovered from these interviews, this is a proposal to build a set of next generation user interfaces for Apache Metron that caters to the different SOC personas that will use Metron. Fore more details on SOC personnas see here: https://cwiki.apache.org/confluence/display/METRON/Metron+User+Personas+And+Benefits


As the below diagram illustrates, the proposal is to move away from Kibana and introduce a number of different views/UIs that caters to different types of SOC users across three Phases:
Phase 1 - This phase focuses on SOC analysts and  Managers. They includes  the following different Views/UIs:
Metron Investigator UI - Ability for SOC analyst to do alert triaging, threat  hunting and searching. It should also provide the ability to export a view based on the filters and seed a new Metron Case with that context.  
Case Management Workflow - The ability to manage Metron cases. This entails supporting different workflow statuses (new, pending, in-progress, escalate, etc..) It also should provide Metron case queue management functionality. 
KPI and Situational Awareness Dashboards - Set of Dashboards that provides situational awareness of the environment as well key performance indicators for managers. 
Phase 2 - The primary focus is on the Platform Engineers / Dev Ops that will manage the Metron application. This involves interfaces to manage topologies, enrichments, threat intel data sources, configuration, ec..
Phase 3 - This phase is all about the data scientist and providing an integrated analytical environment that uses powerful tools such as Zeppelin, Jupiter, Python etc..
     

 

This initial high level proposal will need to be fleshed out as the community provides their feedback.  




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)