[DISCUSS] Metron REST API Requirements

[Ryan Merriman <me...@gmail.com>]

I would like to start a discussion around adding a REST API to Metron.  I
believe this is well worth the effort and would provide several benefits
including:

   - Giving users a uniform, well-tested and well-documented interface for
   interacting with Metron
   - Providing a layer of security (role-based, ACL, etc) around Metron
   functions
   - Enabling developers to build tools and interfaces on top of Metron
   without needing detailed knowledge of Metron internals

 The purpose of this discussion is to gather requirements around what the
REST API should include and the features it should provide.  Assuming we
decide a REST API is a worthy addition and can agree on a comprehensive
list of requirements, I will start a separate more technical discussion
around architecture and design.

I will start it off with a list of requirements that I think are
important.  The REST API should:

   - Be secure and adhere to modern web security standards
   - Provide a pluggable authentication mechanism
   - Be written in Java since the majority of Metron is written in Java
   - Be powered by a mature, open source web development framework with
   strong community support
   - Be as comprehensive as possible, meaning any task or function in
   Metron should be available in the REST API (are there exceptions?)
   - Have detailed documentation that always reflects the current version
   - Have the same level of installation tooling as Metron platform

The following is a initial list of functions I think should be included:

   - CRUD functions for parser, enrichment, and indexing zookeeper
   configurations
   - A function to retrieve versioning and auditing information for
   zookeeper configs
   - Functions for starting/stopping/enabling/disabling various Storm
   topologies
   - A function to retrieve sample data from various Kafka topics
   - A function to test Grok statements
   - A function to test parsers against sample data
   - A function to test enrichments against sample data
   - Functions that give status for the various components in Metron
   (HBase, Storm topologies, enrichments, etc)
   - A function that tests and runs Stellar statements against sample data.

I will continue to add to these lists as I think of more.  Looking forward
to hearing everyone’s ideas and input.

Ryan Merriman

Re: [DISCUSS] Metron REST API Requirements

["Zeolla@GMail.com" <ze...@gmail.com>]

I think this would be a great addition and your list is a good start.  I
have some specific feature suggestions that might fall under your prior
email, but overall I say "yes please."  :)

Jon

On Thu, Sep 29, 2016, 17:55 Ryan Merriman <me...@gmail.com> wrote:

> I would like to start a discussion around adding a REST API to Metron.  I
> believe this is well worth the effort and would provide several benefits
> including:
>
>    - Giving users a uniform, well-tested and well-documented interface for
>    interacting with Metron
>    - Providing a layer of security (role-based, ACL, etc) around Metron
>    functions
>    - Enabling developers to build tools and interfaces on top of Metron
>    without needing detailed knowledge of Metron internals
>
>  The purpose of this discussion is to gather requirements around what the
> REST API should include and the features it should provide.  Assuming we
> decide a REST API is a worthy addition and can agree on a comprehensive
> list of requirements, I will start a separate more technical discussion
> around architecture and design.
>
> I will start it off with a list of requirements that I think are
> important.  The REST API should:
>
>    - Be secure and adhere to modern web security standards
>    - Provide a pluggable authentication mechanism
>    - Be written in Java since the majority of Metron is written in Java
>    - Be powered by a mature, open source web development framework with
>    strong community support
>    - Be as comprehensive as possible, meaning any task or function in
>    Metron should be available in the REST API (are there exceptions?)
>    - Have detailed documentation that always reflects the current version
>    - Have the same level of installation tooling as Metron platform
>
> The following is a initial list of functions I think should be included:
>
>    - CRUD functions for parser, enrichment, and indexing zookeeper
>    configurations
>    - A function to retrieve versioning and auditing information for
>    zookeeper configs
>    - Functions for starting/stopping/enabling/disabling various Storm
>    topologies
>    - A function to retrieve sample data from various Kafka topics
>    - A function to test Grok statements
>    - A function to test parsers against sample data
>    - A function to test enrichments against sample data
>    - Functions that give status for the various components in Metron
>    (HBase, Storm topologies, enrichments, etc)
>    - A function that tests and runs Stellar statements against sample data.
>
> I will continue to add to these lists as I think of more.  Looking forward
> to hearing everyone’s ideas and input.
>
> Ryan Merriman
>
-- 

Jon