[jira] [Commented] (METRON-1046) STELLAR: Configurations should be able to reference Stellar File

["Matt Foley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/METRON-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087636#comment-16087636 ] 

Matt Foley commented on METRON-1046:
------------------------------------

This is a good idea.  I see it as related to METRON-987, which was the first step in allowing sequences of Stellar statements (aka "programs" :-) ) instead of just unrelated groups of single statements.  Your proposal lets us really work with programs as first-class entities.

However, some concerns need to be resolved:

1. Syntax.

Currently Stellar syntax and JSON fit neatly together.  Where would be the cut line for file substitutions?  Referencing METRON-987, would you only allow a file substitution where we currently allow square-bracketed Stellar string sequences?  What about Profile config syntax, where several chunks of code are intimately related (hence want to be located in the same file), but don't all get executed at the same time? (This is not a showstopper question because Profile configs are usually simple and don't really need file substitution.  The need is much greater in Enrichment.)

2. Config Updates.

Currently Metron configuration is stored in ZK, but managed through Curator libraries.  In return for considerable complexity, this gives instant update whenever a config changes, without effort in the BI part of the application.  This differs sharply from file-based configuration, where updates in response to config changes require either a restart, an explicit reload command from the user, or frequent state-checking in the application.

So currently people trying to develop a new enrichment can update the config, and immediately test the result, without restarting and without any explicit reload command.  We probably want to not lose this.

Rather than roll our own file pointer model, can we use JSON Pointers?  Will they work with Curator?  Both of those get into some fairly obscure features, that would need to be studied.  It also actually relates to the syntax question presented above.

> STELLAR: Configurations should be able to reference Stellar File 
> -----------------------------------------------------------------
>
>                 Key: METRON-1046
>                 URL: https://issues.apache.org/jira/browse/METRON-1046
>             Project: Metron
>          Issue Type: Improvement
>            Reporter: Otto Fowler
>            Assignee: Otto Fowler
>
> As we get more and more capability in Stellar, we also get more complexity.
> Managing and testing complex stellar statements only by running things through the parsers and enrichment is not idea.
> An improvement would be to allow configurations ( enrichment and parser ) to be able to reference a Stellar 'file', which perhaps can be shared between parsers.
> This File should should be loaded into a class which presents a clean interface into multiple possible stellar executions, operating on the same message map, returning that map ( or modifying it ).
> These files, or what ever we call the abstraction should be testable on their own as well.
> The files and their 'updates' can be managed in the same way that the updates to their parent configurations are.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)