[DISCUSS] SIDELOADING PARSERS: REST API

[Otto Fowler <ot...@gmail.com>]

As previously discussed here, I have been working on side loading of
parsers.  The goals of this work are:
* Make it possible of developers to create, maintain and deploy parsers
outside of the Metron code tree and not have to fork
* Create maven archetype support for developers of parsers
* Introduce a parser ‘lifecycle’ to support multiple instances and
configurations, states of being installed, under configuration, and deployed
etc.

I would like to have some discussion based on where I am after rebasing
onto METRON-671 which revamps deployment to be totally ambari based.


REST API:

The rest api works right now from what I can see, as everything is in
zookeeper correctly.

but,

Because we don’t have an extension ‘loading’ mechanism, the rest service
explicitly depends on all the parser jars.  And will have to if it wanted to
generically test parsing etc.


I think this is OK in the mean time, but is another thing to points to us
wanting an extension loading and packaging solution.

I also think we may want to add an add parser endpoint, that will handle
the deployment… but we have to sort out ambari, and have the rest stuff
all setup and working by default.

Re: [DISCUSS] SIDELOADING PARSERS: REST API

[Otto Fowler <ot...@gmail.com>]

Yes, I would do that as part of the nar integration ( adding VFSClassloader
to nar )



On March 10, 2017 at 15:13:18, Casey Stella (cestella@gmail.com) wrote:

This is another reason to consider deploying custom parsers to HDFS and
reusing the classloader, we can adjust the REST api to search the
centralized store of 3rd party parsers (HDFS).

On Fri, Mar 10, 2017 at 10:10 AM, Otto Fowler <ot...@gmail.com>
wrote:

> As previously discussed here, I have been working on side loading of
> parsers. The goals of this work are:
> * Make it possible of developers to create, maintain and deploy parsers
> outside of the Metron code tree and not have to fork
> * Create maven archetype support for developers of parsers
> * Introduce a parser ‘lifecycle’ to support multiple instances and
> configurations, states of being installed, under configuration, and
> deployed
> etc.
>
> I would like to have some discussion based on where I am after rebasing
> onto METRON-671 which revamps deployment to be totally ambari based.
>
>
> REST API:
>
> The rest api works right now from what I can see, as everything is in
> zookeeper correctly.
>
> but,
>
> Because we don’t have an extension ‘loading’ mechanism, the rest service
> explicitly depends on all the parser jars. And will have to if it wanted
> to
> generically test parsing etc.
>
>
> I think this is OK in the mean time, but is another thing to points to us
> wanting an extension loading and packaging solution.
>
> I also think we may want to add an add parser endpoint, that will handle
> the deployment… but we have to sort out ambari, and have the rest stuff
> all setup and working by default.
>

Re: [DISCUSS] SIDELOADING PARSERS: REST API

[Casey Stella <ce...@gmail.com>]

This is another reason to consider deploying custom parsers to HDFS and
reusing the classloader, we can adjust the REST api to search the
centralized store of 3rd party parsers (HDFS).

On Fri, Mar 10, 2017 at 10:10 AM, Otto Fowler <ot...@gmail.com>
wrote:

> As previously discussed here, I have been working on side loading of
> parsers.  The goals of this work are:
> * Make it possible of developers to create, maintain and deploy parsers
> outside of the Metron code tree and not have to fork
> * Create maven archetype support for developers of parsers
> * Introduce a parser ‘lifecycle’ to support multiple instances and
> configurations, states of being installed, under configuration, and
> deployed
> etc.
>
> I would like to have some discussion based on where I am after rebasing
> onto METRON-671 which revamps deployment to be totally ambari based.
>
>
> REST API:
>
> The rest api works right now from what I can see, as everything is in
> zookeeper correctly.
>
> but,
>
> Because we don’t have an extension ‘loading’ mechanism, the rest service
> explicitly depends on all the parser jars.  And will have to if it wanted
> to
> generically test parsing etc.
>
>
> I think this is OK in the mean time, but is another thing to points to us
> wanting an extension loading and packaging solution.
>
> I also think we may want to add an add parser endpoint, that will handle
> the deployment… but we have to sort out ambari, and have the rest stuff
> all setup and working by default.
>