You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Otto Fowler <ot...@gmail.com> on 2017/04/27 13:30:59 UTC
[DISCUSS] Metron Rest to Install Parser (METRON-258)
So, assuming ( I know I know ) that METRON-777 eventually lands, it will
have lain the framework for the extension system for parsers out, including
the capability to create parsers outside the metron tree.
The next step is METRON-258, side loading of parsers. This will be the
effort to actually install 3rd party parser and other extensions ( stellar
libs for example ) into metron.
My first inclination is to add new capabilities to the REST services to
accomplish this. We would use the current services, or orchestrate them to
accomplish this.
The ‘process’ required would be the following:
I’ll pseudo code it out:
[rest endpoint]
installExtension(FILE extensionTgz, string extensionType)
workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
deployExtensionBundleToHdfs(workingDir)
if(parser == extenstionType)
pushConfigurationToZK(workingDir)
pushESTemplate(workingDir) *possibly depending on 777 review
setupLogRotate(workingDIr) * possibly depending on 777 review
saveExtensionTgzSomewhere(extensionTgz)
Then, the configuration ui would be extended to front the new api.
* there is still a question of how we get a parser to the ambari
configuration, such that when starting parsers it starts a parser - unless
that happens and I don’t see it
I would like some feedback on this approach.
* Is rest the right way? Should we do an ambari view instead?
* Is this too much to do in a rest call? Will it timeout etc?
???
Any ideas would be appreciated.
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Otto Fowler <ot...@gmail.com>.
I think this approach should be extended to the parsers too down the road
as I’ve mentioned before.
Installing a parser installed the configurations as templates, and from
there you create new instances ( named ).
Those instances are equivalent to what we have now.
Install parser -> configurations to ‘available area’ ( ES, ParserConfig,
???? )
UI -> Create Parser Instance -> take available, clone, edit save/deploy for
XXX
On April 27, 2017 at 10:28:18, Otto Fowler (ottobackwards@gmail.com) wrote:
Maybe the templates can be installed somewhere, not ES but some store. The
UI can list the available templates, and you can edit and deploy from the
UI?
They can still be packaged with the parser, just not go to ES. Does that
address your concerns?
On April 27, 2017 at 10:25:37, Otto Fowler (ottobackwards@gmail.com) wrote:
“As a parser developer, I want to build, test, and maintain my parser
extension as a single deliverable ‘package’”.
I think from the 3rd party developer perspective, everything required to
make the parser work, should be packaged and installed with the parser.
The point is they may only be working with the parser code, and never touch
ambari or other ‘metron’ source. Having essential components
managed otherwise makes it harder for the developers.
I know this pov is true for my teams.
On April 27, 2017 at 10:15:55, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Otto,
Happy to help around this.
Couple of questions and things to maybe think about…
REST seems fine for sending the payload, but the unpacking job should
probably be async (triggered by the job, but not responsible for replying,
i.e. the initial call gets an ACCEPTED response immediately before the
heavy lifting is done.
I’m still not 100% sold on the idea of ES templates being included in the
bundle, and would prefer that to be a broader “register the schema for the
parser” somewhere from which the Metron framework can generate templates,
this allows us to account for things like enrichments in templates, but the
principle you describe holds if we make it non-ES specific (some people use
solr for instance).
If we do this as a REST call, we can easily incorporate it to the UI, or
maybe add an ambari view later, so I’d say the REST approach is definitely
the right first step.
One other thing to think about is whether this belongs in the metron-rest
package (personally i think it probably does) or whether it is a separate
micro-service.
Anyway, happy to help in any way I can on this.
Simon
> On 27 Apr 2017, at 15:09, Otto Fowler <ot...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Otto Fowler <ot...@gmail.com>.
Maybe the templates can be installed somewhere, not ES but some store. The
UI can list the available templates, and you can edit and deploy from the
UI?
They can still be packaged with the parser, just not go to ES. Does that
address your concerns?
On April 27, 2017 at 10:25:37, Otto Fowler (ottobackwards@gmail.com) wrote:
“As a parser developer, I want to build, test, and maintain my parser
extension as a single deliverable ‘package’”.
I think from the 3rd party developer perspective, everything required to
make the parser work, should be packaged and installed with the parser.
The point is they may only be working with the parser code, and never touch
ambari or other ‘metron’ source. Having essential components
managed otherwise makes it harder for the developers.
I know this pov is true for my teams.
On April 27, 2017 at 10:15:55, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Otto,
Happy to help around this.
Couple of questions and things to maybe think about…
REST seems fine for sending the payload, but the unpacking job should
probably be async (triggered by the job, but not responsible for replying,
i.e. the initial call gets an ACCEPTED response immediately before the
heavy lifting is done.
I’m still not 100% sold on the idea of ES templates being included in the
bundle, and would prefer that to be a broader “register the schema for the
parser” somewhere from which the Metron framework can generate templates,
this allows us to account for things like enrichments in templates, but the
principle you describe holds if we make it non-ES specific (some people use
solr for instance).
If we do this as a REST call, we can easily incorporate it to the UI, or
maybe add an ambari view later, so I’d say the REST approach is definitely
the right first step.
One other thing to think about is whether this belongs in the metron-rest
package (personally i think it probably does) or whether it is a separate
micro-service.
Anyway, happy to help in any way I can on this.
Simon
> On 27 Apr 2017, at 15:09, Otto Fowler <ot...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Otto Fowler <ot...@gmail.com>.
“As a parser developer, I want to build, test, and maintain my parser
extension as a single deliverable ‘package’”.
I think from the 3rd party developer perspective, everything required to
make the parser work, should be packaged and installed with the parser.
The point is they may only be working with the parser code, and never touch
ambari or other ‘metron’ source. Having essential components
managed otherwise makes it harder for the developers.
I know this pov is true for my teams.
On April 27, 2017 at 10:15:55, Simon Elliston Ball (
simon@simonellistonball.com) wrote:
Otto,
Happy to help around this.
Couple of questions and things to maybe think about…
REST seems fine for sending the payload, but the unpacking job should
probably be async (triggered by the job, but not responsible for replying,
i.e. the initial call gets an ACCEPTED response immediately before the
heavy lifting is done.
I’m still not 100% sold on the idea of ES templates being included in the
bundle, and would prefer that to be a broader “register the schema for the
parser” somewhere from which the Metron framework can generate templates,
this allows us to account for things like enrichments in templates, but the
principle you describe holds if we make it non-ES specific (some people use
solr for instance).
If we do this as a REST call, we can easily incorporate it to the UI, or
maybe add an ambari view later, so I’d say the REST approach is definitely
the right first step.
One other thing to think about is whether this belongs in the metron-rest
package (personally i think it probably does) or whether it is a separate
micro-service.
Anyway, happy to help in any way I can on this.
Simon
> On 27 Apr 2017, at 15:09, Otto Fowler <ot...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions (
stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them
to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser -
unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Otto,
Happy to help around this.
Couple of questions and things to maybe think about…
REST seems fine for sending the payload, but the unpacking job should probably be async (triggered by the job, but not responsible for replying, i.e. the initial call gets an ACCEPTED response immediately before the heavy lifting is done.
I’m still not 100% sold on the idea of ES templates being included in the bundle, and would prefer that to be a broader “register the schema for the parser” somewhere from which the Metron framework can generate templates, this allows us to account for things like enrichments in templates, but the principle you describe holds if we make it non-ES specific (some people use solr for instance).
If we do this as a REST call, we can easily incorporate it to the UI, or maybe add an ambari view later, so I’d say the REST approach is definitely the right first step.
One other thing to think about is whether this belongs in the metron-rest package (personally i think it probably does) or whether it is a separate micro-service.
Anyway, happy to help in any way I can on this.
Simon
> On 27 Apr 2017, at 15:09, Otto Fowler <ot...@gmail.com> wrote:
>
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com) wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out, including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Ryan Merriman <me...@gmail.com>.
We don't have any examples of async endpoints yet. Fortunately Spring
documentation is awesome and it's easy to find examples:
https://spring.io/guides/gs/async-method/
On Thu, Apr 27, 2017 at 10:12 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Do we have any async methods in the current api I could look at?
>
>
> On April 27, 2017 at 10:45:09, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> I thought as much. I have not worked with the framework before however.
>
>
> On April 27, 2017 at 10:20:20, Ryan Merriman (merrimanr@gmail.com) wrote:
>
> I think leveraging the REST application would work for this use case.
> Services already exist for most of the functions listed in your pseudocode
> (HDFS read/write, Zookeeper read/write). Asynchronous functions are also
> supported so no issues there.
>
> On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > Also, if you want to help that would be great ;)
> >
> >
> > On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
> > wrote:
> >
> > So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> > have lain the framework for the extension system for parsers out,
> including
> > the capability to create parsers outside the metron tree.
> >
> > The next step is METRON-258, side loading of parsers. This will be the
> > effort to actually install 3rd party parser and other extensions (
> stellar
> > libs for example ) into metron.
> >
> > My first inclination is to add new capabilities to the REST services to
> > accomplish this. We would use the current services, or orchestrate them
> to
> > accomplish this.
> >
> > The ‘process’ required would be the following:
> >
> > I’ll pseudo code it out:
> >
> > [rest endpoint]
> > installExtension(FILE extensionTgz, string extensionType)
> >
> > workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
> >
> > deployExtensionBundleToHdfs(workingDir)
> >
> > if(parser == extenstionType)
> > pushConfigurationToZK(workingDir)
> > pushESTemplate(workingDir) *possibly depending on 777 review
> > setupLogRotate(workingDIr) * possibly depending on 777 review
> > saveExtensionTgzSomewhere(extensionTgz)
> >
> >
> > Then, the configuration ui would be extended to front the new api.
> >
> > * there is still a question of how we get a parser to the ambari
> > configuration, such that when starting parsers it starts a parser -
> unless
> > that happens and I don’t see it
> >
> >
> > I would like some feedback on this approach.
> >
> > * Is rest the right way? Should we do an ambari view instead?
> > * Is this too much to do in a rest call? Will it timeout etc?
> > ???
> >
> >
> > Any ideas would be appreciated.
> >
>
>
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Otto Fowler <ot...@gmail.com>.
Do we have any async methods in the current api I could look at?
On April 27, 2017 at 10:45:09, Otto Fowler (ottobackwards@gmail.com) wrote:
I thought as much. I have not worked with the framework before however.
On April 27, 2017 at 10:20:20, Ryan Merriman (merrimanr@gmail.com) wrote:
I think leveraging the REST application would work for this use case.
Services already exist for most of the functions listed in your pseudocode
(HDFS read/write, Zookeeper read/write). Asynchronous functions are also
supported so no issues there.
On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
>
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Otto Fowler <ot...@gmail.com>.
I thought as much. I have not worked with the framework before however.
On April 27, 2017 at 10:20:20, Ryan Merriman (merrimanr@gmail.com) wrote:
I think leveraging the REST application would work for this use case.
Services already exist for most of the functions listed in your pseudocode
(HDFS read/write, Zookeeper read/write). Asynchronous functions are also
supported so no issues there.
On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out,
including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions (
stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them
to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser -
unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
>
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Ryan Merriman <me...@gmail.com>.
I think leveraging the REST application would work for this use case.
Services already exist for most of the functions listed in your pseudocode
(HDFS read/write, Zookeeper read/write). Asynchronous functions are also
supported so no issues there.
On Thu, Apr 27, 2017 at 9:09 AM, Otto Fowler <ot...@gmail.com>
wrote:
> Also, if you want to help that would be great ;)
>
>
> On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> So, assuming ( I know I know ) that METRON-777 eventually lands, it will
> have lain the framework for the extension system for parsers out, including
> the capability to create parsers outside the metron tree.
>
> The next step is METRON-258, side loading of parsers. This will be the
> effort to actually install 3rd party parser and other extensions ( stellar
> libs for example ) into metron.
>
> My first inclination is to add new capabilities to the REST services to
> accomplish this. We would use the current services, or orchestrate them to
> accomplish this.
>
> The ‘process’ required would be the following:
>
> I’ll pseudo code it out:
>
> [rest endpoint]
> installExtension(FILE extensionTgz, string extensionType)
>
> workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
>
> deployExtensionBundleToHdfs(workingDir)
>
> if(parser == extenstionType)
> pushConfigurationToZK(workingDir)
> pushESTemplate(workingDir) *possibly depending on 777 review
> setupLogRotate(workingDIr) * possibly depending on 777 review
> saveExtensionTgzSomewhere(extensionTgz)
>
>
> Then, the configuration ui would be extended to front the new api.
>
> * there is still a question of how we get a parser to the ambari
> configuration, such that when starting parsers it starts a parser - unless
> that happens and I don’t see it
>
>
> I would like some feedback on this approach.
>
> * Is rest the right way? Should we do an ambari view instead?
> * Is this too much to do in a rest call? Will it timeout etc?
> ???
>
>
> Any ideas would be appreciated.
>
Re: [DISCUSS] Metron Rest to Install Parser (METRON-258)
Posted by Otto Fowler <ot...@gmail.com>.
Also, if you want to help that would be great ;)
On April 27, 2017 at 09:30:59, Otto Fowler (ottobackwards@gmail.com) wrote:
So, assuming ( I know I know ) that METRON-777 eventually lands, it will
have lain the framework for the extension system for parsers out, including
the capability to create parsers outside the metron tree.
The next step is METRON-258, side loading of parsers. This will be the
effort to actually install 3rd party parser and other extensions ( stellar
libs for example ) into metron.
My first inclination is to add new capabilities to the REST services to
accomplish this. We would use the current services, or orchestrate them to
accomplish this.
The ‘process’ required would be the following:
I’ll pseudo code it out:
[rest endpoint]
installExtension(FILE extensionTgz, string extensionType)
workingDir = unpackExtensionAssembly(tmpDir, extensionTgz)
deployExtensionBundleToHdfs(workingDir)
if(parser == extenstionType)
pushConfigurationToZK(workingDir)
pushESTemplate(workingDir) *possibly depending on 777 review
setupLogRotate(workingDIr) * possibly depending on 777 review
saveExtensionTgzSomewhere(extensionTgz)
Then, the configuration ui would be extended to front the new api.
* there is still a question of how we get a parser to the ambari
configuration, such that when starting parsers it starts a parser - unless
that happens and I don’t see it
I would like some feedback on this approach.
* Is rest the right way? Should we do an ambari view instead?
* Is this too much to do in a rest call? Will it timeout etc?
???
Any ideas would be appreciated.