You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Ryan Merriman <me...@gmail.com> on 2016/10/13 22:10:21 UTC
[DISCUSS] Metron REST API Architecture and Design
I created a Jira to track this new feature at
https://issues.apache.org/jira/browse/METRON-503. I also started and
attached an architecture doc to that Jira with some of my ideas about how
we should implement it. Please feel free to review and comment or add to
it. Looking forward to everyone's ideas and feedback.
Ryan Merriman
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Sorry, this got buried in my inbox a bit. So the first thing that came to
mind was using the API for loading and exporting alerts/events with other
enterprise systems, but there appear to be other benefits as well. This
assumes that Metron is still intended to be the central location of all
security related data.
DXL -
https://github.com/opendxl/opendxl-client-python/wiki/What-can-you-do-with-OpenDXL%3F
and
https://github.com/opendxl/opendxl-client-python/wiki/Data-Exchange-Layer-Integration-Types
pxGrid - https://developer.cisco.com/site/pxgrid/documents/tutorial/ and
"Inquiries regarding joining the pxGrid ecosystem may be sent to
pxGrid-integration@cisco.com"
Jon
On Wed, Nov 2, 2016 at 5:14 PM Ryan Merriman <me...@gmail.com> wrote:
> Hey Jon sorry for the delay. These are new to me so I'm not sure where to
> start. Is there a common use case we could explore initially?
>
> Ryan
>
> On Thu, Oct 27, 2016 at 9:02 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Another thought that came to me recently - Has there been any
> consideration
> > of using some of the newer standards such as pxGrid
> > <https://developer.cisco.com/site/pxgrid/discover/overview/>, DXL
> > <https://community.mcafee.com/docs/DOC-7296>, etc. to integrate with
> other
> > systems and allow data loading/sharing? I'm not familiar enough with
> those
> > solutions to know if it fits with the current Metron API architecture,
> but
> > at least I wanted to put it out there.
> >
> > Jon
> >
> > On Mon, Oct 24, 2016 at 11:00 AM larry mccay <lm...@apache.org> wrote:
> >
> > > I think this is a reasonable direction for Metron.
> > >
> > > It probably makes sense to make sure that your services can accept
> > identity
> > > propagation from Knox so that they can also be proxied along with
> Hadoop
> > > APIs.
> > >
> > > FWIW - discussing whether a JAXRS programming model is something wanted
> > by
> > > the community wouldn't be a difficult thing to do.
> > > It hasn't been discussed to this point which is why it isn't
> documented -
> > > this is primarily because there hasn't been any explicit demand for it.
> > >
> > >
> > >
> > > On Mon, Oct 24, 2016 at 10:51 AM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > > > Ok, that sounds good to me, I primarily wanted to see whether or not
> it
> > > was
> > > > attempted and if it hit a technical roadblock. Thanks,
> > > >
> > > > Jon
> > > >
> > > > On Mon, Oct 24, 2016 at 10:11 AM Ryan Merriman <me...@gmail.com>
> > > > wrote:
> > > >
> > > > > There is also this comment in that Jira:
> > > > >
> > > > > "Adding the JAXRS services to knox is really easy but we haven't
> > really
> > > > > discussed whether it should be a programming model aspect of Knox
> in
> > > the
> > > > > community"
> > > > >
> > > > > I think that would need to be worked out before we move services
> into
> > > > Knox,
> > > > > if we decide we should do that.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <
> merrimanr@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I spent some time researching the Knox documentation and building
> > > > custom
> > > > > > services (hosted in Knox) was not well-documented. Spring is a
> > great
> > > > > > choice for that and I didn't really get any other feedback on
> which
> > > > > > application development framework to use. So that's what I went
> > > with.
> > > > > >
> > > > > > I think we should plan on adding Knox in front to leverage all
> the
> > > nice
> > > > > > security features and integrations. That is how most Knox
> > > integrations
> > > > > > (HFDS, Storm, etc) are architected.
> > > > > >
> > > > > > Ryan
> > > > > >
> > > > > > On Mon, Oct 24, 2016 at 8:37 AM, Zeolla@GMail.com <
> > zeolla@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> So it looks like, for now, you are not pursuing Knox (per
> comments
> > > in
> > > > > >> METRON-503 and then PR 316). Is there a reason for that?
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >> On Fri, Oct 14, 2016 at 5:59 PM Zeolla@GMail.com <
> > zeolla@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Good question :)
> > > > > >> >
> > > > > >> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <
> merrimanr@gmail.com>
> > > > > wrote:
> > > > > >> >
> > > > > >> > Jon,
> > > > > >> >
> > > > > >> > It wasn't intentional, I ran out of time and wanted to get
> > > something
> > > > > out
> > > > > >> > there. I think it certainly could be open ended though.
> Where
> > > > should
> > > > > >> the
> > > > > >> > REST API project be located?
> > > > > >> >
> > > > > >> > Ryan
> > > > > >> >
> > > > > >> > On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <
> > > zeolla@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Along the lines of:
> > > > > >> > > • Must be deployed to a machine with adequate resources so
> > that
> > > > > >> resource
> > > > > >> > > contention is avoided.
> > > > > >> > > • Will need network access to all other services within
> Metron
> > > > > >> > >
> > > > > >> > > Has there been any consideration of a "Metron Manager" node?
> > In
> > > > the
> > > > > >> old
> > > > > >> > > TP2
> > > > > >> > > bare metal install guide
> > > > > >> > > <https://cwiki.apache.org/confluence/display/METRON/
> > > > > >> > > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > > > > >> > > it mentions a "Metron Installer," but I could see the needs
> > for
> > > > that
> > > > > >> sort
> > > > > >> > > of a system expanding to have the following roles:
> > > > > >> > > - API
> > > > > >> > > - Metron UI
> > > > > >> > > - Metron Installer/upgrades
> > > > > >> > > - Edge/Gateway Node for data loading
> > > > > >> > > - Clients
> > > > > >> > >
> > > > > >> > > Also, at the end it ends mid-sentence under "Organization
> > within
> > > > > >> Metron,"
> > > > > >> > > was that intended to be open ended?
> > > > > >> > >
> > > > > >> > > Jon
> > > > > >> > >
> > > > > >> > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <
> > > > merrimanr@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > > I created a Jira to track this new feature at
> > > > > >> > > > https://issues.apache.org/jira/browse/METRON-503. I also
> > > > started
> > > > > >> and
> > > > > >> > > > attached an architecture doc to that Jira with some of my
> > > ideas
> > > > > >> about
> > > > > >> > how
> > > > > >> > > > we should implement it. Please feel free to review and
> > > comment
> > > > or
> > > > > >> add
> > > > > >> > to
> > > > > >> > > > it. Looking forward to everyone's ideas and feedback.
> > > > > >> > > >
> > > > > >> > > > Ryan Merriman
> > > > > >> > > >
> > > > > >> > > --
> > > > > >> > >
> > > > > >> > > Jon
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> >
> > > > > >> > Jon
> > > > > >> >
> > > > > >> --
> > > > > >>
> > > > > >> Jon
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > --
> > > >
> > > > Jon
> > > >
> > >
> > --
> >
> > Jon
> >
>
--
Jon
Sent from my mobile device
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by Ryan Merriman <me...@gmail.com>.
Hey Jon sorry for the delay. These are new to me so I'm not sure where to
start. Is there a common use case we could explore initially?
Ryan
On Thu, Oct 27, 2016 at 9:02 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Another thought that came to me recently - Has there been any consideration
> of using some of the newer standards such as pxGrid
> <https://developer.cisco.com/site/pxgrid/discover/overview/>, DXL
> <https://community.mcafee.com/docs/DOC-7296>, etc. to integrate with other
> systems and allow data loading/sharing? I'm not familiar enough with those
> solutions to know if it fits with the current Metron API architecture, but
> at least I wanted to put it out there.
>
> Jon
>
> On Mon, Oct 24, 2016 at 11:00 AM larry mccay <lm...@apache.org> wrote:
>
> > I think this is a reasonable direction for Metron.
> >
> > It probably makes sense to make sure that your services can accept
> identity
> > propagation from Knox so that they can also be proxied along with Hadoop
> > APIs.
> >
> > FWIW - discussing whether a JAXRS programming model is something wanted
> by
> > the community wouldn't be a difficult thing to do.
> > It hasn't been discussed to this point which is why it isn't documented -
> > this is primarily because there hasn't been any explicit demand for it.
> >
> >
> >
> > On Mon, Oct 24, 2016 at 10:51 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> > > Ok, that sounds good to me, I primarily wanted to see whether or not it
> > was
> > > attempted and if it hit a technical roadblock. Thanks,
> > >
> > > Jon
> > >
> > > On Mon, Oct 24, 2016 at 10:11 AM Ryan Merriman <me...@gmail.com>
> > > wrote:
> > >
> > > > There is also this comment in that Jira:
> > > >
> > > > "Adding the JAXRS services to knox is really easy but we haven't
> really
> > > > discussed whether it should be a programming model aspect of Knox in
> > the
> > > > community"
> > > >
> > > > I think that would need to be worked out before we move services into
> > > Knox,
> > > > if we decide we should do that.
> > > >
> > > >
> > > >
> > > > On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <me...@gmail.com>
> > > > wrote:
> > > >
> > > > > I spent some time researching the Knox documentation and building
> > > custom
> > > > > services (hosted in Knox) was not well-documented. Spring is a
> great
> > > > > choice for that and I didn't really get any other feedback on which
> > > > > application development framework to use. So that's what I went
> > with.
> > > > >
> > > > > I think we should plan on adding Knox in front to leverage all the
> > nice
> > > > > security features and integrations. That is how most Knox
> > integrations
> > > > > (HFDS, Storm, etc) are architected.
> > > > >
> > > > > Ryan
> > > > >
> > > > > On Mon, Oct 24, 2016 at 8:37 AM, Zeolla@GMail.com <
> zeolla@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> So it looks like, for now, you are not pursuing Knox (per comments
> > in
> > > > >> METRON-503 and then PR 316). Is there a reason for that?
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >> On Fri, Oct 14, 2016 at 5:59 PM Zeolla@GMail.com <
> zeolla@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Good question :)
> > > > >> >
> > > > >> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <me...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > Jon,
> > > > >> >
> > > > >> > It wasn't intentional, I ran out of time and wanted to get
> > something
> > > > out
> > > > >> > there. I think it certainly could be open ended though. Where
> > > should
> > > > >> the
> > > > >> > REST API project be located?
> > > > >> >
> > > > >> > Ryan
> > > > >> >
> > > > >> > On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <
> > zeolla@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Along the lines of:
> > > > >> > > • Must be deployed to a machine with adequate resources so
> that
> > > > >> resource
> > > > >> > > contention is avoided.
> > > > >> > > • Will need network access to all other services within Metron
> > > > >> > >
> > > > >> > > Has there been any consideration of a "Metron Manager" node?
> In
> > > the
> > > > >> old
> > > > >> > > TP2
> > > > >> > > bare metal install guide
> > > > >> > > <https://cwiki.apache.org/confluence/display/METRON/
> > > > >> > > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > > > >> > > it mentions a "Metron Installer," but I could see the needs
> for
> > > that
> > > > >> sort
> > > > >> > > of a system expanding to have the following roles:
> > > > >> > > - API
> > > > >> > > - Metron UI
> > > > >> > > - Metron Installer/upgrades
> > > > >> > > - Edge/Gateway Node for data loading
> > > > >> > > - Clients
> > > > >> > >
> > > > >> > > Also, at the end it ends mid-sentence under "Organization
> within
> > > > >> Metron,"
> > > > >> > > was that intended to be open ended?
> > > > >> > >
> > > > >> > > Jon
> > > > >> > >
> > > > >> > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <
> > > merrimanr@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > I created a Jira to track this new feature at
> > > > >> > > > https://issues.apache.org/jira/browse/METRON-503. I also
> > > started
> > > > >> and
> > > > >> > > > attached an architecture doc to that Jira with some of my
> > ideas
> > > > >> about
> > > > >> > how
> > > > >> > > > we should implement it. Please feel free to review and
> > comment
> > > or
> > > > >> add
> > > > >> > to
> > > > >> > > > it. Looking forward to everyone's ideas and feedback.
> > > > >> > > >
> > > > >> > > > Ryan Merriman
> > > > >> > > >
> > > > >> > > --
> > > > >> > >
> > > > >> > > Jon
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > Jon
> > > > >> >
> > > > >> --
> > > > >>
> > > > >> Jon
> > > > >>
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Another thought that came to me recently - Has there been any consideration
of using some of the newer standards such as pxGrid
<https://developer.cisco.com/site/pxgrid/discover/overview/>, DXL
<https://community.mcafee.com/docs/DOC-7296>, etc. to integrate with other
systems and allow data loading/sharing? I'm not familiar enough with those
solutions to know if it fits with the current Metron API architecture, but
at least I wanted to put it out there.
Jon
On Mon, Oct 24, 2016 at 11:00 AM larry mccay <lm...@apache.org> wrote:
> I think this is a reasonable direction for Metron.
>
> It probably makes sense to make sure that your services can accept identity
> propagation from Knox so that they can also be proxied along with Hadoop
> APIs.
>
> FWIW - discussing whether a JAXRS programming model is something wanted by
> the community wouldn't be a difficult thing to do.
> It hasn't been discussed to this point which is why it isn't documented -
> this is primarily because there hasn't been any explicit demand for it.
>
>
>
> On Mon, Oct 24, 2016 at 10:51 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Ok, that sounds good to me, I primarily wanted to see whether or not it
> was
> > attempted and if it hit a technical roadblock. Thanks,
> >
> > Jon
> >
> > On Mon, Oct 24, 2016 at 10:11 AM Ryan Merriman <me...@gmail.com>
> > wrote:
> >
> > > There is also this comment in that Jira:
> > >
> > > "Adding the JAXRS services to knox is really easy but we haven't really
> > > discussed whether it should be a programming model aspect of Knox in
> the
> > > community"
> > >
> > > I think that would need to be worked out before we move services into
> > Knox,
> > > if we decide we should do that.
> > >
> > >
> > >
> > > On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <me...@gmail.com>
> > > wrote:
> > >
> > > > I spent some time researching the Knox documentation and building
> > custom
> > > > services (hosted in Knox) was not well-documented. Spring is a great
> > > > choice for that and I didn't really get any other feedback on which
> > > > application development framework to use. So that's what I went
> with.
> > > >
> > > > I think we should plan on adding Knox in front to leverage all the
> nice
> > > > security features and integrations. That is how most Knox
> integrations
> > > > (HFDS, Storm, etc) are architected.
> > > >
> > > > Ryan
> > > >
> > > > On Mon, Oct 24, 2016 at 8:37 AM, Zeolla@GMail.com <ze...@gmail.com>
> > > > wrote:
> > > >
> > > >> So it looks like, for now, you are not pursuing Knox (per comments
> in
> > > >> METRON-503 and then PR 316). Is there a reason for that?
> > > >>
> > > >> Jon
> > > >>
> > > >> On Fri, Oct 14, 2016 at 5:59 PM Zeolla@GMail.com <ze...@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Good question :)
> > > >> >
> > > >> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <me...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > Jon,
> > > >> >
> > > >> > It wasn't intentional, I ran out of time and wanted to get
> something
> > > out
> > > >> > there. I think it certainly could be open ended though. Where
> > should
> > > >> the
> > > >> > REST API project be located?
> > > >> >
> > > >> > Ryan
> > > >> >
> > > >> > On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <
> zeolla@gmail.com
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Along the lines of:
> > > >> > > • Must be deployed to a machine with adequate resources so that
> > > >> resource
> > > >> > > contention is avoided.
> > > >> > > • Will need network access to all other services within Metron
> > > >> > >
> > > >> > > Has there been any consideration of a "Metron Manager" node? In
> > the
> > > >> old
> > > >> > > TP2
> > > >> > > bare metal install guide
> > > >> > > <https://cwiki.apache.org/confluence/display/METRON/
> > > >> > > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > > >> > > it mentions a "Metron Installer," but I could see the needs for
> > that
> > > >> sort
> > > >> > > of a system expanding to have the following roles:
> > > >> > > - API
> > > >> > > - Metron UI
> > > >> > > - Metron Installer/upgrades
> > > >> > > - Edge/Gateway Node for data loading
> > > >> > > - Clients
> > > >> > >
> > > >> > > Also, at the end it ends mid-sentence under "Organization within
> > > >> Metron,"
> > > >> > > was that intended to be open ended?
> > > >> > >
> > > >> > > Jon
> > > >> > >
> > > >> > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <
> > merrimanr@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > I created a Jira to track this new feature at
> > > >> > > > https://issues.apache.org/jira/browse/METRON-503. I also
> > started
> > > >> and
> > > >> > > > attached an architecture doc to that Jira with some of my
> ideas
> > > >> about
> > > >> > how
> > > >> > > > we should implement it. Please feel free to review and
> comment
> > or
> > > >> add
> > > >> > to
> > > >> > > > it. Looking forward to everyone's ideas and feedback.
> > > >> > > >
> > > >> > > > Ryan Merriman
> > > >> > > >
> > > >> > > --
> > > >> > >
> > > >> > > Jon
> > > >> > >
> > > >> >
> > > >> > --
> > > >> >
> > > >> > Jon
> > > >> >
> > > >> --
> > > >>
> > > >> Jon
> > > >>
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
--
Jon
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by larry mccay <lm...@apache.org>.
I think this is a reasonable direction for Metron.
It probably makes sense to make sure that your services can accept identity
propagation from Knox so that they can also be proxied along with Hadoop
APIs.
FWIW - discussing whether a JAXRS programming model is something wanted by
the community wouldn't be a difficult thing to do.
It hasn't been discussed to this point which is why it isn't documented -
this is primarily because there hasn't been any explicit demand for it.
On Mon, Oct 24, 2016 at 10:51 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Ok, that sounds good to me, I primarily wanted to see whether or not it was
> attempted and if it hit a technical roadblock. Thanks,
>
> Jon
>
> On Mon, Oct 24, 2016 at 10:11 AM Ryan Merriman <me...@gmail.com>
> wrote:
>
> > There is also this comment in that Jira:
> >
> > "Adding the JAXRS services to knox is really easy but we haven't really
> > discussed whether it should be a programming model aspect of Knox in the
> > community"
> >
> > I think that would need to be worked out before we move services into
> Knox,
> > if we decide we should do that.
> >
> >
> >
> > On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <me...@gmail.com>
> > wrote:
> >
> > > I spent some time researching the Knox documentation and building
> custom
> > > services (hosted in Knox) was not well-documented. Spring is a great
> > > choice for that and I didn't really get any other feedback on which
> > > application development framework to use. So that's what I went with.
> > >
> > > I think we should plan on adding Knox in front to leverage all the nice
> > > security features and integrations. That is how most Knox integrations
> > > (HFDS, Storm, etc) are architected.
> > >
> > > Ryan
> > >
> > > On Mon, Oct 24, 2016 at 8:37 AM, Zeolla@GMail.com <ze...@gmail.com>
> > > wrote:
> > >
> > >> So it looks like, for now, you are not pursuing Knox (per comments in
> > >> METRON-503 and then PR 316). Is there a reason for that?
> > >>
> > >> Jon
> > >>
> > >> On Fri, Oct 14, 2016 at 5:59 PM Zeolla@GMail.com <ze...@gmail.com>
> > >> wrote:
> > >>
> > >> > Good question :)
> > >> >
> > >> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <me...@gmail.com>
> > wrote:
> > >> >
> > >> > Jon,
> > >> >
> > >> > It wasn't intentional, I ran out of time and wanted to get something
> > out
> > >> > there. I think it certainly could be open ended though. Where
> should
> > >> the
> > >> > REST API project be located?
> > >> >
> > >> > Ryan
> > >> >
> > >> > On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > >> > wrote:
> > >> >
> > >> > > Along the lines of:
> > >> > > • Must be deployed to a machine with adequate resources so that
> > >> resource
> > >> > > contention is avoided.
> > >> > > • Will need network access to all other services within Metron
> > >> > >
> > >> > > Has there been any consideration of a "Metron Manager" node? In
> the
> > >> old
> > >> > > TP2
> > >> > > bare metal install guide
> > >> > > <https://cwiki.apache.org/confluence/display/METRON/
> > >> > > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > >> > > it mentions a "Metron Installer," but I could see the needs for
> that
> > >> sort
> > >> > > of a system expanding to have the following roles:
> > >> > > - API
> > >> > > - Metron UI
> > >> > > - Metron Installer/upgrades
> > >> > > - Edge/Gateway Node for data loading
> > >> > > - Clients
> > >> > >
> > >> > > Also, at the end it ends mid-sentence under "Organization within
> > >> Metron,"
> > >> > > was that intended to be open ended?
> > >> > >
> > >> > > Jon
> > >> > >
> > >> > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <
> merrimanr@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > I created a Jira to track this new feature at
> > >> > > > https://issues.apache.org/jira/browse/METRON-503. I also
> started
> > >> and
> > >> > > > attached an architecture doc to that Jira with some of my ideas
> > >> about
> > >> > how
> > >> > > > we should implement it. Please feel free to review and comment
> or
> > >> add
> > >> > to
> > >> > > > it. Looking forward to everyone's ideas and feedback.
> > >> > > >
> > >> > > > Ryan Merriman
> > >> > > >
> > >> > > --
> > >> > >
> > >> > > Jon
> > >> > >
> > >> >
> > >> > --
> > >> >
> > >> > Jon
> > >> >
> > >> --
> > >>
> > >> Jon
> > >>
> > >
> > >
> >
> --
>
> Jon
>
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Ok, that sounds good to me, I primarily wanted to see whether or not it was
attempted and if it hit a technical roadblock. Thanks,
Jon
On Mon, Oct 24, 2016 at 10:11 AM Ryan Merriman <me...@gmail.com> wrote:
> There is also this comment in that Jira:
>
> "Adding the JAXRS services to knox is really easy but we haven't really
> discussed whether it should be a programming model aspect of Knox in the
> community"
>
> I think that would need to be worked out before we move services into Knox,
> if we decide we should do that.
>
>
>
> On Mon, Oct 24, 2016 at 9:06 AM, Ryan Merriman <me...@gmail.com>
> wrote:
>
> > I spent some time researching the Knox documentation and building custom
> > services (hosted in Knox) was not well-documented. Spring is a great
> > choice for that and I didn't really get any other feedback on which
> > application development framework to use. So that's what I went with.
> >
> > I think we should plan on adding Knox in front to leverage all the nice
> > security features and integrations. That is how most Knox integrations
> > (HFDS, Storm, etc) are architected.
> >
> > Ryan
> >
> > On Mon, Oct 24, 2016 at 8:37 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> >
> >> So it looks like, for now, you are not pursuing Knox (per comments in
> >> METRON-503 and then PR 316). Is there a reason for that?
> >>
> >> Jon
> >>
> >> On Fri, Oct 14, 2016 at 5:59 PM Zeolla@GMail.com <ze...@gmail.com>
> >> wrote:
> >>
> >> > Good question :)
> >> >
> >> > On Fri, Oct 14, 2016, 17:07 Ryan Merriman <me...@gmail.com>
> wrote:
> >> >
> >> > Jon,
> >> >
> >> > It wasn't intentional, I ran out of time and wanted to get something
> out
> >> > there. I think it certainly could be open ended though. Where should
> >> the
> >> > REST API project be located?
> >> >
> >> > Ryan
> >> >
> >> > On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <ze...@gmail.com>
> >> > wrote:
> >> >
> >> > > Along the lines of:
> >> > > • Must be deployed to a machine with adequate resources so that
> >> resource
> >> > > contention is avoided.
> >> > > • Will need network access to all other services within Metron
> >> > >
> >> > > Has there been any consideration of a "Metron Manager" node? In the
> >> old
> >> > > TP2
> >> > > bare metal install guide
> >> > > <https://cwiki.apache.org/confluence/display/METRON/
> >> > > Metron+Installation+on+an+Ambari-Managed+Cluster>
> >> > > it mentions a "Metron Installer," but I could see the needs for that
> >> sort
> >> > > of a system expanding to have the following roles:
> >> > > - API
> >> > > - Metron UI
> >> > > - Metron Installer/upgrades
> >> > > - Edge/Gateway Node for data loading
> >> > > - Clients
> >> > >
> >> > > Also, at the end it ends mid-sentence under "Organization within
> >> Metron,"
> >> > > was that intended to be open ended?
> >> > >
> >> > > Jon
> >> > >
> >> > > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <me...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > I created a Jira to track this new feature at
> >> > > > https://issues.apache.org/jira/browse/METRON-503. I also started
> >> and
> >> > > > attached an architecture doc to that Jira with some of my ideas
> >> about
> >> > how
> >> > > > we should implement it. Please feel free to review and comment or
> >> add
> >> > to
> >> > > > it. Looking forward to everyone's ideas and feedback.
> >> > > >
> >> > > > Ryan Merriman
> >> > > >
> >> > > --
> >> > >
> >> > > Jon
> >> > >
> >> >
> >> > --
> >> >
> >> > Jon
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
--
Jon
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
So it looks like, for now, you are not pursuing Knox (per comments in
METRON-503 and then PR 316). Is there a reason for that?
Jon
On Fri, Oct 14, 2016 at 5:59 PM Zeolla@GMail.com <ze...@gmail.com> wrote:
> Good question :)
>
> On Fri, Oct 14, 2016, 17:07 Ryan Merriman <me...@gmail.com> wrote:
>
> Jon,
>
> It wasn't intentional, I ran out of time and wanted to get something out
> there. I think it certainly could be open ended though. Where should the
> REST API project be located?
>
> Ryan
>
> On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Along the lines of:
> > • Must be deployed to a machine with adequate resources so that resource
> > contention is avoided.
> > • Will need network access to all other services within Metron
> >
> > Has there been any consideration of a "Metron Manager" node? In the old
> > TP2
> > bare metal install guide
> > <https://cwiki.apache.org/confluence/display/METRON/
> > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > it mentions a "Metron Installer," but I could see the needs for that sort
> > of a system expanding to have the following roles:
> > - API
> > - Metron UI
> > - Metron Installer/upgrades
> > - Edge/Gateway Node for data loading
> > - Clients
> >
> > Also, at the end it ends mid-sentence under "Organization within Metron,"
> > was that intended to be open ended?
> >
> > Jon
> >
> > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <me...@gmail.com>
> wrote:
> >
> > > I created a Jira to track this new feature at
> > > https://issues.apache.org/jira/browse/METRON-503. I also started and
> > > attached an architecture doc to that Jira with some of my ideas about
> how
> > > we should implement it. Please feel free to review and comment or add
> to
> > > it. Looking forward to everyone's ideas and feedback.
> > >
> > > Ryan Merriman
> > >
> > --
> >
> > Jon
> >
>
> --
>
> Jon
>
--
Jon
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Good question :)
On Fri, Oct 14, 2016, 17:07 Ryan Merriman <me...@gmail.com> wrote:
> Jon,
>
> It wasn't intentional, I ran out of time and wanted to get something out
> there. I think it certainly could be open ended though. Where should the
> REST API project be located?
>
> Ryan
>
> On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
>
> > Along the lines of:
> > • Must be deployed to a machine with adequate resources so that resource
> > contention is avoided.
> > • Will need network access to all other services within Metron
> >
> > Has there been any consideration of a "Metron Manager" node? In the old
> > TP2
> > bare metal install guide
> > <https://cwiki.apache.org/confluence/display/METRON/
> > Metron+Installation+on+an+Ambari-Managed+Cluster>
> > it mentions a "Metron Installer," but I could see the needs for that sort
> > of a system expanding to have the following roles:
> > - API
> > - Metron UI
> > - Metron Installer/upgrades
> > - Edge/Gateway Node for data loading
> > - Clients
> >
> > Also, at the end it ends mid-sentence under "Organization within Metron,"
> > was that intended to be open ended?
> >
> > Jon
> >
> > On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <me...@gmail.com>
> wrote:
> >
> > > I created a Jira to track this new feature at
> > > https://issues.apache.org/jira/browse/METRON-503. I also started and
> > > attached an architecture doc to that Jira with some of my ideas about
> how
> > > we should implement it. Please feel free to review and comment or add
> to
> > > it. Looking forward to everyone's ideas and feedback.
> > >
> > > Ryan Merriman
> > >
> > --
> >
> > Jon
> >
>
--
Jon
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by Ryan Merriman <me...@gmail.com>.
Jon,
It wasn't intentional, I ran out of time and wanted to get something out
there. I think it certainly could be open ended though. Where should the
REST API project be located?
Ryan
On Thu, Oct 13, 2016 at 7:32 PM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Along the lines of:
> • Must be deployed to a machine with adequate resources so that resource
> contention is avoided.
> • Will need network access to all other services within Metron
>
> Has there been any consideration of a "Metron Manager" node? In the old
> TP2
> bare metal install guide
> <https://cwiki.apache.org/confluence/display/METRON/
> Metron+Installation+on+an+Ambari-Managed+Cluster>
> it mentions a "Metron Installer," but I could see the needs for that sort
> of a system expanding to have the following roles:
> - API
> - Metron UI
> - Metron Installer/upgrades
> - Edge/Gateway Node for data loading
> - Clients
>
> Also, at the end it ends mid-sentence under "Organization within Metron,"
> was that intended to be open ended?
>
> Jon
>
> On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <me...@gmail.com> wrote:
>
> > I created a Jira to track this new feature at
> > https://issues.apache.org/jira/browse/METRON-503. I also started and
> > attached an architecture doc to that Jira with some of my ideas about how
> > we should implement it. Please feel free to review and comment or add to
> > it. Looking forward to everyone's ideas and feedback.
> >
> > Ryan Merriman
> >
> --
>
> Jon
>
Re: [DISCUSS] Metron REST API Architecture and Design
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Along the lines of:
• Must be deployed to a machine with adequate resources so that resource
contention is avoided.
• Will need network access to all other services within Metron
Has there been any consideration of a "Metron Manager" node? In the old TP2
bare metal install guide
<https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster>
it mentions a "Metron Installer," but I could see the needs for that sort
of a system expanding to have the following roles:
- API
- Metron UI
- Metron Installer/upgrades
- Edge/Gateway Node for data loading
- Clients
Also, at the end it ends mid-sentence under "Organization within Metron,"
was that intended to be open ended?
Jon
On Thu, Oct 13, 2016 at 6:10 PM Ryan Merriman <me...@gmail.com> wrote:
> I created a Jira to track this new feature at
> https://issues.apache.org/jira/browse/METRON-503. I also started and
> attached an architecture doc to that Jira with some of my ideas about how
> we should implement it. Please feel free to review and comment or add to
> it. Looking forward to everyone's ideas and feedback.
>
> Ryan Merriman
>
--
Jon