You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Ryan Merriman <me...@gmail.com> on 2016/09/30 21:00:35 UTC
[DISCUSS] Dockerize Metron
I would like to open up a discussion around creating Docker images for
Metron. Having this available would provide a leaner alternative to the
ansible/vagrant environment for development tesing (and even demoing or
exploring features). It could also relieve some of the dependency version
conflict issues that we've been experiencing when running integration tests
in a single JVM.
I would suggest the initial version be intended only for development and
testing purposes. The general approach could be to create an image for
each service we depend on and use something like Docker compose to package
them together. A Dockerfile would either install the service from scratch
or extend a community image then add any Metron related dependencies or
configurations on top. The metron-deployment project code could be used as
a guide.
I would like to see these images added initially to support development and
testing:
- Kafka with topics preconfigured
- Storm with metron topology assets installed
- Zookeeper with paths created and sample configs loaded
- HBase with sample enrichments and threat intel loaded
- Elasticsearch configured for Metron
- MySQL with databases/tables/users created and geo data loaded
Other images that could also be useful:
- Images for each sensor
- Ambari?
- Solr
Looking forward to hearing what everyone thinks.
Ryan Merriman
Re: [DISCUSS] Dockerize Metron
Posted by "Tseytlin, Keren" <Ke...@capitalone.com>.
Really liking this conversation on Docker. One additional feature for Dockerized Metron that would be awesome is if there are some sort of proxy configs/capability set up for those of us who work in corporate settings.
Whenever I use Docker at work I run into a nasty bug where if I switch networks then I basically have to start from the base of the container again. Would be fabulous if this container didn’t require me to switch networks.
Best,
Keren
On 10/1/16, 8:24 PM, "Michael Miklavcic" <mi...@gmail.com> wrote:
I really like this
On Oct 1, 2016 5:03 PM, "Ryan Merriman" <me...@gmail.com> wrote:
> I don't think this will be an issue. The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now). The Storm topology
> continues to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers. We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/
> metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/
> integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger. I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <ce...@gmail.com> wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklavcic@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org>
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers. I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >> I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >> Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >> ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > > or
> > > > > >> exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >> conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >> in a single JVM.
> > > > > >>
> > > > > >> I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >> testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >> each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >> them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >> or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >> configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >> a guide.
> > > > > >>
> > > > > >> I would like to see these images added initially to support
> > > > > development and
> > > > > >> testing:
> > > > > >>
> > > > > >> - Kafka with topics preconfigured
> > > > > >> - Storm with metron topology assets installed
> > > > > >> - Zookeeper with paths created and sample configs loaded
> > > > > >> - HBase with sample enrichments and threat intel loaded
> > > > > >> - Elasticsearch configured for Metron
> > > > > >> - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >> Other images that could also be useful:
> > > > > >>
> > > > > >> - Images for each sensor
> > > > > >> - Ambari?
> > > > > >> - Solr
> > > > > >>
> > > > > >> Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >> Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <ni...@nickallen.org>
> > > > >
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> > > take
> > > > a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >> Metron. Having this available would provide a leaner
> alternative
> > to
> > > > the
> > > > > >> ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > or
> > > > > >> exploring features). It could also relieve some of the
> dependency
> > > > > version
> > > > > >> conflict issues that we've been experiencing when running
> > > integration
> > > > > tests
> > > > > >> in a single JVM.
> > > > > >>
> > > > > >> I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >> testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >> each service we depend on and use something like Docker compose
> > to
> > > > > package
> > > > > >> them together. A Dockerfile would either install the service
> from
> > > > > scratch
> > > > > >> or extend a community image then add any Metron related
> > > dependencies
> > > > or
> > > > > >> configurations on top. The metron-deployment project code could
> > be
> > > > > used as
> > > > > >> a guide.
> > > > > >>
> > > > > >> I would like to see these images added initially to support
> > > > > development and
> > > > > >> testing:
> > > > > >>
> > > > > >> - Kafka with topics preconfigured
> > > > > >> - Storm with metron topology assets installed
> > > > > >> - Zookeeper with paths created and sample configs loaded
> > > > > >> - HBase with sample enrichments and threat intel loaded
> > > > > >> - Elasticsearch configured for Metron
> > > > > >> - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >> Other images that could also be useful:
> > > > > >>
> > > > > >> - Images for each sensor
> > > > > >> - Ambari?
> > > > > >> - Solr
> > > > > >>
> > > > > >> Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >> Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <ni...@nickallen.org>
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <ni...@nickallen.org>
> >
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: [DISCUSS] Dockerize Metron
Posted by Michael Miklavcic <mi...@gmail.com>.
I really like this
On Oct 1, 2016 5:03 PM, "Ryan Merriman" <me...@gmail.com> wrote:
> I don't think this will be an issue. The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now). The Storm topology
> continues to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers. We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/
> metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/
> integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger. I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <ce...@gmail.com> wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklavcic@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org>
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers. I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >> I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >> Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >> ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > > or
> > > > > >> exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >> conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >> in a single JVM.
> > > > > >>
> > > > > >> I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >> testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >> each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >> them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >> or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >> configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >> a guide.
> > > > > >>
> > > > > >> I would like to see these images added initially to support
> > > > > development and
> > > > > >> testing:
> > > > > >>
> > > > > >> - Kafka with topics preconfigured
> > > > > >> - Storm with metron topology assets installed
> > > > > >> - Zookeeper with paths created and sample configs loaded
> > > > > >> - HBase with sample enrichments and threat intel loaded
> > > > > >> - Elasticsearch configured for Metron
> > > > > >> - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >> Other images that could also be useful:
> > > > > >>
> > > > > >> - Images for each sensor
> > > > > >> - Ambari?
> > > > > >> - Solr
> > > > > >>
> > > > > >> Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >> Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <ni...@nickallen.org>
> > > > >
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> > > take
> > > > a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >> Metron. Having this available would provide a leaner
> alternative
> > to
> > > > the
> > > > > >> ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > or
> > > > > >> exploring features). It could also relieve some of the
> dependency
> > > > > version
> > > > > >> conflict issues that we've been experiencing when running
> > > integration
> > > > > tests
> > > > > >> in a single JVM.
> > > > > >>
> > > > > >> I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >> testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >> each service we depend on and use something like Docker compose
> > to
> > > > > package
> > > > > >> them together. A Dockerfile would either install the service
> from
> > > > > scratch
> > > > > >> or extend a community image then add any Metron related
> > > dependencies
> > > > or
> > > > > >> configurations on top. The metron-deployment project code could
> > be
> > > > > used as
> > > > > >> a guide.
> > > > > >>
> > > > > >> I would like to see these images added initially to support
> > > > > development and
> > > > > >> testing:
> > > > > >>
> > > > > >> - Kafka with topics preconfigured
> > > > > >> - Storm with metron topology assets installed
> > > > > >> - Zookeeper with paths created and sample configs loaded
> > > > > >> - HBase with sample enrichments and threat intel loaded
> > > > > >> - Elasticsearch configured for Metron
> > > > > >> - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >> Other images that could also be useful:
> > > > > >>
> > > > > >> - Images for each sensor
> > > > > >> - Ambari?
> > > > > >> - Solr
> > > > > >>
> > > > > >> Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >> Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <ni...@nickallen.org>
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <ni...@nickallen.org>
> >
>
Re: [DISCUSS] Dockerize Metron
Posted by Casey Stella <ce...@gmail.com>.
Oh yeah I like that strategy. It lets us isolate some of our more
"challenging" dependencies in the tests (e.g. Hbase and storm) and still
debug in the ide.
On Sat, Oct 1, 2016 at 17:03 Ryan Merriman <me...@gmail.com> wrote:
> I don't think this will be an issue. The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now). The Storm topology
> continues to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers. We can continue to leverage
>
> https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger. I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <ce...@gmail.com> wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklavcic@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org>
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers. I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >> I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >> Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >> ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > > or
> > > > > >> exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >> conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >> in a single JVM.
> > > > > >>
> > > > > >> I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >> testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >> each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >> them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >> or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >> configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >> a guide.
> > > > > >>
> > > > > >> I would like to see these images added initially to support
> > > > > development and
> > > > > >> testing:
> > > > > >>
> > > > > >> - Kafka with topics preconfigured
> > > > > >> - Storm with metron topology assets installed
> > > > > >> - Zookeeper with paths created and sample configs loaded
> > > > > >> - HBase with sample enrichments and threat intel loaded
> > > > > >> - Elasticsearch configured for Metron
> > > > > >> - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >> Other images that could also be useful:
> > > > > >>
> > > > > >> - Images for each sensor
> > > > > >> - Ambari?
> > > > > >> - Solr
> > > > > >>
> > > > > >> Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >> Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <ni...@nickallen.org>
> > > > >
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> > > take
> > > > a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >> Metron. Having this available would provide a leaner
> alternative
> > to
> > > > the
> > > > > >> ansible/vagrant environment for development tesing (and even
> > > demoing
> > > > or
> > > > > >> exploring features). It could also relieve some of the
> dependency
> > > > > version
> > > > > >> conflict issues that we've been experiencing when running
> > > integration
> > > > > tests
> > > > > >> in a single JVM.
> > > > > >>
> > > > > >> I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >> testing purposes. The general approach could be to create an
> > image
> > > > for
> > > > > >> each service we depend on and use something like Docker compose
> > to
> > > > > package
> > > > > >> them together. A Dockerfile would either install the service
> from
> > > > > scratch
> > > > > >> or extend a community image then add any Metron related
> > > dependencies
> > > > or
> > > > > >> configurations on top. The metron-deployment project code could
> > be
> > > > > used as
> > > > > >> a guide.
> > > > > >>
> > > > > >> I would like to see these images added initially to support
> > > > > development and
> > > > > >> testing:
> > > > > >>
> > > > > >> - Kafka with topics preconfigured
> > > > > >> - Storm with metron topology assets installed
> > > > > >> - Zookeeper with paths created and sample configs loaded
> > > > > >> - HBase with sample enrichments and threat intel loaded
> > > > > >> - Elasticsearch configured for Metron
> > > > > >> - MySQL with databases/tables/users created and geo data
> > loaded
> > > > > >>
> > > > > >> Other images that could also be useful:
> > > > > >>
> > > > > >> - Images for each sensor
> > > > > >> - Ambari?
> > > > > >> - Solr
> > > > > >>
> > > > > >> Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >> Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <ni...@nickallen.org>
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <ni...@nickallen.org>
> >
>
Re: [DISCUSS] Dockerize Metron
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
At least on the Storm side, remote debugging can be problematic. If you pause execution of a worker process, it will stop heart beating and supervisor will kill it. Just something to be aware of.
I think we might have a flag to disable that behavior, but I'd have to check.
I usually use local mode for debugging.
-Taylor
> On Oct 1, 2016, at 5:03 PM, Ryan Merriman <me...@gmail.com> wrote:
>
> I don't think this will be an issue. The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now). The Storm topology
> continues to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers. We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
>> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <ni...@nickallen.org> wrote:
>>
>> It's definitely possible to debug in a Docker environment using a remote
>> debugger. I remember reading something along these lines a few weeks ago.
>>
>> https://blog.docker.com/2016/09/java-development-using-docker/
>>
>>> On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <ce...@gmail.com> wrote:
>>>
>>> I agree with that concern. Being able to debug a running topology has
>>> helped in so many circumstances. Not sure how to accomplish this in a
>>> dockerized environment.
>>>
>>> On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
>>> michael.miklavcic@gmail.com>
>>> wrote:
>>>
>>>> Any ideas on how debugging works when leveraging Docker? In spite of
>> the
>>>> classpath troubles, one of the benefits of the current single-JVM
>>> approach
>>>> is that you can easily debug, set break points, etc within an IDE. Is
>>> that
>>>> still doable? Seems like there might need to be some remote debugging
>>> magic
>>>> done to accomplish this.
>>>>
>>>>> On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org>
>>>> wrote:
>>>>
>>>>> I agree with making an effort to create containers. I suggest doing
>> it
>>>> on
>>>>> a feature branch until we are feature complete and are able to
>> migrate
>>>> our
>>>>> integration tests into a dockerized environment.
>>>>>
>>>>> 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
>>>>>> Relieve dependency version conflict issues? Umm, yes please. I'll
>>>> take a
>>>>>> second helping of that too.
>>>>>>
>>>>>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
>> merrimanr@gmail.com
>>>>
>>>>> wrote:
>>>>>>
>>>>>>> I would like to open up a discussion around creating Docker
>> images
>>>> for
>>>>>>> Metron. Having this available would provide a leaner alternative
>>> to
>>>>> the
>>>>>>> ansible/vagrant environment for development tesing (and even
>>> demoing
>>>>> or
>>>>>>> exploring features). It could also relieve some of the
>> dependency
>>>>> version
>>>>>>> conflict issues that we've been experiencing when running
>>>> integration
>>>>> tests
>>>>>>> in a single JVM.
>>>>>>>
>>>>>>> I would suggest the initial version be intended only for
>>> development
>>>>> and
>>>>>>> testing purposes. The general approach could be to create an
>> image
>>>> for
>>>>>>> each service we depend on and use something like Docker compose
>> to
>>>>> package
>>>>>>> them together. A Dockerfile would either install the service
>> from
>>>>> scratch
>>>>>>> or extend a community image then add any Metron related
>>> dependencies
>>>>> or
>>>>>>> configurations on top. The metron-deployment project code could
>> be
>>>>> used as
>>>>>>> a guide.
>>>>>>>
>>>>>>> I would like to see these images added initially to support
>>>>> development and
>>>>>>> testing:
>>>>>>>
>>>>>>> - Kafka with topics preconfigured
>>>>>>> - Storm with metron topology assets installed
>>>>>>> - Zookeeper with paths created and sample configs loaded
>>>>>>> - HBase with sample enrichments and threat intel loaded
>>>>>>> - Elasticsearch configured for Metron
>>>>>>> - MySQL with databases/tables/users created and geo data
>> loaded
>>>>>>>
>>>>>>> Other images that could also be useful:
>>>>>>>
>>>>>>> - Images for each sensor
>>>>>>> - Ambari?
>>>>>>> - Solr
>>>>>>>
>>>>>>> Looking forward to hearing what everyone thinks.
>>>>>>>
>>>>>>> Ryan Merriman
>>>>>>
>>>>>> --
>>>>>> Nick Allen <ni...@nickallen.org>
>>>>>
>>>>>
>>>>> -------------------
>>>>> Thank you,
>>>>>
>>>>> James Sirota
>>>>> PPMC- Apache Metron (Incubating)
>>>>> jsirota AT apache DOT org
>>>>>
>>>>> 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
>>>>>> Relieve dependency version conflict issues? Umm, yes please. I'll
>>> take
>>>> a
>>>>>> second helping of that too.
>>>>>>
>>>>>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
>> merrimanr@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>>> I would like to open up a discussion around creating Docker
>> images
>>>> for
>>>>>>> Metron. Having this available would provide a leaner alternative
>> to
>>>> the
>>>>>>> ansible/vagrant environment for development tesing (and even
>>> demoing
>>>> or
>>>>>>> exploring features). It could also relieve some of the dependency
>>>>> version
>>>>>>> conflict issues that we've been experiencing when running
>>> integration
>>>>> tests
>>>>>>> in a single JVM.
>>>>>>>
>>>>>>> I would suggest the initial version be intended only for
>>> development
>>>>> and
>>>>>>> testing purposes. The general approach could be to create an
>> image
>>>> for
>>>>>>> each service we depend on and use something like Docker compose
>> to
>>>>> package
>>>>>>> them together. A Dockerfile would either install the service from
>>>>> scratch
>>>>>>> or extend a community image then add any Metron related
>>> dependencies
>>>> or
>>>>>>> configurations on top. The metron-deployment project code could
>> be
>>>>> used as
>>>>>>> a guide.
>>>>>>>
>>>>>>> I would like to see these images added initially to support
>>>>> development and
>>>>>>> testing:
>>>>>>>
>>>>>>> - Kafka with topics preconfigured
>>>>>>> - Storm with metron topology assets installed
>>>>>>> - Zookeeper with paths created and sample configs loaded
>>>>>>> - HBase with sample enrichments and threat intel loaded
>>>>>>> - Elasticsearch configured for Metron
>>>>>>> - MySQL with databases/tables/users created and geo data
>> loaded
>>>>>>>
>>>>>>> Other images that could also be useful:
>>>>>>>
>>>>>>> - Images for each sensor
>>>>>>> - Ambari?
>>>>>>> - Solr
>>>>>>>
>>>>>>> Looking forward to hearing what everyone thinks.
>>>>>>>
>>>>>>> Ryan Merriman
>>>>>>
>>>>>> --
>>>>>> Nick Allen <ni...@nickallen.org>
>>>>>
>>>>> -------------------
>>>>> Thank you,
>>>>>
>>>>> James Sirota
>>>>> PPMC- Apache Metron (Incubating)
>>>>> jsirota AT apache DOT org
>>
>>
>>
>> --
>> Nick Allen <ni...@nickallen.org>
>>
Re: [DISCUSS] Dockerize Metron
Posted by Ryan Merriman <me...@gmail.com>.
I don't think this will be an issue. The remote debugging strategy should
work if you want to go that route but I would just run the topology in
local mode (that's how the integration tests work now). The Storm topology
continues to runs locally in your IDE while all the other components that
were in-memory components (HBase, Kafka, etc) now run in Docker
containers. We can continue to leverage
https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
for parser topologies and the --local option for flux-based topologies.
On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <ni...@nickallen.org> wrote:
> It's definitely possible to debug in a Docker environment using a remote
> debugger. I remember reading something along these lines a few weeks ago.
>
> https://blog.docker.com/2016/09/java-development-using-docker/
>
> On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <ce...@gmail.com> wrote:
>
> > I agree with that concern. Being able to debug a running topology has
> > helped in so many circumstances. Not sure how to accomplish this in a
> > dockerized environment.
> >
> > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > michael.miklavcic@gmail.com>
> > wrote:
> >
> > > Any ideas on how debugging works when leveraging Docker? In spite of
> the
> > > classpath troubles, one of the benefits of the current single-JVM
> > approach
> > > is that you can easily debug, set break points, etc within an IDE. Is
> > that
> > > still doable? Seems like there might need to be some remote debugging
> > magic
> > > done to accomplish this.
> > >
> > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org>
> > wrote:
> > >
> > > > I agree with making an effort to create containers. I suggest doing
> it
> > > on
> > > > a feature branch until we are feature complete and are able to
> migrate
> > > our
> > > > integration tests into a dockerized environment.
> > > >
> > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> > > take a
> > > > > second helping of that too.
> > > > >
> > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> merrimanr@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > >> I would like to open up a discussion around creating Docker
> images
> > > for
> > > > >> Metron. Having this available would provide a leaner alternative
> > to
> > > > the
> > > > >> ansible/vagrant environment for development tesing (and even
> > demoing
> > > > or
> > > > >> exploring features). It could also relieve some of the
> dependency
> > > > version
> > > > >> conflict issues that we've been experiencing when running
> > > integration
> > > > tests
> > > > >> in a single JVM.
> > > > >>
> > > > >> I would suggest the initial version be intended only for
> > development
> > > > and
> > > > >> testing purposes. The general approach could be to create an
> image
> > > for
> > > > >> each service we depend on and use something like Docker compose
> to
> > > > package
> > > > >> them together. A Dockerfile would either install the service
> from
> > > > scratch
> > > > >> or extend a community image then add any Metron related
> > dependencies
> > > > or
> > > > >> configurations on top. The metron-deployment project code could
> be
> > > > used as
> > > > >> a guide.
> > > > >>
> > > > >> I would like to see these images added initially to support
> > > > development and
> > > > >> testing:
> > > > >>
> > > > >> - Kafka with topics preconfigured
> > > > >> - Storm with metron topology assets installed
> > > > >> - Zookeeper with paths created and sample configs loaded
> > > > >> - HBase with sample enrichments and threat intel loaded
> > > > >> - Elasticsearch configured for Metron
> > > > >> - MySQL with databases/tables/users created and geo data
> loaded
> > > > >>
> > > > >> Other images that could also be useful:
> > > > >>
> > > > >> - Images for each sensor
> > > > >> - Ambari?
> > > > >> - Solr
> > > > >>
> > > > >> Looking forward to hearing what everyone thinks.
> > > > >>
> > > > >> Ryan Merriman
> > > > >
> > > > > --
> > > > > Nick Allen <ni...@nickallen.org>
> > > >
> > > >
> > > > -------------------
> > > > Thank you,
> > > >
> > > > James Sirota
> > > > PPMC- Apache Metron (Incubating)
> > > > jsirota AT apache DOT org
> > > >
> > > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> > take
> > > a
> > > > > second helping of that too.
> > > > >
> > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> merrimanr@gmail.com>
> > > > wrote:
> > > > >
> > > > >> I would like to open up a discussion around creating Docker
> images
> > > for
> > > > >> Metron. Having this available would provide a leaner alternative
> to
> > > the
> > > > >> ansible/vagrant environment for development tesing (and even
> > demoing
> > > or
> > > > >> exploring features). It could also relieve some of the dependency
> > > > version
> > > > >> conflict issues that we've been experiencing when running
> > integration
> > > > tests
> > > > >> in a single JVM.
> > > > >>
> > > > >> I would suggest the initial version be intended only for
> > development
> > > > and
> > > > >> testing purposes. The general approach could be to create an
> image
> > > for
> > > > >> each service we depend on and use something like Docker compose
> to
> > > > package
> > > > >> them together. A Dockerfile would either install the service from
> > > > scratch
> > > > >> or extend a community image then add any Metron related
> > dependencies
> > > or
> > > > >> configurations on top. The metron-deployment project code could
> be
> > > > used as
> > > > >> a guide.
> > > > >>
> > > > >> I would like to see these images added initially to support
> > > > development and
> > > > >> testing:
> > > > >>
> > > > >> - Kafka with topics preconfigured
> > > > >> - Storm with metron topology assets installed
> > > > >> - Zookeeper with paths created and sample configs loaded
> > > > >> - HBase with sample enrichments and threat intel loaded
> > > > >> - Elasticsearch configured for Metron
> > > > >> - MySQL with databases/tables/users created and geo data
> loaded
> > > > >>
> > > > >> Other images that could also be useful:
> > > > >>
> > > > >> - Images for each sensor
> > > > >> - Ambari?
> > > > >> - Solr
> > > > >>
> > > > >> Looking forward to hearing what everyone thinks.
> > > > >>
> > > > >> Ryan Merriman
> > > > >
> > > > > --
> > > > > Nick Allen <ni...@nickallen.org>
> > > >
> > > > -------------------
> > > > Thank you,
> > > >
> > > > James Sirota
> > > > PPMC- Apache Metron (Incubating)
> > > > jsirota AT apache DOT org
> > > >
> > >
> >
>
>
>
> --
> Nick Allen <ni...@nickallen.org>
>
Re: [DISCUSS] Dockerize Metron
Posted by Nick Allen <ni...@nickallen.org>.
It's definitely possible to debug in a Docker environment using a remote
debugger. I remember reading something along these lines a few weeks ago.
https://blog.docker.com/2016/09/java-development-using-docker/
On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <ce...@gmail.com> wrote:
> I agree with that concern. Being able to debug a running topology has
> helped in so many circumstances. Not sure how to accomplish this in a
> dockerized environment.
>
> On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> michael.miklavcic@gmail.com>
> wrote:
>
> > Any ideas on how debugging works when leveraging Docker? In spite of the
> > classpath troubles, one of the benefits of the current single-JVM
> approach
> > is that you can easily debug, set break points, etc within an IDE. Is
> that
> > still doable? Seems like there might need to be some remote debugging
> magic
> > done to accomplish this.
> >
> > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org>
> wrote:
> >
> > > I agree with making an effort to create containers. I suggest doing it
> > on
> > > a feature branch until we are feature complete and are able to migrate
> > our
> > > integration tests into a dockerized environment.
> > >
> > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> > take a
> > > > second helping of that too.
> > > >
> > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <merrimanr@gmail.com
> >
> > > wrote:
> > > >
> > > >> I would like to open up a discussion around creating Docker images
> > for
> > > >> Metron. Having this available would provide a leaner alternative
> to
> > > the
> > > >> ansible/vagrant environment for development tesing (and even
> demoing
> > > or
> > > >> exploring features). It could also relieve some of the dependency
> > > version
> > > >> conflict issues that we've been experiencing when running
> > integration
> > > tests
> > > >> in a single JVM.
> > > >>
> > > >> I would suggest the initial version be intended only for
> development
> > > and
> > > >> testing purposes. The general approach could be to create an image
> > for
> > > >> each service we depend on and use something like Docker compose to
> > > package
> > > >> them together. A Dockerfile would either install the service from
> > > scratch
> > > >> or extend a community image then add any Metron related
> dependencies
> > > or
> > > >> configurations on top. The metron-deployment project code could be
> > > used as
> > > >> a guide.
> > > >>
> > > >> I would like to see these images added initially to support
> > > development and
> > > >> testing:
> > > >>
> > > >> - Kafka with topics preconfigured
> > > >> - Storm with metron topology assets installed
> > > >> - Zookeeper with paths created and sample configs loaded
> > > >> - HBase with sample enrichments and threat intel loaded
> > > >> - Elasticsearch configured for Metron
> > > >> - MySQL with databases/tables/users created and geo data loaded
> > > >>
> > > >> Other images that could also be useful:
> > > >>
> > > >> - Images for each sensor
> > > >> - Ambari?
> > > >> - Solr
> > > >>
> > > >> Looking forward to hearing what everyone thinks.
> > > >>
> > > >> Ryan Merriman
> > > >
> > > > --
> > > > Nick Allen <ni...@nickallen.org>
> > >
> > >
> > > -------------------
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> > > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > > Relieve dependency version conflict issues? Umm, yes please. I'll
> take
> > a
> > > > second helping of that too.
> > > >
> > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com>
> > > wrote:
> > > >
> > > >> I would like to open up a discussion around creating Docker images
> > for
> > > >> Metron. Having this available would provide a leaner alternative to
> > the
> > > >> ansible/vagrant environment for development tesing (and even
> demoing
> > or
> > > >> exploring features). It could also relieve some of the dependency
> > > version
> > > >> conflict issues that we've been experiencing when running
> integration
> > > tests
> > > >> in a single JVM.
> > > >>
> > > >> I would suggest the initial version be intended only for
> development
> > > and
> > > >> testing purposes. The general approach could be to create an image
> > for
> > > >> each service we depend on and use something like Docker compose to
> > > package
> > > >> them together. A Dockerfile would either install the service from
> > > scratch
> > > >> or extend a community image then add any Metron related
> dependencies
> > or
> > > >> configurations on top. The metron-deployment project code could be
> > > used as
> > > >> a guide.
> > > >>
> > > >> I would like to see these images added initially to support
> > > development and
> > > >> testing:
> > > >>
> > > >> - Kafka with topics preconfigured
> > > >> - Storm with metron topology assets installed
> > > >> - Zookeeper with paths created and sample configs loaded
> > > >> - HBase with sample enrichments and threat intel loaded
> > > >> - Elasticsearch configured for Metron
> > > >> - MySQL with databases/tables/users created and geo data loaded
> > > >>
> > > >> Other images that could also be useful:
> > > >>
> > > >> - Images for each sensor
> > > >> - Ambari?
> > > >> - Solr
> > > >>
> > > >> Looking forward to hearing what everyone thinks.
> > > >>
> > > >> Ryan Merriman
> > > >
> > > > --
> > > > Nick Allen <ni...@nickallen.org>
> > >
> > > -------------------
> > > Thank you,
> > >
> > > James Sirota
> > > PPMC- Apache Metron (Incubating)
> > > jsirota AT apache DOT org
> > >
> >
>
--
Nick Allen <ni...@nickallen.org>
Re: [DISCUSS] Dockerize Metron
Posted by Casey Stella <ce...@gmail.com>.
I agree with that concern. Being able to debug a running topology has
helped in so many circumstances. Not sure how to accomplish this in a
dockerized environment.
On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <mi...@gmail.com>
wrote:
> Any ideas on how debugging works when leveraging Docker? In spite of the
> classpath troubles, one of the benefits of the current single-JVM approach
> is that you can easily debug, set break points, etc within an IDE. Is that
> still doable? Seems like there might need to be some remote debugging magic
> done to accomplish this.
>
> On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org> wrote:
>
> > I agree with making an effort to create containers. I suggest doing it
> on
> > a feature branch until we are feature complete and are able to migrate
> our
> > integration tests into a dockerized environment.
> >
> > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > Relieve dependency version conflict issues? Umm, yes please. I'll
> take a
> > > second helping of that too.
> > >
> > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com>
> > wrote:
> > >
> > >> I would like to open up a discussion around creating Docker images
> for
> > >> Metron. Having this available would provide a leaner alternative to
> > the
> > >> ansible/vagrant environment for development tesing (and even demoing
> > or
> > >> exploring features). It could also relieve some of the dependency
> > version
> > >> conflict issues that we've been experiencing when running
> integration
> > tests
> > >> in a single JVM.
> > >>
> > >> I would suggest the initial version be intended only for development
> > and
> > >> testing purposes. The general approach could be to create an image
> for
> > >> each service we depend on and use something like Docker compose to
> > package
> > >> them together. A Dockerfile would either install the service from
> > scratch
> > >> or extend a community image then add any Metron related dependencies
> > or
> > >> configurations on top. The metron-deployment project code could be
> > used as
> > >> a guide.
> > >>
> > >> I would like to see these images added initially to support
> > development and
> > >> testing:
> > >>
> > >> - Kafka with topics preconfigured
> > >> - Storm with metron topology assets installed
> > >> - Zookeeper with paths created and sample configs loaded
> > >> - HBase with sample enrichments and threat intel loaded
> > >> - Elasticsearch configured for Metron
> > >> - MySQL with databases/tables/users created and geo data loaded
> > >>
> > >> Other images that could also be useful:
> > >>
> > >> - Images for each sensor
> > >> - Ambari?
> > >> - Solr
> > >>
> > >> Looking forward to hearing what everyone thinks.
> > >>
> > >> Ryan Merriman
> > >
> > > --
> > > Nick Allen <ni...@nickallen.org>
> >
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
> > 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > > Relieve dependency version conflict issues? Umm, yes please. I'll take
> a
> > > second helping of that too.
> > >
> > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com>
> > wrote:
> > >
> > >> I would like to open up a discussion around creating Docker images
> for
> > >> Metron. Having this available would provide a leaner alternative to
> the
> > >> ansible/vagrant environment for development tesing (and even demoing
> or
> > >> exploring features). It could also relieve some of the dependency
> > version
> > >> conflict issues that we've been experiencing when running integration
> > tests
> > >> in a single JVM.
> > >>
> > >> I would suggest the initial version be intended only for development
> > and
> > >> testing purposes. The general approach could be to create an image
> for
> > >> each service we depend on and use something like Docker compose to
> > package
> > >> them together. A Dockerfile would either install the service from
> > scratch
> > >> or extend a community image then add any Metron related dependencies
> or
> > >> configurations on top. The metron-deployment project code could be
> > used as
> > >> a guide.
> > >>
> > >> I would like to see these images added initially to support
> > development and
> > >> testing:
> > >>
> > >> - Kafka with topics preconfigured
> > >> - Storm with metron topology assets installed
> > >> - Zookeeper with paths created and sample configs loaded
> > >> - HBase with sample enrichments and threat intel loaded
> > >> - Elasticsearch configured for Metron
> > >> - MySQL with databases/tables/users created and geo data loaded
> > >>
> > >> Other images that could also be useful:
> > >>
> > >> - Images for each sensor
> > >> - Ambari?
> > >> - Solr
> > >>
> > >> Looking forward to hearing what everyone thinks.
> > >>
> > >> Ryan Merriman
> > >
> > > --
> > > Nick Allen <ni...@nickallen.org>
> >
> > -------------------
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
> >
>
Re: [DISCUSS] Dockerize Metron
Posted by Michael Miklavcic <mi...@gmail.com>.
Any ideas on how debugging works when leveraging Docker? In spite of the
classpath troubles, one of the benefits of the current single-JVM approach
is that you can easily debug, set break points, etc within an IDE. Is that
still doable? Seems like there might need to be some remote debugging magic
done to accomplish this.
On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <js...@apache.org> wrote:
> I agree with making an effort to create containers. I suggest doing it on
> a feature branch until we are feature complete and are able to migrate our
> integration tests into a dockerized environment.
>
> 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > Relieve dependency version conflict issues? Umm, yes please. I'll take a
> > second helping of that too.
> >
> > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com>
> wrote:
> >
> >> I would like to open up a discussion around creating Docker images for
> >> Metron. Having this available would provide a leaner alternative to
> the
> >> ansible/vagrant environment for development tesing (and even demoing
> or
> >> exploring features). It could also relieve some of the dependency
> version
> >> conflict issues that we've been experiencing when running integration
> tests
> >> in a single JVM.
> >>
> >> I would suggest the initial version be intended only for development
> and
> >> testing purposes. The general approach could be to create an image for
> >> each service we depend on and use something like Docker compose to
> package
> >> them together. A Dockerfile would either install the service from
> scratch
> >> or extend a community image then add any Metron related dependencies
> or
> >> configurations on top. The metron-deployment project code could be
> used as
> >> a guide.
> >>
> >> I would like to see these images added initially to support
> development and
> >> testing:
> >>
> >> - Kafka with topics preconfigured
> >> - Storm with metron topology assets installed
> >> - Zookeeper with paths created and sample configs loaded
> >> - HBase with sample enrichments and threat intel loaded
> >> - Elasticsearch configured for Metron
> >> - MySQL with databases/tables/users created and geo data loaded
> >>
> >> Other images that could also be useful:
> >>
> >> - Images for each sensor
> >> - Ambari?
> >> - Solr
> >>
> >> Looking forward to hearing what everyone thinks.
> >>
> >> Ryan Merriman
> >
> > --
> > Nick Allen <ni...@nickallen.org>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
> 30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> > Relieve dependency version conflict issues? Umm, yes please. I'll take a
> > second helping of that too.
> >
> > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com>
> wrote:
> >
> >> I would like to open up a discussion around creating Docker images for
> >> Metron. Having this available would provide a leaner alternative to the
> >> ansible/vagrant environment for development tesing (and even demoing or
> >> exploring features). It could also relieve some of the dependency
> version
> >> conflict issues that we've been experiencing when running integration
> tests
> >> in a single JVM.
> >>
> >> I would suggest the initial version be intended only for development
> and
> >> testing purposes. The general approach could be to create an image for
> >> each service we depend on and use something like Docker compose to
> package
> >> them together. A Dockerfile would either install the service from
> scratch
> >> or extend a community image then add any Metron related dependencies or
> >> configurations on top. The metron-deployment project code could be
> used as
> >> a guide.
> >>
> >> I would like to see these images added initially to support
> development and
> >> testing:
> >>
> >> - Kafka with topics preconfigured
> >> - Storm with metron topology assets installed
> >> - Zookeeper with paths created and sample configs loaded
> >> - HBase with sample enrichments and threat intel loaded
> >> - Elasticsearch configured for Metron
> >> - MySQL with databases/tables/users created and geo data loaded
> >>
> >> Other images that could also be useful:
> >>
> >> - Images for each sensor
> >> - Ambari?
> >> - Solr
> >>
> >> Looking forward to hearing what everyone thinks.
> >>
> >> Ryan Merriman
> >
> > --
> > Nick Allen <ni...@nickallen.org>
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
Re: [DISCUSS] Dockerize Metron
Posted by James Sirota <js...@apache.org>.
I agree with making an effort to create containers. I suggest doing it on a feature branch until we are feature complete and are able to migrate our integration tests into a dockerized environment.
30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> �Relieve dependency version conflict issues? Umm, yes please. I'll take a
> �second helping of that too.
>
> �On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com> wrote:
>
>> ��I would like to open up a discussion around creating Docker images for
>> ��Metron. Having this available would provide a leaner alternative to the
>> ��ansible/vagrant environment for development tesing (and even demoing or
>> ��exploring features). It could also relieve some of the dependency version
>> ��conflict issues that we've been experiencing when running integration tests
>> ��in a single JVM.
>>
>> ��I would suggest the initial version be intended only for development and
>> ��testing purposes. The general approach could be to create an image for
>> ��each service we depend on and use something like Docker compose to package
>> ��them together. A Dockerfile would either install the service from scratch
>> ��or extend a community image then add any Metron related dependencies or
>> ��configurations on top. The metron-deployment project code could be used as
>> ��a guide.
>>
>> ��I would like to see these images added initially to support development and
>> ��testing:
>>
>> �����- Kafka with topics preconfigured
>> �����- Storm with metron topology assets installed
>> �����- Zookeeper with paths created and sample configs loaded
>> �����- HBase with sample enrichments and threat intel loaded
>> �����- Elasticsearch configured for Metron
>> �����- MySQL with databases/tables/users created and geo data loaded
>>
>> ��Other images that could also be useful:
>>
>> �����- Images for each sensor
>> �����- Ambari?
>> �����- Solr
>>
>> ��Looking forward to hearing what everyone thinks.
>>
>> ��Ryan Merriman
>
> �--
> �Nick Allen <ni...@nickallen.org>
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
30.09.2016, 14:52, "Nick Allen" <ni...@nickallen.org>:
> Relieve dependency version conflict issues? Umm, yes please. I'll take a
> second helping of that too.
>
> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com> wrote:
>
>> �I would like to open up a discussion around creating Docker images for
>> �Metron. Having this available would provide a leaner alternative to the
>> �ansible/vagrant environment for development tesing (and even demoing or
>> �exploring features). It could also relieve some of the dependency version
>> �conflict issues that we've been experiencing when running integration tests
>> �in a single JVM.
>>
>> �I would suggest the initial version be intended only for development and
>> �testing purposes. The general approach could be to create an image for
>> �each service we depend on and use something like Docker compose to package
>> �them together. A Dockerfile would either install the service from scratch
>> �or extend a community image then add any Metron related dependencies or
>> �configurations on top. The metron-deployment project code could be used as
>> �a guide.
>>
>> �I would like to see these images added initially to support development and
>> �testing:
>>
>> ����- Kafka with topics preconfigured
>> ����- Storm with metron topology assets installed
>> ����- Zookeeper with paths created and sample configs loaded
>> ����- HBase with sample enrichments and threat intel loaded
>> ����- Elasticsearch configured for Metron
>> ����- MySQL with databases/tables/users created and geo data loaded
>>
>> �Other images that could also be useful:
>>
>> ����- Images for each sensor
>> ����- Ambari?
>> ����- Solr
>>
>> �Looking forward to hearing what everyone thinks.
>>
>> �Ryan Merriman
>
> --
> Nick Allen <ni...@nickallen.org>
-------------------�
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Dockerize Metron
Posted by Nick Allen <ni...@nickallen.org>.
Relieve dependency version conflict issues? Umm, yes please. I'll take a
second helping of that too.
On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <me...@gmail.com> wrote:
> I would like to open up a discussion around creating Docker images for
> Metron. Having this available would provide a leaner alternative to the
> ansible/vagrant environment for development tesing (and even demoing or
> exploring features). It could also relieve some of the dependency version
> conflict issues that we've been experiencing when running integration tests
> in a single JVM.
>
> I would suggest the initial version be intended only for development and
> testing purposes. The general approach could be to create an image for
> each service we depend on and use something like Docker compose to package
> them together. A Dockerfile would either install the service from scratch
> or extend a community image then add any Metron related dependencies or
> configurations on top. The metron-deployment project code could be used as
> a guide.
>
> I would like to see these images added initially to support development and
> testing:
>
> - Kafka with topics preconfigured
> - Storm with metron topology assets installed
> - Zookeeper with paths created and sample configs loaded
> - HBase with sample enrichments and threat intel loaded
> - Elasticsearch configured for Metron
> - MySQL with databases/tables/users created and geo data loaded
>
> Other images that could also be useful:
>
> - Images for each sensor
> - Ambari?
> - Solr
>
> Looking forward to hearing what everyone thinks.
>
> Ryan Merriman
>
--
Nick Allen <ni...@nickallen.org>