You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Nick Allen <ni...@nickallen.org> on 2017/02/05 21:22:53 UTC
[Discuss] Direction of metron-docker
Where is the `metron-docker` code base headed? What do we want that to be?
How will it work with the other deployment mechanisms?
So far a lot of work has gone into creating the Ambari MPack and we have
been moving away from the legacy Ansible deployments. I have a limited
understanding of the `metron-docker` stuff, but it seems to introduce a
third deployment mechanism via the Docker files.
Is there no way to leverage the existing deployment paths for
`metron-docker`?
---------- Forwarded message ----------
From: nickwallen <gi...@git.apache.org>
Date: Sun, Feb 5, 2017 at 4:09 PM
Subject: [GitHub] incubator-metron issue #441: METRON-646: Add index
templates to metron-docke...
To: dev@metron.incubator.apache.org
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/441
Hi @kylerichardson - I don't want to throw cold water on your effort,
but I am hesitant to create a third deployment code base for
`metron-docker` (in addition to MPack and Ansible.) Do you think that is
what this is or would become?
Besides just the index templates, we'd have to add and support a lot of
other functionality too. Seems like we should have a goal to move towards
a single deployment mechanism that works across multiple platforms (Docker,
Metal, etc).
I don't even know if this is feasible, but it may be worth a community
discussion. I'll kick something off.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [Discuss] Direction of metron-docker
Posted by Nick Allen <ni...@nickallen.org>.
Is having a goal of replacing Vagrant/Virtualbox for Docker in "Quick Dev"
and "Full Dev" mutually exclusive of the goals you outlined above? We
could have both, no? I am unsure if you are objecting to this specific
goal or not.
On Mon, Feb 6, 2017 at 1:03 PM, Ryan Merriman <me...@gmail.com> wrote:
> From the README:
>
> "Metron Docker is a Docker Compose application that is intended for
> development and integration testing of Metron. Use this instead of Vagrant
> when:
>
> - You want an environment that can be built and spun up quickly
> - You need to frequently rebuild and restart services
> - You only need to test, troubleshoot or develop against a subset of
> services"
>
> The "Quick Dev" environment actually serves 2 purposes: a development
> environment and an end-to-end testing environment. This module was
> intended to supplement or provide an alternative to the development
> environment part of "Quick Dev", not the end-to-end testing part. It does
> have "Docker" in the name of the module so I can see how that might suggest
> a fully supported deployment option. It shouldn't be used for that though
> because it doesn't include Ambari or MPack and isn't a true representation
> of a production Metron cluster.
>
> What is the direction? I could see this evolving into a collection of
> profiles or recipes. Need to development a custom parser? Spin up an
> application that only includes the Storm, Kafka and Zookeeper images. Want
> to develop a custom Kibana dashboard? Spin up Elasticsearch and Kibana
> images preloaded with data. Maybe an analytics profile could be created
> that only includes the tools you need for that? The application that
> exists now in metron-docker could be considered a "rest" profile or a
> collection of containers that support all the functions of the rest API.
> It's very general purpose and supports a lot of use cases so I considered
> it a good starting point. It's very useful if you're developing a UI and
> have limited knowledge of Ambari or big data platform services. That was
> the initial motivation.
>
> I think you should view this as more of a toolbox and not a turnkey
> installation solution. Maintaining and building development environments
> is something Docker is a really good fit for and I have found this works
> much better than our Ansible/Vagrant environment. It's really fast and
> stays up all the time.
>
> But it's completely optional. Use it if you think it will help you. Or
> don't if "Quick Dev" is good enough and you've figured out how to tune it
> so that it's not completely unusable. If everybody thinks it's confusing
> and no one uses it then we can take it out and I'll just go back to
> maintaining it privately. But then I would miss out on Kyle's awesome
> contribution :)
>
> Ryan
>
> On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > So what is the direction then, Ryan? Can you describe what this is
> > supposed to be used for?
> >
> > I had thought people wanted this to replace the existing Vagrant-based
> > "Quick Dev"? But apparently this is the assumption that you think I am
> > wrong on.
> >
> >
> >
> > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <me...@gmail.com>
> > wrote:
> >
> > > I agree with everything Kyle said and I think some of Nick's
> assumptions
> > > are false. I don't see this a third deployment option.
> > >
> > > I can understand people not wanting to maintain another deployment path
> > > with Metron already being as big as it is. Ensuring that you've tested
> > and
> > > updated all the appropriate components is already tedious. But in the
> > case
> > > of this module, is it something that needs to updated anytime someone
> > makes
> > > a deployment related change? I don't think so and I've never had that
> > > expectation. The build won't fail and nothing from this project is
> ever
> > > deployed or shipped. For me, maintaining this tool as needed is good
> > > enough. What happens if a change is introduced that breaks
> something? I
> > > discover it as I'm using the tool, fix it, contribute it back and move
> > on.
> > > No big deal. I had been maintaining this privately for a while before
> > the
> > > PR was submitted and the work to keep it current with master was pretty
> > > minimal. Does that mean it should live somewhere else besides the
> master
> > > branch in Metron? I'm not sure what the answer is but there should be
> a
> > > way to share and collaborate with the community on tools like this that
> > > aren't necessarily deployed to production. Kyle's contribution is
> > valuable
> > > and something I would definitely use.
> > >
> > > Ryan
> > >
> >
>
Re: [Discuss] Direction of metron-docker
Posted by Dima Kovalyov <Di...@sstech.us>.
From a user perspective,
We used Vagrant when we first encountered Metron to see and learn about
it, it was quick and easy to deploy - handy. Now we switched to Ambari
Mpack for internal use. We mostly write and deploy Parsers for Metron,
so having just Ambari Mpack is enough for us. I haven't used Vagrant
ever since we started using Ambari Mpack (which was harder to grasp then
Vagrant though).
I like what Ryan suggests to use Docker for, if it will be as easy to
spin-up (basically if setup will be documented) and allow seamless
development for the core team we then would kill two rabbits with that.
Casey,
> *Is the docker infrastructure sufficient to replace vagrant at the moment?*
>
> I do not consider it to be a sufficient environment to acceptance test
> features because it does not install Metron in a realistic manner that
> mimics a user. Vagrant isn't currently where it should be in that regard
> and that is the reason that it is currently getting an overhaul to get
> closer to that ideal.
Considering that we move towards Ambari Mpack (something that was not
considered main deployment solution originally) can we really call it
realistic manner once Ansible (and if) will be deprecated? What will be
the difference between Vagrant and just spinning HDP + Metron inside
Virtualbox?
- Dima
On 02/07/2017 05:12 AM, Kyle Richardson wrote:
> I like the idea of porting some of the integration tests to metron-docker.
> I believe the maven plugin used in the rpm-docker project could be used to
> support that goal.
>
> I agree with Ryan in that I see this as more of a toolbox for developers
> than a supported deployment method. That is the vain I originally created
> this PR in actually. I could continue to load the elasticsearch templates
> manually when working with metron-docker but thought it would be worthwhile
> to automate with a few lines of code.
>
> I have another PR just about ready to go to include a hadoop/hdfs container
> in metron-docker. Would folks see value in including this? The idea was to
> provide an easier way to iterate on HDFS indexing options for cold
> storage/archive data.
>
> As for maintainability, the minimum would be to keep consistent versions of
> storm, hbase, etc between the docker containers and the current supported
> HDP stack. The automation pieces are nice to haves (not blockers in my
> mind) and will continue to simplify as we move more configs into zookeeper
> from the filesystem. I can't think of anything too onerous here but I may
> be missing something obvious.
>
> -Kyle
>
> On Mon, Feb 6, 2017 at 2:30 PM, Otto Fowler <ot...@gmail.com> wrote:
>
>> Beyond the utility, is the cost of maintaining the docker path. It is just
>> another thing that reviewers and committers have to keep in mind or know
>> about when looking at PR’s. Maybe if there was a better and wider spread
>> understanding of the work that is done and how continue it, it would not
>> seem so onerous. It can’t be something that as long as one or two specific
>> people keep up with it, it will be OK, or rather it should not be. Even
>> if, or perhaps because it won’t break the build.
>>
>> There is a lot of utility and value to metron-docker, maybe we just need to
>> think through the sustainability and maintaining issues, so it is a how can
>> we make it work to the project’s satisfaction.
>>
>> On February 6, 2017 at 14:11:04, Casey Stella (cestella@gmail.com) wrote:
>>
>> So, I'm late chiming in here, but I'll go ahead anyway. :)
>>
>> There are a couple of questions here that stand out:
>>
>> *Is the docker infrastructure sufficient to replace vagrant at the moment?*
>>
>> I do not consider it to be a sufficient environment to acceptance test
>> features because it does not install Metron in a realistic manner that
>> mimics a user. Vagrant isn't currently where it should be in that regard
>> and that is the reason that it is currently getting an overhaul to get
>> closer to that ideal.
>>
>> *Does it scratch an itch?*
>>
>> Yes, it does, I think. For those who want a limited portion of metron spun
>> up to smoke-test features in a targeted way, this works well. That being
>> said, in my opinion, you still need to test in vagrant or a cluster. Matt
>> brings up a good point as well about integration test infrastructure. I
>> think there could be an even bigger itch to scratch there as the cost of
>> spinning up and down integration testing components per-test can be time
>> consuming and lead to long build times.
>>
>> *Can we unify them?*
>>
>> I don't know; I'd like to, honestly. I think that it'd be a good
>> discussion to have and it'd be nice to have a path to victory there,
>> because I'm not thrilled about having so many avenues to install. If we
>> don't unify them, I feel that docker will eventually get so far out of date
>> that it will become unusable, frankly.
>>
>>
>> Ultimately, I don't care about the tech stack that we use, docker vs
>> vagrant vs vagrant on docker vs whatever; I just want
>>
>> 1. A way to spin up Metron in an automated way using the same mechanism
>> that the user uses to install (management pack)
>> 2. It'd be nice to be able to slice and dice capabilities (sensors
>> on/off, etc)
>> 3. It'd be nice for it to not cause my machine to sound like a jet is
>> taking off
>>
>>
>> Anyway, those are my $0.02 and I want to thank Nick for bringing up the
>> conversation. I think it's a good one to have, for sure.
>>
>> Casey
>>
>> On Mon, Feb 6, 2017 at 1:44 PM, David Lyle <dl...@gmail.com> wrote:
>>
>>> Exactly that, Matt. I think of it as an integration test enabler, so it's
>>> squarely pointed at the use case you describe.
>>>
>>> Ryan - I didn't hear anyone asking for it to be removed, just some
>>> clarification about its purpose and future use.
>>>
>>> Wrt "completely unusable": perhaps, since we require committed code to be
>>> run through Quick Dev, we should have a focused community effort to make
>> it
>>> a bit more usable. Fwiw, with some recent changes from Justin and Nick,
>>> it's working better than it had been in recent memory. It will be working
>>> even better once I can get my current stuff pushed out. If it's still
>> unsat
>>> after that, we gotta fix it.
>>>
>>> -D...
>>>
>>>
>>> On Mon, Feb 6, 2017 at 1:29 PM, Matt Foley <ma...@apache.org> wrote:
>>>
>>>> There may be another area of application for this. I’m not certain, so
>>>> tell me if I’m off base.
>>>>
>>>> In the context of METRON-322 (adding batchTimeout and
>>>> 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are
>>> some
>>>> fairly obvious unit tests needed, but I find them inadequate to give me
>>>> certainty that some fairly complex-interaction changes are actually
>> doing
>>>> what they’re supposed to be doing. So I was thinking how to do
>>> integration
>>>> testing of only a Bolt component (or worst case, only an Indexing
>>> topology)
>>>> outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t
>>>> coming up with any good answers :-) so I was about to ask the list
>>> anyway.
>>>> Is it feasible/advisable to use Docker to do automated integration
>>> testing
>>>> of small chunks of Metron, such as a single component or a single
>>>> topology? What’s doable? Other better ideas?
>>>>
>>>> Thanks,
>>>> --Matt
>>>>
>>>>
>>>> On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
>>>>
>>>> From the README:
>>>>
>>>> "Metron Docker is a Docker Compose application that is intended for
>>>> development and integration testing of Metron. Use this instead of
>>>> Vagrant
>>>> when:
>>>>
>>>> - You want an environment that can be built and spun up quickly
>>>> - You need to frequently rebuild and restart services
>>>> - You only need to test, troubleshoot or develop against a subset of
>>>> services"
>>>>
>>>> The "Quick Dev" environment actually serves 2 purposes: a
>>> development
>>>> environment and an end-to-end testing environment. This module was
>>>> intended to supplement or provide an alternative to the development
>>>> environment part of "Quick Dev", not the end-to-end testing part. It
>>>> does
>>>> have "Docker" in the name of the module so I can see how that might
>>>> suggest
>>>> a fully supported deployment option. It shouldn't be used for that
>>>> though
>>>> because it doesn't include Ambari or MPack and isn't a true
>>>> representation
>>>> of a production Metron cluster.
>>>>
>>>> What is the direction? I could see this evolving into a collection
>>> of
>>>> profiles or recipes. Need to development a custom parser? Spin up
>>> an
>>>> application that only includes the Storm, Kafka and Zookeeper images.
>>>> Want
>>>> to develop a custom Kibana dashboard? Spin up Elasticsearch and
>>> Kibana
>>>> images preloaded with data. Maybe an analytics profile could be
>>>> created
>>>> that only includes the tools you need for that? The application that
>>>> exists now in metron-docker could be considered a "rest" profile or a
>>>> collection of containers that support all the functions of the rest
>>>> API.
>>>> It's very general purpose and supports a lot of use cases so I
>>>> considered
>>>> it a good starting point. It's very useful if you're developing a UI
>>>> and
>>>> have limited knowledge of Ambari or big data platform services. That
>>>> was
>>>> the initial motivation.
>>>>
>>>> I think you should view this as more of a toolbox and not a turnkey
>>>> installation solution. Maintaining and building development
>>>> environments
>>>> is something Docker is a really good fit for and I have found this
>>>> works
>>>> much better than our Ansible/Vagrant environment. It's really fast
>>> and
>>>> stays up all the time.
>>>>
>>>> But it's completely optional. Use it if you think it will help you.
>>>> Or
>>>> don't if "Quick Dev" is good enough and you've figured out how to
>>> tune
>>>> it
>>>> so that it's not completely unusable. If everybody thinks it's
>>>> confusing
>>>> and no one uses it then we can take it out and I'll just go back to
>>>> maintaining it privately. But then I would miss out on Kyle's
>>> awesome
>>>> contribution :)
>>>>
>>>> Ryan
>>>>
>>>> On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org>
>>>> wrote:
>>>>
>>>>> So what is the direction then, Ryan? Can you describe what this is
>>>>> supposed to be used for?
>>>>>
>>>>> I had thought people wanted this to replace the existing
>>>> Vagrant-based
>>>>> "Quick Dev"? But apparently this is the assumption that you think
>>> I
>>>> am
>>>>> wrong on.
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <
>>> merrimanr@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I agree with everything Kyle said and I think some of Nick's
>>>> assumptions
>>>>>> are false. I don't see this a third deployment option.
>>>>>>
>>>>>> I can understand people not wanting to maintain another
>>> deployment
>>>> path
>>>>>> with Metron already being as big as it is. Ensuring that you've
>>>> tested
>>>>> and
>>>>>> updated all the appropriate components is already tedious. But
>>> in
>>>> the
>>>>> case
>>>>>> of this module, is it something that needs to updated anytime
>>>> someone
>>>>> makes
>>>>>> a deployment related change? I don't think so and I've never had
>>>> that
>>>>>> expectation. The build won't fail and nothing from this project
>>>> is ever
>>>>>> deployed or shipped. For me, maintaining this tool as needed is
>>>> good
>>>>>> enough. What happens if a change is introduced that breaks
>>>> something? I
>>>>>> discover it as I'm using the tool, fix it, contribute it back and
>>>> move
>>>>> on.
>>>>>> No big deal. I had been maintaining this privately for a while
>>>> before
>>>>> the
>>>>>> PR was submitted and the work to keep it current with master was
>>>> pretty
>>>>>> minimal. Does that mean it should live somewhere else besides
>>> the
>>>> master
>>>>>> branch in Metron? I'm not sure what the answer is but there
>>>> should be a
>>>>>> way to share and collaborate with the community on tools like
>>> this
>>>> that
>>>>>> aren't necessarily deployed to production. Kyle's contribution
>>> is
>>>>> valuable
>>>>>> and something I would definitely use.
>>>>>>
>>>>>> Ryan
>>>>>>
>>>>
>>>>
>>>>
Re: [Discuss] Direction of metron-docker
Posted by Kyle Richardson <ky...@gmail.com>.
I like the idea of porting some of the integration tests to metron-docker.
I believe the maven plugin used in the rpm-docker project could be used to
support that goal.
I agree with Ryan in that I see this as more of a toolbox for developers
than a supported deployment method. That is the vain I originally created
this PR in actually. I could continue to load the elasticsearch templates
manually when working with metron-docker but thought it would be worthwhile
to automate with a few lines of code.
I have another PR just about ready to go to include a hadoop/hdfs container
in metron-docker. Would folks see value in including this? The idea was to
provide an easier way to iterate on HDFS indexing options for cold
storage/archive data.
As for maintainability, the minimum would be to keep consistent versions of
storm, hbase, etc between the docker containers and the current supported
HDP stack. The automation pieces are nice to haves (not blockers in my
mind) and will continue to simplify as we move more configs into zookeeper
from the filesystem. I can't think of anything too onerous here but I may
be missing something obvious.
-Kyle
On Mon, Feb 6, 2017 at 2:30 PM, Otto Fowler <ot...@gmail.com> wrote:
> Beyond the utility, is the cost of maintaining the docker path. It is just
> another thing that reviewers and committers have to keep in mind or know
> about when looking at PR’s. Maybe if there was a better and wider spread
> understanding of the work that is done and how continue it, it would not
> seem so onerous. It can’t be something that as long as one or two specific
> people keep up with it, it will be OK, or rather it should not be. Even
> if, or perhaps because it won’t break the build.
>
> There is a lot of utility and value to metron-docker, maybe we just need to
> think through the sustainability and maintaining issues, so it is a how can
> we make it work to the project’s satisfaction.
>
> On February 6, 2017 at 14:11:04, Casey Stella (cestella@gmail.com) wrote:
>
> So, I'm late chiming in here, but I'll go ahead anyway. :)
>
> There are a couple of questions here that stand out:
>
> *Is the docker infrastructure sufficient to replace vagrant at the moment?*
>
> I do not consider it to be a sufficient environment to acceptance test
> features because it does not install Metron in a realistic manner that
> mimics a user. Vagrant isn't currently where it should be in that regard
> and that is the reason that it is currently getting an overhaul to get
> closer to that ideal.
>
> *Does it scratch an itch?*
>
> Yes, it does, I think. For those who want a limited portion of metron spun
> up to smoke-test features in a targeted way, this works well. That being
> said, in my opinion, you still need to test in vagrant or a cluster. Matt
> brings up a good point as well about integration test infrastructure. I
> think there could be an even bigger itch to scratch there as the cost of
> spinning up and down integration testing components per-test can be time
> consuming and lead to long build times.
>
> *Can we unify them?*
>
> I don't know; I'd like to, honestly. I think that it'd be a good
> discussion to have and it'd be nice to have a path to victory there,
> because I'm not thrilled about having so many avenues to install. If we
> don't unify them, I feel that docker will eventually get so far out of date
> that it will become unusable, frankly.
>
>
> Ultimately, I don't care about the tech stack that we use, docker vs
> vagrant vs vagrant on docker vs whatever; I just want
>
> 1. A way to spin up Metron in an automated way using the same mechanism
> that the user uses to install (management pack)
> 2. It'd be nice to be able to slice and dice capabilities (sensors
> on/off, etc)
> 3. It'd be nice for it to not cause my machine to sound like a jet is
> taking off
>
>
> Anyway, those are my $0.02 and I want to thank Nick for bringing up the
> conversation. I think it's a good one to have, for sure.
>
> Casey
>
> On Mon, Feb 6, 2017 at 1:44 PM, David Lyle <dl...@gmail.com> wrote:
>
> > Exactly that, Matt. I think of it as an integration test enabler, so it's
> > squarely pointed at the use case you describe.
> >
> > Ryan - I didn't hear anyone asking for it to be removed, just some
> > clarification about its purpose and future use.
> >
> > Wrt "completely unusable": perhaps, since we require committed code to be
> > run through Quick Dev, we should have a focused community effort to make
> it
> > a bit more usable. Fwiw, with some recent changes from Justin and Nick,
> > it's working better than it had been in recent memory. It will be working
> > even better once I can get my current stuff pushed out. If it's still
> unsat
> > after that, we gotta fix it.
> >
> > -D...
> >
> >
> > On Mon, Feb 6, 2017 at 1:29 PM, Matt Foley <ma...@apache.org> wrote:
> >
> > > There may be another area of application for this. I’m not certain, so
> > > tell me if I’m off base.
> > >
> > > In the context of METRON-322 (adding batchTimeout and
> > > 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are
> > some
> > > fairly obvious unit tests needed, but I find them inadequate to give me
> > > certainty that some fairly complex-interaction changes are actually
> doing
> > > what they’re supposed to be doing. So I was thinking how to do
> > integration
> > > testing of only a Bolt component (or worst case, only an Indexing
> > topology)
> > > outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t
> > > coming up with any good answers :-) so I was about to ask the list
> > anyway.
> > >
> > > Is it feasible/advisable to use Docker to do automated integration
> > testing
> > > of small chunks of Metron, such as a single component or a single
> > > topology? What’s doable? Other better ideas?
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > > On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
> > >
> > > From the README:
> > >
> > > "Metron Docker is a Docker Compose application that is intended for
> > > development and integration testing of Metron. Use this instead of
> > > Vagrant
> > > when:
> > >
> > > - You want an environment that can be built and spun up quickly
> > > - You need to frequently rebuild and restart services
> > > - You only need to test, troubleshoot or develop against a subset of
> > > services"
> > >
> > > The "Quick Dev" environment actually serves 2 purposes: a
> > development
> > > environment and an end-to-end testing environment. This module was
> > > intended to supplement or provide an alternative to the development
> > > environment part of "Quick Dev", not the end-to-end testing part. It
> > > does
> > > have "Docker" in the name of the module so I can see how that might
> > > suggest
> > > a fully supported deployment option. It shouldn't be used for that
> > > though
> > > because it doesn't include Ambari or MPack and isn't a true
> > > representation
> > > of a production Metron cluster.
> > >
> > > What is the direction? I could see this evolving into a collection
> > of
> > > profiles or recipes. Need to development a custom parser? Spin up
> > an
> > > application that only includes the Storm, Kafka and Zookeeper images.
> > > Want
> > > to develop a custom Kibana dashboard? Spin up Elasticsearch and
> > Kibana
> > > images preloaded with data. Maybe an analytics profile could be
> > > created
> > > that only includes the tools you need for that? The application that
> > > exists now in metron-docker could be considered a "rest" profile or a
> > > collection of containers that support all the functions of the rest
> > > API.
> > > It's very general purpose and supports a lot of use cases so I
> > > considered
> > > it a good starting point. It's very useful if you're developing a UI
> > > and
> > > have limited knowledge of Ambari or big data platform services. That
> > > was
> > > the initial motivation.
> > >
> > > I think you should view this as more of a toolbox and not a turnkey
> > > installation solution. Maintaining and building development
> > > environments
> > > is something Docker is a really good fit for and I have found this
> > > works
> > > much better than our Ansible/Vagrant environment. It's really fast
> > and
> > > stays up all the time.
> > >
> > > But it's completely optional. Use it if you think it will help you.
> > > Or
> > > don't if "Quick Dev" is good enough and you've figured out how to
> > tune
> > > it
> > > so that it's not completely unusable. If everybody thinks it's
> > > confusing
> > > and no one uses it then we can take it out and I'll just go back to
> > > maintaining it privately. But then I would miss out on Kyle's
> > awesome
> > > contribution :)
> > >
> > > Ryan
> > >
> > > On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org>
> > > wrote:
> > >
> > > > So what is the direction then, Ryan? Can you describe what this is
> > > > supposed to be used for?
> > > >
> > > > I had thought people wanted this to replace the existing
> > > Vagrant-based
> > > > "Quick Dev"? But apparently this is the assumption that you think
> > I
> > > am
> > > > wrong on.
> > > >
> > > >
> > > >
> > > > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <
> > merrimanr@gmail.com>
> > > > wrote:
> > > >
> > > > > I agree with everything Kyle said and I think some of Nick's
> > > assumptions
> > > > > are false. I don't see this a third deployment option.
> > > > >
> > > > > I can understand people not wanting to maintain another
> > deployment
> > > path
> > > > > with Metron already being as big as it is. Ensuring that you've
> > > tested
> > > > and
> > > > > updated all the appropriate components is already tedious. But
> > in
> > > the
> > > > case
> > > > > of this module, is it something that needs to updated anytime
> > > someone
> > > > makes
> > > > > a deployment related change? I don't think so and I've never had
> > > that
> > > > > expectation. The build won't fail and nothing from this project
> > > is ever
> > > > > deployed or shipped. For me, maintaining this tool as needed is
> > > good
> > > > > enough. What happens if a change is introduced that breaks
> > > something? I
> > > > > discover it as I'm using the tool, fix it, contribute it back and
> > > move
> > > > on.
> > > > > No big deal. I had been maintaining this privately for a while
> > > before
> > > > the
> > > > > PR was submitted and the work to keep it current with master was
> > > pretty
> > > > > minimal. Does that mean it should live somewhere else besides
> > the
> > > master
> > > > > branch in Metron? I'm not sure what the answer is but there
> > > should be a
> > > > > way to share and collaborate with the community on tools like
> > this
> > > that
> > > > > aren't necessarily deployed to production. Kyle's contribution
> > is
> > > > valuable
> > > > > and something I would definitely use.
> > > > >
> > > > > Ryan
> > > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: [Discuss] Direction of metron-docker
Posted by Otto Fowler <ot...@gmail.com>.
Beyond the utility, is the cost of maintaining the docker path. It is just
another thing that reviewers and committers have to keep in mind or know
about when looking at PR’s. Maybe if there was a better and wider spread
understanding of the work that is done and how continue it, it would not
seem so onerous. It can’t be something that as long as one or two specific
people keep up with it, it will be OK, or rather it should not be. Even
if, or perhaps because it won’t break the build.
There is a lot of utility and value to metron-docker, maybe we just need to
think through the sustainability and maintaining issues, so it is a how can
we make it work to the project’s satisfaction.
On February 6, 2017 at 14:11:04, Casey Stella (cestella@gmail.com) wrote:
So, I'm late chiming in here, but I'll go ahead anyway. :)
There are a couple of questions here that stand out:
*Is the docker infrastructure sufficient to replace vagrant at the moment?*
I do not consider it to be a sufficient environment to acceptance test
features because it does not install Metron in a realistic manner that
mimics a user. Vagrant isn't currently where it should be in that regard
and that is the reason that it is currently getting an overhaul to get
closer to that ideal.
*Does it scratch an itch?*
Yes, it does, I think. For those who want a limited portion of metron spun
up to smoke-test features in a targeted way, this works well. That being
said, in my opinion, you still need to test in vagrant or a cluster. Matt
brings up a good point as well about integration test infrastructure. I
think there could be an even bigger itch to scratch there as the cost of
spinning up and down integration testing components per-test can be time
consuming and lead to long build times.
*Can we unify them?*
I don't know; I'd like to, honestly. I think that it'd be a good
discussion to have and it'd be nice to have a path to victory there,
because I'm not thrilled about having so many avenues to install. If we
don't unify them, I feel that docker will eventually get so far out of date
that it will become unusable, frankly.
Ultimately, I don't care about the tech stack that we use, docker vs
vagrant vs vagrant on docker vs whatever; I just want
1. A way to spin up Metron in an automated way using the same mechanism
that the user uses to install (management pack)
2. It'd be nice to be able to slice and dice capabilities (sensors
on/off, etc)
3. It'd be nice for it to not cause my machine to sound like a jet is
taking off
Anyway, those are my $0.02 and I want to thank Nick for bringing up the
conversation. I think it's a good one to have, for sure.
Casey
On Mon, Feb 6, 2017 at 1:44 PM, David Lyle <dl...@gmail.com> wrote:
> Exactly that, Matt. I think of it as an integration test enabler, so it's
> squarely pointed at the use case you describe.
>
> Ryan - I didn't hear anyone asking for it to be removed, just some
> clarification about its purpose and future use.
>
> Wrt "completely unusable": perhaps, since we require committed code to be
> run through Quick Dev, we should have a focused community effort to make
it
> a bit more usable. Fwiw, with some recent changes from Justin and Nick,
> it's working better than it had been in recent memory. It will be working
> even better once I can get my current stuff pushed out. If it's still
unsat
> after that, we gotta fix it.
>
> -D...
>
>
> On Mon, Feb 6, 2017 at 1:29 PM, Matt Foley <ma...@apache.org> wrote:
>
> > There may be another area of application for this. I’m not certain, so
> > tell me if I’m off base.
> >
> > In the context of METRON-322 (adding batchTimeout and
> > 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are
> some
> > fairly obvious unit tests needed, but I find them inadequate to give me
> > certainty that some fairly complex-interaction changes are actually
doing
> > what they’re supposed to be doing. So I was thinking how to do
> integration
> > testing of only a Bolt component (or worst case, only an Indexing
> topology)
> > outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t
> > coming up with any good answers :-) so I was about to ask the list
> anyway.
> >
> > Is it feasible/advisable to use Docker to do automated integration
> testing
> > of small chunks of Metron, such as a single component or a single
> > topology? What’s doable? Other better ideas?
> >
> > Thanks,
> > --Matt
> >
> >
> > On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
> >
> > From the README:
> >
> > "Metron Docker is a Docker Compose application that is intended for
> > development and integration testing of Metron. Use this instead of
> > Vagrant
> > when:
> >
> > - You want an environment that can be built and spun up quickly
> > - You need to frequently rebuild and restart services
> > - You only need to test, troubleshoot or develop against a subset of
> > services"
> >
> > The "Quick Dev" environment actually serves 2 purposes: a
> development
> > environment and an end-to-end testing environment. This module was
> > intended to supplement or provide an alternative to the development
> > environment part of "Quick Dev", not the end-to-end testing part. It
> > does
> > have "Docker" in the name of the module so I can see how that might
> > suggest
> > a fully supported deployment option. It shouldn't be used for that
> > though
> > because it doesn't include Ambari or MPack and isn't a true
> > representation
> > of a production Metron cluster.
> >
> > What is the direction? I could see this evolving into a collection
> of
> > profiles or recipes. Need to development a custom parser? Spin up
> an
> > application that only includes the Storm, Kafka and Zookeeper images.
> > Want
> > to develop a custom Kibana dashboard? Spin up Elasticsearch and
> Kibana
> > images preloaded with data. Maybe an analytics profile could be
> > created
> > that only includes the tools you need for that? The application that
> > exists now in metron-docker could be considered a "rest" profile or a
> > collection of containers that support all the functions of the rest
> > API.
> > It's very general purpose and supports a lot of use cases so I
> > considered
> > it a good starting point. It's very useful if you're developing a UI
> > and
> > have limited knowledge of Ambari or big data platform services. That
> > was
> > the initial motivation.
> >
> > I think you should view this as more of a toolbox and not a turnkey
> > installation solution. Maintaining and building development
> > environments
> > is something Docker is a really good fit for and I have found this
> > works
> > much better than our Ansible/Vagrant environment. It's really fast
> and
> > stays up all the time.
> >
> > But it's completely optional. Use it if you think it will help you.
> > Or
> > don't if "Quick Dev" is good enough and you've figured out how to
> tune
> > it
> > so that it's not completely unusable. If everybody thinks it's
> > confusing
> > and no one uses it then we can take it out and I'll just go back to
> > maintaining it privately. But then I would miss out on Kyle's
> awesome
> > contribution :)
> >
> > Ryan
> >
> > On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> >
> > > So what is the direction then, Ryan? Can you describe what this is
> > > supposed to be used for?
> > >
> > > I had thought people wanted this to replace the existing
> > Vagrant-based
> > > "Quick Dev"? But apparently this is the assumption that you think
> I
> > am
> > > wrong on.
> > >
> > >
> > >
> > > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <
> merrimanr@gmail.com>
> > > wrote:
> > >
> > > > I agree with everything Kyle said and I think some of Nick's
> > assumptions
> > > > are false. I don't see this a third deployment option.
> > > >
> > > > I can understand people not wanting to maintain another
> deployment
> > path
> > > > with Metron already being as big as it is. Ensuring that you've
> > tested
> > > and
> > > > updated all the appropriate components is already tedious. But
> in
> > the
> > > case
> > > > of this module, is it something that needs to updated anytime
> > someone
> > > makes
> > > > a deployment related change? I don't think so and I've never had
> > that
> > > > expectation. The build won't fail and nothing from this project
> > is ever
> > > > deployed or shipped. For me, maintaining this tool as needed is
> > good
> > > > enough. What happens if a change is introduced that breaks
> > something? I
> > > > discover it as I'm using the tool, fix it, contribute it back and
> > move
> > > on.
> > > > No big deal. I had been maintaining this privately for a while
> > before
> > > the
> > > > PR was submitted and the work to keep it current with master was
> > pretty
> > > > minimal. Does that mean it should live somewhere else besides
> the
> > master
> > > > branch in Metron? I'm not sure what the answer is but there
> > should be a
> > > > way to share and collaborate with the community on tools like
> this
> > that
> > > > aren't necessarily deployed to production. Kyle's contribution
> is
> > > valuable
> > > > and something I would definitely use.
> > > >
> > > > Ryan
> > > >
> > >
> >
> >
> >
> >
>
Re: [Discuss] Direction of metron-docker
Posted by Casey Stella <ce...@gmail.com>.
So, I'm late chiming in here, but I'll go ahead anyway. :)
There are a couple of questions here that stand out:
*Is the docker infrastructure sufficient to replace vagrant at the moment?*
I do not consider it to be a sufficient environment to acceptance test
features because it does not install Metron in a realistic manner that
mimics a user. Vagrant isn't currently where it should be in that regard
and that is the reason that it is currently getting an overhaul to get
closer to that ideal.
*Does it scratch an itch?*
Yes, it does, I think. For those who want a limited portion of metron spun
up to smoke-test features in a targeted way, this works well. That being
said, in my opinion, you still need to test in vagrant or a cluster. Matt
brings up a good point as well about integration test infrastructure. I
think there could be an even bigger itch to scratch there as the cost of
spinning up and down integration testing components per-test can be time
consuming and lead to long build times.
*Can we unify them?*
I don't know; I'd like to, honestly. I think that it'd be a good
discussion to have and it'd be nice to have a path to victory there,
because I'm not thrilled about having so many avenues to install. If we
don't unify them, I feel that docker will eventually get so far out of date
that it will become unusable, frankly.
Ultimately, I don't care about the tech stack that we use, docker vs
vagrant vs vagrant on docker vs whatever; I just want
1. A way to spin up Metron in an automated way using the same mechanism
that the user uses to install (management pack)
2. It'd be nice to be able to slice and dice capabilities (sensors
on/off, etc)
3. It'd be nice for it to not cause my machine to sound like a jet is
taking off
Anyway, those are my $0.02 and I want to thank Nick for bringing up the
conversation. I think it's a good one to have, for sure.
Casey
On Mon, Feb 6, 2017 at 1:44 PM, David Lyle <dl...@gmail.com> wrote:
> Exactly that, Matt. I think of it as an integration test enabler, so it's
> squarely pointed at the use case you describe.
>
> Ryan - I didn't hear anyone asking for it to be removed, just some
> clarification about its purpose and future use.
>
> Wrt "completely unusable": perhaps, since we require committed code to be
> run through Quick Dev, we should have a focused community effort to make it
> a bit more usable. Fwiw, with some recent changes from Justin and Nick,
> it's working better than it had been in recent memory. It will be working
> even better once I can get my current stuff pushed out. If it's still unsat
> after that, we gotta fix it.
>
> -D...
>
>
> On Mon, Feb 6, 2017 at 1:29 PM, Matt Foley <ma...@apache.org> wrote:
>
> > There may be another area of application for this. I’m not certain, so
> > tell me if I’m off base.
> >
> > In the context of METRON-322 (adding batchTimeout and
> > 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are
> some
> > fairly obvious unit tests needed, but I find them inadequate to give me
> > certainty that some fairly complex-interaction changes are actually doing
> > what they’re supposed to be doing. So I was thinking how to do
> integration
> > testing of only a Bolt component (or worst case, only an Indexing
> topology)
> > outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t
> > coming up with any good answers :-) so I was about to ask the list
> anyway.
> >
> > Is it feasible/advisable to use Docker to do automated integration
> testing
> > of small chunks of Metron, such as a single component or a single
> > topology? What’s doable? Other better ideas?
> >
> > Thanks,
> > --Matt
> >
> >
> > On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
> >
> > From the README:
> >
> > "Metron Docker is a Docker Compose application that is intended for
> > development and integration testing of Metron. Use this instead of
> > Vagrant
> > when:
> >
> > - You want an environment that can be built and spun up quickly
> > - You need to frequently rebuild and restart services
> > - You only need to test, troubleshoot or develop against a subset of
> > services"
> >
> > The "Quick Dev" environment actually serves 2 purposes: a
> development
> > environment and an end-to-end testing environment. This module was
> > intended to supplement or provide an alternative to the development
> > environment part of "Quick Dev", not the end-to-end testing part. It
> > does
> > have "Docker" in the name of the module so I can see how that might
> > suggest
> > a fully supported deployment option. It shouldn't be used for that
> > though
> > because it doesn't include Ambari or MPack and isn't a true
> > representation
> > of a production Metron cluster.
> >
> > What is the direction? I could see this evolving into a collection
> of
> > profiles or recipes. Need to development a custom parser? Spin up
> an
> > application that only includes the Storm, Kafka and Zookeeper images.
> > Want
> > to develop a custom Kibana dashboard? Spin up Elasticsearch and
> Kibana
> > images preloaded with data. Maybe an analytics profile could be
> > created
> > that only includes the tools you need for that? The application that
> > exists now in metron-docker could be considered a "rest" profile or a
> > collection of containers that support all the functions of the rest
> > API.
> > It's very general purpose and supports a lot of use cases so I
> > considered
> > it a good starting point. It's very useful if you're developing a UI
> > and
> > have limited knowledge of Ambari or big data platform services. That
> > was
> > the initial motivation.
> >
> > I think you should view this as more of a toolbox and not a turnkey
> > installation solution. Maintaining and building development
> > environments
> > is something Docker is a really good fit for and I have found this
> > works
> > much better than our Ansible/Vagrant environment. It's really fast
> and
> > stays up all the time.
> >
> > But it's completely optional. Use it if you think it will help you.
> > Or
> > don't if "Quick Dev" is good enough and you've figured out how to
> tune
> > it
> > so that it's not completely unusable. If everybody thinks it's
> > confusing
> > and no one uses it then we can take it out and I'll just go back to
> > maintaining it privately. But then I would miss out on Kyle's
> awesome
> > contribution :)
> >
> > Ryan
> >
> > On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org>
> > wrote:
> >
> > > So what is the direction then, Ryan? Can you describe what this is
> > > supposed to be used for?
> > >
> > > I had thought people wanted this to replace the existing
> > Vagrant-based
> > > "Quick Dev"? But apparently this is the assumption that you think
> I
> > am
> > > wrong on.
> > >
> > >
> > >
> > > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <
> merrimanr@gmail.com>
> > > wrote:
> > >
> > > > I agree with everything Kyle said and I think some of Nick's
> > assumptions
> > > > are false. I don't see this a third deployment option.
> > > >
> > > > I can understand people not wanting to maintain another
> deployment
> > path
> > > > with Metron already being as big as it is. Ensuring that you've
> > tested
> > > and
> > > > updated all the appropriate components is already tedious. But
> in
> > the
> > > case
> > > > of this module, is it something that needs to updated anytime
> > someone
> > > makes
> > > > a deployment related change? I don't think so and I've never had
> > that
> > > > expectation. The build won't fail and nothing from this project
> > is ever
> > > > deployed or shipped. For me, maintaining this tool as needed is
> > good
> > > > enough. What happens if a change is introduced that breaks
> > something? I
> > > > discover it as I'm using the tool, fix it, contribute it back and
> > move
> > > on.
> > > > No big deal. I had been maintaining this privately for a while
> > before
> > > the
> > > > PR was submitted and the work to keep it current with master was
> > pretty
> > > > minimal. Does that mean it should live somewhere else besides
> the
> > master
> > > > branch in Metron? I'm not sure what the answer is but there
> > should be a
> > > > way to share and collaborate with the community on tools like
> this
> > that
> > > > aren't necessarily deployed to production. Kyle's contribution
> is
> > > valuable
> > > > and something I would definitely use.
> > > >
> > > > Ryan
> > > >
> > >
> >
> >
> >
> >
>
Re: [Discuss] Direction of metron-docker
Posted by David Lyle <dl...@gmail.com>.
Exactly that, Matt. I think of it as an integration test enabler, so it's
squarely pointed at the use case you describe.
Ryan - I didn't hear anyone asking for it to be removed, just some
clarification about its purpose and future use.
Wrt "completely unusable": perhaps, since we require committed code to be
run through Quick Dev, we should have a focused community effort to make it
a bit more usable. Fwiw, with some recent changes from Justin and Nick,
it's working better than it had been in recent memory. It will be working
even better once I can get my current stuff pushed out. If it's still unsat
after that, we gotta fix it.
-D...
On Mon, Feb 6, 2017 at 1:29 PM, Matt Foley <ma...@apache.org> wrote:
> There may be another area of application for this. I’m not certain, so
> tell me if I’m off base.
>
> In the context of METRON-322 (adding batchTimeout and
> 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are some
> fairly obvious unit tests needed, but I find them inadequate to give me
> certainty that some fairly complex-interaction changes are actually doing
> what they’re supposed to be doing. So I was thinking how to do integration
> testing of only a Bolt component (or worst case, only an Indexing topology)
> outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t
> coming up with any good answers :-) so I was about to ask the list anyway.
>
> Is it feasible/advisable to use Docker to do automated integration testing
> of small chunks of Metron, such as a single component or a single
> topology? What’s doable? Other better ideas?
>
> Thanks,
> --Matt
>
>
> On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
>
> From the README:
>
> "Metron Docker is a Docker Compose application that is intended for
> development and integration testing of Metron. Use this instead of
> Vagrant
> when:
>
> - You want an environment that can be built and spun up quickly
> - You need to frequently rebuild and restart services
> - You only need to test, troubleshoot or develop against a subset of
> services"
>
> The "Quick Dev" environment actually serves 2 purposes: a development
> environment and an end-to-end testing environment. This module was
> intended to supplement or provide an alternative to the development
> environment part of "Quick Dev", not the end-to-end testing part. It
> does
> have "Docker" in the name of the module so I can see how that might
> suggest
> a fully supported deployment option. It shouldn't be used for that
> though
> because it doesn't include Ambari or MPack and isn't a true
> representation
> of a production Metron cluster.
>
> What is the direction? I could see this evolving into a collection of
> profiles or recipes. Need to development a custom parser? Spin up an
> application that only includes the Storm, Kafka and Zookeeper images.
> Want
> to develop a custom Kibana dashboard? Spin up Elasticsearch and Kibana
> images preloaded with data. Maybe an analytics profile could be
> created
> that only includes the tools you need for that? The application that
> exists now in metron-docker could be considered a "rest" profile or a
> collection of containers that support all the functions of the rest
> API.
> It's very general purpose and supports a lot of use cases so I
> considered
> it a good starting point. It's very useful if you're developing a UI
> and
> have limited knowledge of Ambari or big data platform services. That
> was
> the initial motivation.
>
> I think you should view this as more of a toolbox and not a turnkey
> installation solution. Maintaining and building development
> environments
> is something Docker is a really good fit for and I have found this
> works
> much better than our Ansible/Vagrant environment. It's really fast and
> stays up all the time.
>
> But it's completely optional. Use it if you think it will help you.
> Or
> don't if "Quick Dev" is good enough and you've figured out how to tune
> it
> so that it's not completely unusable. If everybody thinks it's
> confusing
> and no one uses it then we can take it out and I'll just go back to
> maintaining it privately. But then I would miss out on Kyle's awesome
> contribution :)
>
> Ryan
>
> On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org>
> wrote:
>
> > So what is the direction then, Ryan? Can you describe what this is
> > supposed to be used for?
> >
> > I had thought people wanted this to replace the existing
> Vagrant-based
> > "Quick Dev"? But apparently this is the assumption that you think I
> am
> > wrong on.
> >
> >
> >
> > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <me...@gmail.com>
> > wrote:
> >
> > > I agree with everything Kyle said and I think some of Nick's
> assumptions
> > > are false. I don't see this a third deployment option.
> > >
> > > I can understand people not wanting to maintain another deployment
> path
> > > with Metron already being as big as it is. Ensuring that you've
> tested
> > and
> > > updated all the appropriate components is already tedious. But in
> the
> > case
> > > of this module, is it something that needs to updated anytime
> someone
> > makes
> > > a deployment related change? I don't think so and I've never had
> that
> > > expectation. The build won't fail and nothing from this project
> is ever
> > > deployed or shipped. For me, maintaining this tool as needed is
> good
> > > enough. What happens if a change is introduced that breaks
> something? I
> > > discover it as I'm using the tool, fix it, contribute it back and
> move
> > on.
> > > No big deal. I had been maintaining this privately for a while
> before
> > the
> > > PR was submitted and the work to keep it current with master was
> pretty
> > > minimal. Does that mean it should live somewhere else besides the
> master
> > > branch in Metron? I'm not sure what the answer is but there
> should be a
> > > way to share and collaborate with the community on tools like this
> that
> > > aren't necessarily deployed to production. Kyle's contribution is
> > valuable
> > > and something I would definitely use.
> > >
> > > Ryan
> > >
> >
>
>
>
>
Re: [Discuss] Direction of metron-docker
Posted by Ryan Merriman <me...@gmail.com>.
Matt, if you want it to be automated you will need to use our integration
testing framework since Docker isn't a part of our build process right
now. If you want to use Docker to aid in your development and get you to a
point where things are working as expected, I think it would work well for
that.
Ryan
On Mon, Feb 6, 2017 at 12:29 PM, Matt Foley <ma...@apache.org> wrote:
> There may be another area of application for this. I’m not certain, so
> tell me if I’m off base.
>
> In the context of METRON-322 (adding batchTimeout and
> 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are some
> fairly obvious unit tests needed, but I find them inadequate to give me
> certainty that some fairly complex-interaction changes are actually doing
> what they’re supposed to be doing. So I was thinking how to do integration
> testing of only a Bolt component (or worst case, only an Indexing topology)
> outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t
> coming up with any good answers :-) so I was about to ask the list anyway.
>
> Is it feasible/advisable to use Docker to do automated integration testing
> of small chunks of Metron, such as a single component or a single
> topology? What’s doable? Other better ideas?
>
> Thanks,
> --Matt
>
>
> On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
>
> From the README:
>
> "Metron Docker is a Docker Compose application that is intended for
> development and integration testing of Metron. Use this instead of
> Vagrant
> when:
>
> - You want an environment that can be built and spun up quickly
> - You need to frequently rebuild and restart services
> - You only need to test, troubleshoot or develop against a subset of
> services"
>
> The "Quick Dev" environment actually serves 2 purposes: a development
> environment and an end-to-end testing environment. This module was
> intended to supplement or provide an alternative to the development
> environment part of "Quick Dev", not the end-to-end testing part. It
> does
> have "Docker" in the name of the module so I can see how that might
> suggest
> a fully supported deployment option. It shouldn't be used for that
> though
> because it doesn't include Ambari or MPack and isn't a true
> representation
> of a production Metron cluster.
>
> What is the direction? I could see this evolving into a collection of
> profiles or recipes. Need to development a custom parser? Spin up an
> application that only includes the Storm, Kafka and Zookeeper images.
> Want
> to develop a custom Kibana dashboard? Spin up Elasticsearch and Kibana
> images preloaded with data. Maybe an analytics profile could be
> created
> that only includes the tools you need for that? The application that
> exists now in metron-docker could be considered a "rest" profile or a
> collection of containers that support all the functions of the rest
> API.
> It's very general purpose and supports a lot of use cases so I
> considered
> it a good starting point. It's very useful if you're developing a UI
> and
> have limited knowledge of Ambari or big data platform services. That
> was
> the initial motivation.
>
> I think you should view this as more of a toolbox and not a turnkey
> installation solution. Maintaining and building development
> environments
> is something Docker is a really good fit for and I have found this
> works
> much better than our Ansible/Vagrant environment. It's really fast and
> stays up all the time.
>
> But it's completely optional. Use it if you think it will help you.
> Or
> don't if "Quick Dev" is good enough and you've figured out how to tune
> it
> so that it's not completely unusable. If everybody thinks it's
> confusing
> and no one uses it then we can take it out and I'll just go back to
> maintaining it privately. But then I would miss out on Kyle's awesome
> contribution :)
>
> Ryan
>
> On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org>
> wrote:
>
> > So what is the direction then, Ryan? Can you describe what this is
> > supposed to be used for?
> >
> > I had thought people wanted this to replace the existing
> Vagrant-based
> > "Quick Dev"? But apparently this is the assumption that you think I
> am
> > wrong on.
> >
> >
> >
> > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <me...@gmail.com>
> > wrote:
> >
> > > I agree with everything Kyle said and I think some of Nick's
> assumptions
> > > are false. I don't see this a third deployment option.
> > >
> > > I can understand people not wanting to maintain another deployment
> path
> > > with Metron already being as big as it is. Ensuring that you've
> tested
> > and
> > > updated all the appropriate components is already tedious. But in
> the
> > case
> > > of this module, is it something that needs to updated anytime
> someone
> > makes
> > > a deployment related change? I don't think so and I've never had
> that
> > > expectation. The build won't fail and nothing from this project
> is ever
> > > deployed or shipped. For me, maintaining this tool as needed is
> good
> > > enough. What happens if a change is introduced that breaks
> something? I
> > > discover it as I'm using the tool, fix it, contribute it back and
> move
> > on.
> > > No big deal. I had been maintaining this privately for a while
> before
> > the
> > > PR was submitted and the work to keep it current with master was
> pretty
> > > minimal. Does that mean it should live somewhere else besides the
> master
> > > branch in Metron? I'm not sure what the answer is but there
> should be a
> > > way to share and collaborate with the community on tools like this
> that
> > > aren't necessarily deployed to production. Kyle's contribution is
> > valuable
> > > and something I would definitely use.
> > >
> > > Ryan
> > >
> >
>
>
>
>
Re: [Discuss] Direction of metron-docker
Posted by Matt Foley <ma...@apache.org>.
There may be another area of application for this. I’m not certain, so tell me if I’m off base.
In the context of METRON-322 (adding batchTimeout and 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are some fairly obvious unit tests needed, but I find them inadequate to give me certainty that some fairly complex-interaction changes are actually doing what they’re supposed to be doing. So I was thinking how to do integration testing of only a Bolt component (or worst case, only an Indexing topology) outside of spinning up a whole Metron / Hadoop Stack cluster. I wasn’t coming up with any good answers :-) so I was about to ask the list anyway.
Is it feasible/advisable to use Docker to do automated integration testing of small chunks of Metron, such as a single component or a single topology? What’s doable? Other better ideas?
Thanks,
--Matt
On 2/6/17, 10:03 AM, "Ryan Merriman" <me...@gmail.com> wrote:
From the README:
"Metron Docker is a Docker Compose application that is intended for
development and integration testing of Metron. Use this instead of Vagrant
when:
- You want an environment that can be built and spun up quickly
- You need to frequently rebuild and restart services
- You only need to test, troubleshoot or develop against a subset of
services"
The "Quick Dev" environment actually serves 2 purposes: a development
environment and an end-to-end testing environment. This module was
intended to supplement or provide an alternative to the development
environment part of "Quick Dev", not the end-to-end testing part. It does
have "Docker" in the name of the module so I can see how that might suggest
a fully supported deployment option. It shouldn't be used for that though
because it doesn't include Ambari or MPack and isn't a true representation
of a production Metron cluster.
What is the direction? I could see this evolving into a collection of
profiles or recipes. Need to development a custom parser? Spin up an
application that only includes the Storm, Kafka and Zookeeper images. Want
to develop a custom Kibana dashboard? Spin up Elasticsearch and Kibana
images preloaded with data. Maybe an analytics profile could be created
that only includes the tools you need for that? The application that
exists now in metron-docker could be considered a "rest" profile or a
collection of containers that support all the functions of the rest API.
It's very general purpose and supports a lot of use cases so I considered
it a good starting point. It's very useful if you're developing a UI and
have limited knowledge of Ambari or big data platform services. That was
the initial motivation.
I think you should view this as more of a toolbox and not a turnkey
installation solution. Maintaining and building development environments
is something Docker is a really good fit for and I have found this works
much better than our Ansible/Vagrant environment. It's really fast and
stays up all the time.
But it's completely optional. Use it if you think it will help you. Or
don't if "Quick Dev" is good enough and you've figured out how to tune it
so that it's not completely unusable. If everybody thinks it's confusing
and no one uses it then we can take it out and I'll just go back to
maintaining it privately. But then I would miss out on Kyle's awesome
contribution :)
Ryan
On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org> wrote:
> So what is the direction then, Ryan? Can you describe what this is
> supposed to be used for?
>
> I had thought people wanted this to replace the existing Vagrant-based
> "Quick Dev"? But apparently this is the assumption that you think I am
> wrong on.
>
>
>
> On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <me...@gmail.com>
> wrote:
>
> > I agree with everything Kyle said and I think some of Nick's assumptions
> > are false. I don't see this a third deployment option.
> >
> > I can understand people not wanting to maintain another deployment path
> > with Metron already being as big as it is. Ensuring that you've tested
> and
> > updated all the appropriate components is already tedious. But in the
> case
> > of this module, is it something that needs to updated anytime someone
> makes
> > a deployment related change? I don't think so and I've never had that
> > expectation. The build won't fail and nothing from this project is ever
> > deployed or shipped. For me, maintaining this tool as needed is good
> > enough. What happens if a change is introduced that breaks something? I
> > discover it as I'm using the tool, fix it, contribute it back and move
> on.
> > No big deal. I had been maintaining this privately for a while before
> the
> > PR was submitted and the work to keep it current with master was pretty
> > minimal. Does that mean it should live somewhere else besides the master
> > branch in Metron? I'm not sure what the answer is but there should be a
> > way to share and collaborate with the community on tools like this that
> > aren't necessarily deployed to production. Kyle's contribution is
> valuable
> > and something I would definitely use.
> >
> > Ryan
> >
>
Re: [Discuss] Direction of metron-docker
Posted by Ryan Merriman <me...@gmail.com>.
From the README:
"Metron Docker is a Docker Compose application that is intended for
development and integration testing of Metron. Use this instead of Vagrant
when:
- You want an environment that can be built and spun up quickly
- You need to frequently rebuild and restart services
- You only need to test, troubleshoot or develop against a subset of
services"
The "Quick Dev" environment actually serves 2 purposes: a development
environment and an end-to-end testing environment. This module was
intended to supplement or provide an alternative to the development
environment part of "Quick Dev", not the end-to-end testing part. It does
have "Docker" in the name of the module so I can see how that might suggest
a fully supported deployment option. It shouldn't be used for that though
because it doesn't include Ambari or MPack and isn't a true representation
of a production Metron cluster.
What is the direction? I could see this evolving into a collection of
profiles or recipes. Need to development a custom parser? Spin up an
application that only includes the Storm, Kafka and Zookeeper images. Want
to develop a custom Kibana dashboard? Spin up Elasticsearch and Kibana
images preloaded with data. Maybe an analytics profile could be created
that only includes the tools you need for that? The application that
exists now in metron-docker could be considered a "rest" profile or a
collection of containers that support all the functions of the rest API.
It's very general purpose and supports a lot of use cases so I considered
it a good starting point. It's very useful if you're developing a UI and
have limited knowledge of Ambari or big data platform services. That was
the initial motivation.
I think you should view this as more of a toolbox and not a turnkey
installation solution. Maintaining and building development environments
is something Docker is a really good fit for and I have found this works
much better than our Ansible/Vagrant environment. It's really fast and
stays up all the time.
But it's completely optional. Use it if you think it will help you. Or
don't if "Quick Dev" is good enough and you've figured out how to tune it
so that it's not completely unusable. If everybody thinks it's confusing
and no one uses it then we can take it out and I'll just go back to
maintaining it privately. But then I would miss out on Kyle's awesome
contribution :)
Ryan
On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <ni...@nickallen.org> wrote:
> So what is the direction then, Ryan? Can you describe what this is
> supposed to be used for?
>
> I had thought people wanted this to replace the existing Vagrant-based
> "Quick Dev"? But apparently this is the assumption that you think I am
> wrong on.
>
>
>
> On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <me...@gmail.com>
> wrote:
>
> > I agree with everything Kyle said and I think some of Nick's assumptions
> > are false. I don't see this a third deployment option.
> >
> > I can understand people not wanting to maintain another deployment path
> > with Metron already being as big as it is. Ensuring that you've tested
> and
> > updated all the appropriate components is already tedious. But in the
> case
> > of this module, is it something that needs to updated anytime someone
> makes
> > a deployment related change? I don't think so and I've never had that
> > expectation. The build won't fail and nothing from this project is ever
> > deployed or shipped. For me, maintaining this tool as needed is good
> > enough. What happens if a change is introduced that breaks something? I
> > discover it as I'm using the tool, fix it, contribute it back and move
> on.
> > No big deal. I had been maintaining this privately for a while before
> the
> > PR was submitted and the work to keep it current with master was pretty
> > minimal. Does that mean it should live somewhere else besides the master
> > branch in Metron? I'm not sure what the answer is but there should be a
> > way to share and collaborate with the community on tools like this that
> > aren't necessarily deployed to production. Kyle's contribution is
> valuable
> > and something I would definitely use.
> >
> > Ryan
> >
>
Re: [Discuss] Direction of metron-docker
Posted by Nick Allen <ni...@nickallen.org>.
So what is the direction then, Ryan? Can you describe what this is
supposed to be used for?
I had thought people wanted this to replace the existing Vagrant-based
"Quick Dev"? But apparently this is the assumption that you think I am
wrong on.
On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <me...@gmail.com> wrote:
> I agree with everything Kyle said and I think some of Nick's assumptions
> are false. I don't see this a third deployment option.
>
> I can understand people not wanting to maintain another deployment path
> with Metron already being as big as it is. Ensuring that you've tested and
> updated all the appropriate components is already tedious. But in the case
> of this module, is it something that needs to updated anytime someone makes
> a deployment related change? I don't think so and I've never had that
> expectation. The build won't fail and nothing from this project is ever
> deployed or shipped. For me, maintaining this tool as needed is good
> enough. What happens if a change is introduced that breaks something? I
> discover it as I'm using the tool, fix it, contribute it back and move on.
> No big deal. I had been maintaining this privately for a while before the
> PR was submitted and the work to keep it current with master was pretty
> minimal. Does that mean it should live somewhere else besides the master
> branch in Metron? I'm not sure what the answer is but there should be a
> way to share and collaborate with the community on tools like this that
> aren't necessarily deployed to production. Kyle's contribution is valuable
> and something I would definitely use.
>
> Ryan
>
Re: [Discuss] Direction of metron-docker
Posted by Ryan Merriman <me...@gmail.com>.
I agree with everything Kyle said and I think some of Nick's assumptions
are false. I don't see this a third deployment option.
I can understand people not wanting to maintain another deployment path
with Metron already being as big as it is. Ensuring that you've tested and
updated all the appropriate components is already tedious. But in the case
of this module, is it something that needs to updated anytime someone makes
a deployment related change? I don't think so and I've never had that
expectation. The build won't fail and nothing from this project is ever
deployed or shipped. For me, maintaining this tool as needed is good
enough. What happens if a change is introduced that breaks something? I
discover it as I'm using the tool, fix it, contribute it back and move on.
No big deal. I had been maintaining this privately for a while before the
PR was submitted and the work to keep it current with master was pretty
minimal. Does that mean it should live somewhere else besides the master
branch in Metron? I'm not sure what the answer is but there should be a
way to share and collaborate with the community on tools like this that
aren't necessarily deployed to production. Kyle's contribution is valuable
and something I would definitely use.
Ryan
Re: [Discuss] Direction of metron-docker
Posted by Kyle Richardson <ky...@gmail.com>.
My working assumption has been that metron-docker would provide a lightweight development environment without all of the overhead of a full cluster. The ability to quickly spin up and down specific components while coding is a big win in my book.
Unfortunately, since metron-docker containers are built on the individual binaries and not Ambari/HDP, it's not really feasible to reuse the MPack deployment. There's potential to reuse some of the Ansible roles but I haven't looked into it in detail.
To Nick's point, I'm not sure we want another entire deployment solution. What's the right balance? Honestly, I'm not sure. I'd like to think it's enough automation to support end-to-end data flow but not so much to become a burden to maintain for the community.
I'm very interested to hear others thoughts on this.
-Kyle
> On Feb 5, 2017, at 4:22 PM, Nick Allen <ni...@nickallen.org> wrote:
>
> Where is the `metron-docker` code base headed? What do we want that to be?
> How will it work with the other deployment mechanisms?
>
> So far a lot of work has gone into creating the Ambari MPack and we have
> been moving away from the legacy Ansible deployments. I have a limited
> understanding of the `metron-docker` stuff, but it seems to introduce a
> third deployment mechanism via the Docker files.
>
> Is there no way to leverage the existing deployment paths for
> `metron-docker`?
>
>
>
> ---------- Forwarded message ----------
> From: nickwallen <gi...@git.apache.org>
> Date: Sun, Feb 5, 2017 at 4:09 PM
> Subject: [GitHub] incubator-metron issue #441: METRON-646: Add index
> templates to metron-docke...
> To: dev@metron.incubator.apache.org
>
>
> Github user nickwallen commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/441
>
> Hi @kylerichardson - I don't want to throw cold water on your effort,
> but I am hesitant to create a third deployment code base for
> `metron-docker` (in addition to MPack and Ansible.) Do you think that is
> what this is or would become?
>
> Besides just the index templates, we'd have to add and support a lot of
> other functionality too. Seems like we should have a goal to move towards
> a single deployment mechanism that works across multiple platforms (Docker,
> Metal, etc).
>
> I don't even know if this is feasible, but it may be worth a community
> discussion. I'll kick something off.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---