You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Michael Miklavcic <mi...@gmail.com> on 2018/02/16 21:10:05 UTC
[DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron
This came up earlier when discussing work around the ES upgrade:
https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
Looks like Otto made this suggestion and Kyle is on board. I was originally
opposed to this because it did not seem worth the effort to support 2
separate MPacks. However, now that we are working on the Solr upgrade, it
seems like an appropriate solution for enabling us to make the indexing
piece pluggable. I propose that we commence with this solution.
Cheers,
Mike
Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron
Posted by David Lyle <dl...@gmail.com>.
Hi Mike,
I agree. Def +1 on the first approach. Keep it with the Metron version.
-D...
On Wed, Feb 21, 2018 at 11:30 AM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Anyone have any opinions on how we should version the ES/Kibana MPack?
>
> We currently rev the Metron one based on current Metron version and apply
> it to both the overall MPack as well as the individual service versions,
> e.g. metron_mpack-0.4.3.0/common-services/METRON/0.4.3. For ES we have
> been
> keeping the service version matched to the current ES version because it's
> independent of Metron, ie 5.6.2. The upshot is that we can handle this one
> of two ways.
>
> 1. Keep the same approach after splitting off ES and Kibana - ES MPack
> version is set to the current Metron version (0.4.3) and the service
> itself
> is set to the ES version (5.6.2).
> 2. Use ES version for both the MPack version and service version
> (5.6.2).
>
>
> I personally recommend and prefer the first approach because it allows us
> to make changes to the mpack itself without necessarily changing the
> version of ES or Kibana, which is something that is likely to happen. It's
> also consistent and seamless with our current versioning approach. Lastly,
> I don't believe the ES versions provide much contextual sense in the Metron
> world for an MPack version - setting the service version definitely makes
> sense to indicate what exact ES version we're using, but the MPack is
> really our way of providing custom functionality that wraps a specific
> version of ES/Kibana. Hope this makes sense.
>
> Mike
>
>
> On Sat, Feb 17, 2018 at 11:13 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > +1 I agree with Otto's idea. It makes a lot of sense for Elasticsearch
> and
> > Kibana to live in a separate Mpack.
> >
> > This provides us a path forward to support additional indexers like Solr.
> >
> > We should also not force our users to install an external component (like
> > Elasticsearch) using the Mpack. There are just too many installation
> > configurations for us to reasonably support; especially on larger
> > installations. Supporting the Elasticsearch MPack is a project unto
> > itself. That being said, the functionality will still be there for those
> > that want to use it.
> >
> >
> >
> > On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> > > This came up earlier when discussing work around the ES upgrade:
> > > https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> > > 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> > > https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> > > c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
> > >
> > > Looks like Otto made this suggestion and Kyle is on board. I was
> > originally
> > > opposed to this because it did not seem worth the effort to support 2
> > > separate MPacks. However, now that we are working on the Solr upgrade,
> it
> > > seems like an appropriate solution for enabling us to make the indexing
> > > piece pluggable. I propose that we commence with this solution.
> > >
> > > Cheers,
> > > Mike
> > >
> >
>
Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
I agree, the first approach makes the most sense to me.
Jon
On Wed, Feb 21, 2018 at 11:45 AM Nick Allen <ni...@nickallen.org> wrote:
> +1 to the first approach, as you've laid it out. That makes the most sense
> to me. We need a way to rev the version of the ES Mpack independent of the
> ES version.
>
> On Wed, Feb 21, 2018 at 11:30 AM, Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > Anyone have any opinions on how we should version the ES/Kibana MPack?
> >
> > We currently rev the Metron one based on current Metron version and apply
> > it to both the overall MPack as well as the individual service versions,
> > e.g. metron_mpack-0.4.3.0/common-services/METRON/0.4.3. For ES we have
> > been
> > keeping the service version matched to the current ES version because
> it's
> > independent of Metron, ie 5.6.2. The upshot is that we can handle this
> one
> > of two ways.
> >
> > 1. Keep the same approach after splitting off ES and Kibana - ES MPack
> > version is set to the current Metron version (0.4.3) and the service
> > itself
> > is set to the ES version (5.6.2).
> > 2. Use ES version for both the MPack version and service version
> > (5.6.2).
> >
> >
> > I personally recommend and prefer the first approach because it allows us
> > to make changes to the mpack itself without necessarily changing the
> > version of ES or Kibana, which is something that is likely to happen.
> It's
> > also consistent and seamless with our current versioning approach.
> Lastly,
> > I don't believe the ES versions provide much contextual sense in the
> Metron
> > world for an MPack version - setting the service version definitely makes
> > sense to indicate what exact ES version we're using, but the MPack is
> > really our way of providing custom functionality that wraps a specific
> > version of ES/Kibana. Hope this makes sense.
> >
> > Mike
> >
> >
> > On Sat, Feb 17, 2018 at 11:13 AM, Nick Allen <ni...@nickallen.org> wrote:
> >
> > > +1 I agree with Otto's idea. It makes a lot of sense for Elasticsearch
> > and
> > > Kibana to live in a separate Mpack.
> > >
> > > This provides us a path forward to support additional indexers like
> Solr.
> > >
> > > We should also not force our users to install an external component
> (like
> > > Elasticsearch) using the Mpack. There are just too many installation
> > > configurations for us to reasonably support; especially on larger
> > > installations. Supporting the Elasticsearch MPack is a project unto
> > > itself. That being said, the functionality will still be there for
> those
> > > that want to use it.
> > >
> > >
> > >
> > > On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
> > > michael.miklavcic@gmail.com> wrote:
> > >
> > > > This came up earlier when discussing work around the ES upgrade:
> > > > https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> > > > 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> > > > https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> > > > c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
> > > >
> > > > Looks like Otto made this suggestion and Kyle is on board. I was
> > > originally
> > > > opposed to this because it did not seem worth the effort to support 2
> > > > separate MPacks. However, now that we are working on the Solr
> upgrade,
> > it
> > > > seems like an appropriate solution for enabling us to make the
> indexing
> > > > piece pluggable. I propose that we commence with this solution.
> > > >
> > > > Cheers,
> > > > Mike
> > > >
> > >
> >
>
--
Jon
Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron
Posted by Nick Allen <ni...@nickallen.org>.
+1 to the first approach, as you've laid it out. That makes the most sense
to me. We need a way to rev the version of the ES Mpack independent of the
ES version.
On Wed, Feb 21, 2018 at 11:30 AM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> Anyone have any opinions on how we should version the ES/Kibana MPack?
>
> We currently rev the Metron one based on current Metron version and apply
> it to both the overall MPack as well as the individual service versions,
> e.g. metron_mpack-0.4.3.0/common-services/METRON/0.4.3. For ES we have
> been
> keeping the service version matched to the current ES version because it's
> independent of Metron, ie 5.6.2. The upshot is that we can handle this one
> of two ways.
>
> 1. Keep the same approach after splitting off ES and Kibana - ES MPack
> version is set to the current Metron version (0.4.3) and the service
> itself
> is set to the ES version (5.6.2).
> 2. Use ES version for both the MPack version and service version
> (5.6.2).
>
>
> I personally recommend and prefer the first approach because it allows us
> to make changes to the mpack itself without necessarily changing the
> version of ES or Kibana, which is something that is likely to happen. It's
> also consistent and seamless with our current versioning approach. Lastly,
> I don't believe the ES versions provide much contextual sense in the Metron
> world for an MPack version - setting the service version definitely makes
> sense to indicate what exact ES version we're using, but the MPack is
> really our way of providing custom functionality that wraps a specific
> version of ES/Kibana. Hope this makes sense.
>
> Mike
>
>
> On Sat, Feb 17, 2018 at 11:13 AM, Nick Allen <ni...@nickallen.org> wrote:
>
> > +1 I agree with Otto's idea. It makes a lot of sense for Elasticsearch
> and
> > Kibana to live in a separate Mpack.
> >
> > This provides us a path forward to support additional indexers like Solr.
> >
> > We should also not force our users to install an external component (like
> > Elasticsearch) using the Mpack. There are just too many installation
> > configurations for us to reasonably support; especially on larger
> > installations. Supporting the Elasticsearch MPack is a project unto
> > itself. That being said, the functionality will still be there for those
> > that want to use it.
> >
> >
> >
> > On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> > > This came up earlier when discussing work around the ES upgrade:
> > > https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> > > 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> > > https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> > > c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
> > >
> > > Looks like Otto made this suggestion and Kyle is on board. I was
> > originally
> > > opposed to this because it did not seem worth the effort to support 2
> > > separate MPacks. However, now that we are working on the Solr upgrade,
> it
> > > seems like an appropriate solution for enabling us to make the indexing
> > > piece pluggable. I propose that we commence with this solution.
> > >
> > > Cheers,
> > > Mike
> > >
> >
>
Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron
Posted by Michael Miklavcic <mi...@gmail.com>.
Anyone have any opinions on how we should version the ES/Kibana MPack?
We currently rev the Metron one based on current Metron version and apply
it to both the overall MPack as well as the individual service versions,
e.g. metron_mpack-0.4.3.0/common-services/METRON/0.4.3. For ES we have been
keeping the service version matched to the current ES version because it's
independent of Metron, ie 5.6.2. The upshot is that we can handle this one
of two ways.
1. Keep the same approach after splitting off ES and Kibana - ES MPack
version is set to the current Metron version (0.4.3) and the service itself
is set to the ES version (5.6.2).
2. Use ES version for both the MPack version and service version (5.6.2).
I personally recommend and prefer the first approach because it allows us
to make changes to the mpack itself without necessarily changing the
version of ES or Kibana, which is something that is likely to happen. It's
also consistent and seamless with our current versioning approach. Lastly,
I don't believe the ES versions provide much contextual sense in the Metron
world for an MPack version - setting the service version definitely makes
sense to indicate what exact ES version we're using, but the MPack is
really our way of providing custom functionality that wraps a specific
version of ES/Kibana. Hope this makes sense.
Mike
On Sat, Feb 17, 2018 at 11:13 AM, Nick Allen <ni...@nickallen.org> wrote:
> +1 I agree with Otto's idea. It makes a lot of sense for Elasticsearch and
> Kibana to live in a separate Mpack.
>
> This provides us a path forward to support additional indexers like Solr.
>
> We should also not force our users to install an external component (like
> Elasticsearch) using the Mpack. There are just too many installation
> configurations for us to reasonably support; especially on larger
> installations. Supporting the Elasticsearch MPack is a project unto
> itself. That being said, the functionality will still be there for those
> that want to use it.
>
>
>
> On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > This came up earlier when discussing work around the ES upgrade:
> > https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> > 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> > https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> > c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
> >
> > Looks like Otto made this suggestion and Kyle is on board. I was
> originally
> > opposed to this because it did not seem worth the effort to support 2
> > separate MPacks. However, now that we are working on the Solr upgrade, it
> > seems like an appropriate solution for enabling us to make the indexing
> > piece pluggable. I propose that we commence with this solution.
> >
> > Cheers,
> > Mike
> >
>
Re: [DISCUSS] Split Elasticsearch and Kibana into separate MPack from Metron
Posted by Nick Allen <ni...@nickallen.org>.
+1 I agree with Otto's idea. It makes a lot of sense for Elasticsearch and
Kibana to live in a separate Mpack.
This provides us a path forward to support additional indexers like Solr.
We should also not force our users to install an external component (like
Elasticsearch) using the Mpack. There are just too many installation
configurations for us to reasonably support; especially on larger
installations. Supporting the Elasticsearch MPack is a project unto
itself. That being said, the functionality will still be there for those
that want to use it.
On Fri, Feb 16, 2018 at 4:10 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> This came up earlier when discussing work around the ES upgrade:
> https://lists.apache.org/thread.html/66280bc061afbba2c353221c3c05fd
> 74b247b970921c009c29edc815@%3Cdev.metron.apache.org%3E
> https://lists.apache.org/thread.html/8ec83b6a3ef39057c9466ff72a2f63
> c9308452f1ebc1804e67cb495b@%3Cdev.metron.apache.org%3E
>
> Looks like Otto made this suggestion and Kyle is on board. I was originally
> opposed to this because it did not seem worth the effort to support 2
> separate MPacks. However, now that we are working on the Solr upgrade, it
> seems like an appropriate solution for enabling us to make the indexing
> piece pluggable. I propose that we commence with this solution.
>
> Cheers,
> Mike
>