You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Simon Elliston Ball <si...@simonellistonball.com> on 2017/10/04 16:45:24 UTC
[DISCUSS] Dropping support for elastic 2.x
A number of people are currently working on upgrading the ES support in Metron to 5.x (including the clients, and the mpack managed install).
Would anyone have any objections to dropping formal support for 2.x as a result of this work? In theory the clients should be backward compatible against older data stores, so metron could be upgraded without needing an elastic upgrade.
In practice, we would need to do pretty extensive testing and I wouldn’t want us to have to code around long term support on older clients if no-one in the community cares enough about the older ES. Do we think there is a case to be made for maintaining long term support for older clients?
Simon
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Casey Stella <ce...@gmail.com>.
Regarding backwards compatibility at the code level, what are the pros/cons
(outside of the obvious con that the transport client will be deprecated)?
I guess what I'm trying to get at is what do we get in terms of
functionality moving to a new backwards-incompatible transport client?
A separate conversation would be what it would take to make Ambari manage
ES 5.x (we could punt here and simply turn off ambari management of ES and
upgrade ES independently as previously said).
On Wed, Oct 4, 2017 at 1:21 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> I should note that there's a difference between supporting INSTALLING
> multiple versions versus being able to manage them.
>
> On Wed, Oct 4, 2017 at 11:20 AM, Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > The question comes back to the DISCUSS I opened the other day about
> > upgrading ES. I believe we could theoretically maintain backwards
> > compatibility, but we'd have to keep the existing TransportClient. It's
> not
> > deprecated yet, but it will be. Keeping the ability to manage ES 2.x and
> > 5.x+ via Ambari might be some work, but it looks like it's just calling
> > shell commands from python, e.g. https://github.com/
> > apache/metron/blob/master/metron-deployment/packaging/
> > ambari/metron-mpack/src/main/resources/common-services/
> > ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py#L43-L45
> >
> >
> > On Wed, Oct 4, 2017 at 11:08 AM, Casey Stella <ce...@gmail.com>
> wrote:
> >
> >> Ok, so, whoever does this ES work, we should ensure the upgrade path is
> at
> >> least spelled out in the Upgrade doc. This would also probably, IMO,
> >> necessitate a major version change in metron.
> >>
> >> On Wed, Oct 4, 2017 at 1:07 PM, Justin Leet <ju...@gmail.com>
> >> wrote:
> >>
> >> > Forgot the link
> >> > https://www.elastic.co/guide/en/elasticsearch/reference/
> >> > current/setup-upgrade.html
> >> >
> >> > On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
> >> > simon@simonellistonball.com> wrote:
> >> >
> >> > > The simplest option would probably be to upgrade the ES and then
> >> reindex
> >> > > from the HDFS store. Alternatively there are means to do inplace
> >> upgrades
> >> > > from 2.x to 5.x I believe.
> >> > >
> >> > > Simon
> >> > >
> >> > > > On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
> >> > > >
> >> > > > So, how would this work in an upgrade scenario that does not
> involve
> >> > > losing
> >> > > > the existing indexed data?
> >> > > >
> >> > > > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> >> > > > michael.miklavcic@gmail.com> wrote:
> >> > > >
> >> > > >> The client I'm currently working on moving towards would *not* be
> >> > > backwards
> >> > > >> compatible.
> >> > > >> https://www.elastic.co/guide/en/elasticsearch/client/java-
> >> > > >> rest/current/java-rest-high-compatibility.html
> >> > > >>
> >> > > >> "
> >> > > >> The High Level Client is guaranteed to be able to communicate
> with
> >> any
> >> > > >> Elasticsearch node running on the same major version and greater
> or
> >> > > equal
> >> > > >> minor version. It doesn’t need to be in the same minor version as
> >> the
> >> > > >> Elasticsearch nodes it communicates with, as it is forward
> >> compatible
> >> > > >> meaning that it supports communicating with later versions of
> >> > > Elasticsearch
> >> > > >> than the one it was developed for.
> >> > > >>
> >> > > >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> >> > > Previous
> >> > > >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully)
> >> supported.
> >> > > >> "
> >> > > >>
> >> > > >> Best,
> >> > > >> Mike
> >> > > >>
> >> > > >>
> >> > > >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> >> > > >> simon@simonellistonball.com> wrote:
> >> > > >>
> >> > > >>> A number of people are currently working on upgrading the ES
> >> support
> >> > in
> >> > > >>> Metron to 5.x (including the clients, and the mpack managed
> >> install).
> >> > > >>>
> >> > > >>> Would anyone have any objections to dropping formal support for
> >> 2.x
> >> > as
> >> > > a
> >> > > >>> result of this work? In theory the clients should be backward
> >> > > compatible
> >> > > >>> against older data stores, so metron could be upgraded without
> >> > needing
> >> > > an
> >> > > >>> elastic upgrade.
> >> > > >>>
> >> > > >>> In practice, we would need to do pretty extensive testing and I
> >> > > wouldn’t
> >> > > >>> want us to have to code around long term support on older
> clients
> >> if
> >> > > >> no-one
> >> > > >>> in the community cares enough about the older ES. Do we think
> >> there
> >> > is
> >> > > a
> >> > > >>> case to be made for maintaining long term support for older
> >> clients?
> >> > > >>>
> >> > > >>> Simon
> >> > > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Michael Miklavcic <mi...@gmail.com>.
I should note that there's a difference between supporting INSTALLING
multiple versions versus being able to manage them.
On Wed, Oct 4, 2017 at 11:20 AM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> The question comes back to the DISCUSS I opened the other day about
> upgrading ES. I believe we could theoretically maintain backwards
> compatibility, but we'd have to keep the existing TransportClient. It's not
> deprecated yet, but it will be. Keeping the ability to manage ES 2.x and
> 5.x+ via Ambari might be some work, but it looks like it's just calling
> shell commands from python, e.g. https://github.com/
> apache/metron/blob/master/metron-deployment/packaging/
> ambari/metron-mpack/src/main/resources/common-services/
> ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py#L43-L45
>
>
> On Wed, Oct 4, 2017 at 11:08 AM, Casey Stella <ce...@gmail.com> wrote:
>
>> Ok, so, whoever does this ES work, we should ensure the upgrade path is at
>> least spelled out in the Upgrade doc. This would also probably, IMO,
>> necessitate a major version change in metron.
>>
>> On Wed, Oct 4, 2017 at 1:07 PM, Justin Leet <ju...@gmail.com>
>> wrote:
>>
>> > Forgot the link
>> > https://www.elastic.co/guide/en/elasticsearch/reference/
>> > current/setup-upgrade.html
>> >
>> > On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
>> > simon@simonellistonball.com> wrote:
>> >
>> > > The simplest option would probably be to upgrade the ES and then
>> reindex
>> > > from the HDFS store. Alternatively there are means to do inplace
>> upgrades
>> > > from 2.x to 5.x I believe.
>> > >
>> > > Simon
>> > >
>> > > > On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
>> > > >
>> > > > So, how would this work in an upgrade scenario that does not involve
>> > > losing
>> > > > the existing indexed data?
>> > > >
>> > > > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
>> > > > michael.miklavcic@gmail.com> wrote:
>> > > >
>> > > >> The client I'm currently working on moving towards would *not* be
>> > > backwards
>> > > >> compatible.
>> > > >> https://www.elastic.co/guide/en/elasticsearch/client/java-
>> > > >> rest/current/java-rest-high-compatibility.html
>> > > >>
>> > > >> "
>> > > >> The High Level Client is guaranteed to be able to communicate with
>> any
>> > > >> Elasticsearch node running on the same major version and greater or
>> > > equal
>> > > >> minor version. It doesn’t need to be in the same minor version as
>> the
>> > > >> Elasticsearch nodes it communicates with, as it is forward
>> compatible
>> > > >> meaning that it supports communicating with later versions of
>> > > Elasticsearch
>> > > >> than the one it was developed for.
>> > > >>
>> > > >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
>> > > Previous
>> > > >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully)
>> supported.
>> > > >> "
>> > > >>
>> > > >> Best,
>> > > >> Mike
>> > > >>
>> > > >>
>> > > >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
>> > > >> simon@simonellistonball.com> wrote:
>> > > >>
>> > > >>> A number of people are currently working on upgrading the ES
>> support
>> > in
>> > > >>> Metron to 5.x (including the clients, and the mpack managed
>> install).
>> > > >>>
>> > > >>> Would anyone have any objections to dropping formal support for
>> 2.x
>> > as
>> > > a
>> > > >>> result of this work? In theory the clients should be backward
>> > > compatible
>> > > >>> against older data stores, so metron could be upgraded without
>> > needing
>> > > an
>> > > >>> elastic upgrade.
>> > > >>>
>> > > >>> In practice, we would need to do pretty extensive testing and I
>> > > wouldn’t
>> > > >>> want us to have to code around long term support on older clients
>> if
>> > > >> no-one
>> > > >>> in the community cares enough about the older ES. Do we think
>> there
>> > is
>> > > a
>> > > >>> case to be made for maintaining long term support for older
>> clients?
>> > > >>>
>> > > >>> Simon
>> > > >>
>> > >
>> > >
>> >
>>
>
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Otto Fowler <ot...@gmail.com>.
If we break out the indexing from the hdfswriting, we could just have two
different topologies to configure couldn’t we?
On October 4, 2017 at 13:20:19, Michael Miklavcic (
michael.miklavcic@gmail.com) wrote:
The question comes back to the DISCUSS I opened the other day about
upgrading ES. I believe we could theoretically maintain backwards
compatibility, but we'd have to keep the existing TransportClient. It's not
deprecated yet, but it will be. Keeping the ability to manage ES 2.x and
5.x+ via Ambari might be some work, but it looks like it's just calling
shell commands from python, e.g.
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py#L43-L45
On Wed, Oct 4, 2017 at 11:08 AM, Casey Stella <ce...@gmail.com> wrote:
> Ok, so, whoever does this ES work, we should ensure the upgrade path is
at
> least spelled out in the Upgrade doc. This would also probably, IMO,
> necessitate a major version change in metron.
>
> On Wed, Oct 4, 2017 at 1:07 PM, Justin Leet <ju...@gmail.com>
wrote:
>
> > Forgot the link
> > https://www.elastic.co/guide/en/elasticsearch/reference/
> > current/setup-upgrade.html
> >
> > On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
> > simon@simonellistonball.com> wrote:
> >
> > > The simplest option would probably be to upgrade the ES and then
> reindex
> > > from the HDFS store. Alternatively there are means to do inplace
> upgrades
> > > from 2.x to 5.x I believe.
> > >
> > > Simon
> > >
> > > > On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
> > > >
> > > > So, how would this work in an upgrade scenario that does not
involve
> > > losing
> > > > the existing indexed data?
> > > >
> > > > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> > > > michael.miklavcic@gmail.com> wrote:
> > > >
> > > >> The client I'm currently working on moving towards would *not* be
> > > backwards
> > > >> compatible.
> > > >> https://www.elastic.co/guide/en/elasticsearch/client/java-
> > > >> rest/current/java-rest-high-compatibility.html
> > > >>
> > > >> "
> > > >> The High Level Client is guaranteed to be able to communicate with
> any
> > > >> Elasticsearch node running on the same major version and greater
or
> > > equal
> > > >> minor version. It doesn’t need to be in the same minor version as
> the
> > > >> Elasticsearch nodes it communicates with, as it is forward
> compatible
> > > >> meaning that it supports communicating with later versions of
> > > Elasticsearch
> > > >> than the one it was developed for.
> > > >>
> > > >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> > > Previous
> > > >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully)
supported.
> > > >> "
> > > >>
> > > >> Best,
> > > >> Mike
> > > >>
> > > >>
> > > >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> > > >> simon@simonellistonball.com> wrote:
> > > >>
> > > >>> A number of people are currently working on upgrading the ES
> support
> > in
> > > >>> Metron to 5.x (including the clients, and the mpack managed
> install).
> > > >>>
> > > >>> Would anyone have any objections to dropping formal support for
2.x
> > as
> > > a
> > > >>> result of this work? In theory the clients should be backward
> > > compatible
> > > >>> against older data stores, so metron could be upgraded without
> > needing
> > > an
> > > >>> elastic upgrade.
> > > >>>
> > > >>> In practice, we would need to do pretty extensive testing and I
> > > wouldn’t
> > > >>> want us to have to code around long term support on older clients
> if
> > > >> no-one
> > > >>> in the community cares enough about the older ES. Do we think
there
> > is
> > > a
> > > >>> case to be made for maintaining long term support for older
> clients?
> > > >>>
> > > >>> Simon
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Michael Miklavcic <mi...@gmail.com>.
The question comes back to the DISCUSS I opened the other day about
upgrading ES. I believe we could theoretically maintain backwards
compatibility, but we'd have to keep the existing TransportClient. It's not
deprecated yet, but it will be. Keeping the ability to manage ES 2.x and
5.x+ via Ambari might be some work, but it looks like it's just calling
shell commands from python, e.g.
https://github.com/apache/metron/blob/master/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/package/scripts/elastic_master.py#L43-L45
On Wed, Oct 4, 2017 at 11:08 AM, Casey Stella <ce...@gmail.com> wrote:
> Ok, so, whoever does this ES work, we should ensure the upgrade path is at
> least spelled out in the Upgrade doc. This would also probably, IMO,
> necessitate a major version change in metron.
>
> On Wed, Oct 4, 2017 at 1:07 PM, Justin Leet <ju...@gmail.com> wrote:
>
> > Forgot the link
> > https://www.elastic.co/guide/en/elasticsearch/reference/
> > current/setup-upgrade.html
> >
> > On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
> > simon@simonellistonball.com> wrote:
> >
> > > The simplest option would probably be to upgrade the ES and then
> reindex
> > > from the HDFS store. Alternatively there are means to do inplace
> upgrades
> > > from 2.x to 5.x I believe.
> > >
> > > Simon
> > >
> > > > On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
> > > >
> > > > So, how would this work in an upgrade scenario that does not involve
> > > losing
> > > > the existing indexed data?
> > > >
> > > > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> > > > michael.miklavcic@gmail.com> wrote:
> > > >
> > > >> The client I'm currently working on moving towards would *not* be
> > > backwards
> > > >> compatible.
> > > >> https://www.elastic.co/guide/en/elasticsearch/client/java-
> > > >> rest/current/java-rest-high-compatibility.html
> > > >>
> > > >> "
> > > >> The High Level Client is guaranteed to be able to communicate with
> any
> > > >> Elasticsearch node running on the same major version and greater or
> > > equal
> > > >> minor version. It doesn’t need to be in the same minor version as
> the
> > > >> Elasticsearch nodes it communicates with, as it is forward
> compatible
> > > >> meaning that it supports communicating with later versions of
> > > Elasticsearch
> > > >> than the one it was developed for.
> > > >>
> > > >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> > > Previous
> > > >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> > > >> "
> > > >>
> > > >> Best,
> > > >> Mike
> > > >>
> > > >>
> > > >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> > > >> simon@simonellistonball.com> wrote:
> > > >>
> > > >>> A number of people are currently working on upgrading the ES
> support
> > in
> > > >>> Metron to 5.x (including the clients, and the mpack managed
> install).
> > > >>>
> > > >>> Would anyone have any objections to dropping formal support for 2.x
> > as
> > > a
> > > >>> result of this work? In theory the clients should be backward
> > > compatible
> > > >>> against older data stores, so metron could be upgraded without
> > needing
> > > an
> > > >>> elastic upgrade.
> > > >>>
> > > >>> In practice, we would need to do pretty extensive testing and I
> > > wouldn’t
> > > >>> want us to have to code around long term support on older clients
> if
> > > >> no-one
> > > >>> in the community cares enough about the older ES. Do we think there
> > is
> > > a
> > > >>> case to be made for maintaining long term support for older
> clients?
> > > >>>
> > > >>> Simon
> > > >>
> > >
> > >
> >
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Casey Stella <ce...@gmail.com>.
Ok, so, whoever does this ES work, we should ensure the upgrade path is at
least spelled out in the Upgrade doc. This would also probably, IMO,
necessitate a major version change in metron.
On Wed, Oct 4, 2017 at 1:07 PM, Justin Leet <ju...@gmail.com> wrote:
> Forgot the link
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/setup-upgrade.html
>
> On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> > The simplest option would probably be to upgrade the ES and then reindex
> > from the HDFS store. Alternatively there are means to do inplace upgrades
> > from 2.x to 5.x I believe.
> >
> > Simon
> >
> > > On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
> > >
> > > So, how would this work in an upgrade scenario that does not involve
> > losing
> > > the existing indexed data?
> > >
> > > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> > > michael.miklavcic@gmail.com> wrote:
> > >
> > >> The client I'm currently working on moving towards would *not* be
> > backwards
> > >> compatible.
> > >> https://www.elastic.co/guide/en/elasticsearch/client/java-
> > >> rest/current/java-rest-high-compatibility.html
> > >>
> > >> "
> > >> The High Level Client is guaranteed to be able to communicate with any
> > >> Elasticsearch node running on the same major version and greater or
> > equal
> > >> minor version. It doesn’t need to be in the same minor version as the
> > >> Elasticsearch nodes it communicates with, as it is forward compatible
> > >> meaning that it supports communicating with later versions of
> > Elasticsearch
> > >> than the one it was developed for.
> > >>
> > >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> > Previous
> > >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> > >> "
> > >>
> > >> Best,
> > >> Mike
> > >>
> > >>
> > >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> > >> simon@simonellistonball.com> wrote:
> > >>
> > >>> A number of people are currently working on upgrading the ES support
> in
> > >>> Metron to 5.x (including the clients, and the mpack managed install).
> > >>>
> > >>> Would anyone have any objections to dropping formal support for 2.x
> as
> > a
> > >>> result of this work? In theory the clients should be backward
> > compatible
> > >>> against older data stores, so metron could be upgraded without
> needing
> > an
> > >>> elastic upgrade.
> > >>>
> > >>> In practice, we would need to do pretty extensive testing and I
> > wouldn’t
> > >>> want us to have to code around long term support on older clients if
> > >> no-one
> > >>> in the community cares enough about the older ES. Do we think there
> is
> > a
> > >>> case to be made for maintaining long term support for older clients?
> > >>>
> > >>> Simon
> > >>
> >
> >
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Justin Leet <ju...@gmail.com>.
Forgot the link
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
On Wed, Oct 4, 2017 at 1:07 PM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> The simplest option would probably be to upgrade the ES and then reindex
> from the HDFS store. Alternatively there are means to do inplace upgrades
> from 2.x to 5.x I believe.
>
> Simon
>
> > On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
> >
> > So, how would this work in an upgrade scenario that does not involve
> losing
> > the existing indexed data?
> >
> > On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> > michael.miklavcic@gmail.com> wrote:
> >
> >> The client I'm currently working on moving towards would *not* be
> backwards
> >> compatible.
> >> https://www.elastic.co/guide/en/elasticsearch/client/java-
> >> rest/current/java-rest-high-compatibility.html
> >>
> >> "
> >> The High Level Client is guaranteed to be able to communicate with any
> >> Elasticsearch node running on the same major version and greater or
> equal
> >> minor version. It doesn’t need to be in the same minor version as the
> >> Elasticsearch nodes it communicates with, as it is forward compatible
> >> meaning that it supports communicating with later versions of
> Elasticsearch
> >> than the one it was developed for.
> >>
> >> The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> Previous
> >> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> >> "
> >>
> >> Best,
> >> Mike
> >>
> >>
> >> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> >> simon@simonellistonball.com> wrote:
> >>
> >>> A number of people are currently working on upgrading the ES support in
> >>> Metron to 5.x (including the clients, and the mpack managed install).
> >>>
> >>> Would anyone have any objections to dropping formal support for 2.x as
> a
> >>> result of this work? In theory the clients should be backward
> compatible
> >>> against older data stores, so metron could be upgraded without needing
> an
> >>> elastic upgrade.
> >>>
> >>> In practice, we would need to do pretty extensive testing and I
> wouldn’t
> >>> want us to have to code around long term support on older clients if
> >> no-one
> >>> in the community cares enough about the older ES. Do we think there is
> a
> >>> case to be made for maintaining long term support for older clients?
> >>>
> >>> Simon
> >>
>
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
The simplest option would probably be to upgrade the ES and then reindex from the HDFS store. Alternatively there are means to do inplace upgrades from 2.x to 5.x I believe.
Simon
> On 4 Oct 2017, at 18:05, Casey Stella <ce...@gmail.com> wrote:
>
> So, how would this work in an upgrade scenario that does not involve losing
> the existing indexed data?
>
> On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> The client I'm currently working on moving towards would *not* be backwards
>> compatible.
>> https://www.elastic.co/guide/en/elasticsearch/client/java-
>> rest/current/java-rest-high-compatibility.html
>>
>> "
>> The High Level Client is guaranteed to be able to communicate with any
>> Elasticsearch node running on the same major version and greater or equal
>> minor version. It doesn’t need to be in the same minor version as the
>> Elasticsearch nodes it communicates with, as it is forward compatible
>> meaning that it supports communicating with later versions of Elasticsearch
>> than the one it was developed for.
>>
>> The 5.6 client can communicate with any 5.6.x Elasticsearch node. Previous
>> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
>> "
>>
>> Best,
>> Mike
>>
>>
>> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
>> simon@simonellistonball.com> wrote:
>>
>>> A number of people are currently working on upgrading the ES support in
>>> Metron to 5.x (including the clients, and the mpack managed install).
>>>
>>> Would anyone have any objections to dropping formal support for 2.x as a
>>> result of this work? In theory the clients should be backward compatible
>>> against older data stores, so metron could be upgraded without needing an
>>> elastic upgrade.
>>>
>>> In practice, we would need to do pretty extensive testing and I wouldn’t
>>> want us to have to code around long term support on older clients if
>> no-one
>>> in the community cares enough about the older ES. Do we think there is a
>>> case to be made for maintaining long term support for older clients?
>>>
>>> Simon
>>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Justin Leet <ju...@gmail.com>.
ES should be upgradeable without wiping. It's the client itself that isn't
backwards compatible. It'll require both an upgrade of Metron and an ES
cluster.
On Wed, Oct 4, 2017 at 1:05 PM, Casey Stella <ce...@gmail.com> wrote:
> So, how would this work in an upgrade scenario that does not involve losing
> the existing indexed data?
>
> On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > The client I'm currently working on moving towards would *not* be
> backwards
> > compatible.
> > https://www.elastic.co/guide/en/elasticsearch/client/java-
> > rest/current/java-rest-high-compatibility.html
> >
> > "
> > The High Level Client is guaranteed to be able to communicate with any
> > Elasticsearch node running on the same major version and greater or equal
> > minor version. It doesn’t need to be in the same minor version as the
> > Elasticsearch nodes it communicates with, as it is forward compatible
> > meaning that it supports communicating with later versions of
> Elasticsearch
> > than the one it was developed for.
> >
> > The 5.6 client can communicate with any 5.6.x Elasticsearch node.
> Previous
> > 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> > "
> >
> > Best,
> > Mike
> >
> >
> > On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> > simon@simonellistonball.com> wrote:
> >
> > > A number of people are currently working on upgrading the ES support in
> > > Metron to 5.x (including the clients, and the mpack managed install).
> > >
> > > Would anyone have any objections to dropping formal support for 2.x as
> a
> > > result of this work? In theory the clients should be backward
> compatible
> > > against older data stores, so metron could be upgraded without needing
> an
> > > elastic upgrade.
> > >
> > > In practice, we would need to do pretty extensive testing and I
> wouldn’t
> > > want us to have to code around long term support on older clients if
> > no-one
> > > in the community cares enough about the older ES. Do we think there is
> a
> > > case to be made for maintaining long term support for older clients?
> > >
> > > Simon
> >
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Casey Stella <ce...@gmail.com>.
So, how would this work in an upgrade scenario that does not involve losing
the existing indexed data?
On Wed, Oct 4, 2017 at 12:55 PM, Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:
> The client I'm currently working on moving towards would *not* be backwards
> compatible.
> https://www.elastic.co/guide/en/elasticsearch/client/java-
> rest/current/java-rest-high-compatibility.html
>
> "
> The High Level Client is guaranteed to be able to communicate with any
> Elasticsearch node running on the same major version and greater or equal
> minor version. It doesn’t need to be in the same minor version as the
> Elasticsearch nodes it communicates with, as it is forward compatible
> meaning that it supports communicating with later versions of Elasticsearch
> than the one it was developed for.
>
> The 5.6 client can communicate with any 5.6.x Elasticsearch node. Previous
> 5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
> "
>
> Best,
> Mike
>
>
> On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> > A number of people are currently working on upgrading the ES support in
> > Metron to 5.x (including the clients, and the mpack managed install).
> >
> > Would anyone have any objections to dropping formal support for 2.x as a
> > result of this work? In theory the clients should be backward compatible
> > against older data stores, so metron could be upgraded without needing an
> > elastic upgrade.
> >
> > In practice, we would need to do pretty extensive testing and I wouldn’t
> > want us to have to code around long term support on older clients if
> no-one
> > in the community cares enough about the older ES. Do we think there is a
> > case to be made for maintaining long term support for older clients?
> >
> > Simon
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Michael Miklavcic <mi...@gmail.com>.
The client I'm currently working on moving towards would *not* be backwards
compatible.
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
"
The High Level Client is guaranteed to be able to communicate with any
Elasticsearch node running on the same major version and greater or equal
minor version. It doesn’t need to be in the same minor version as the
Elasticsearch nodes it communicates with, as it is forward compatible
meaning that it supports communicating with later versions of Elasticsearch
than the one it was developed for.
The 5.6 client can communicate with any 5.6.x Elasticsearch node. Previous
5.x minor versions like 5.5.x, 5.4.x etc. are not (fully) supported.
"
Best,
Mike
On Wed, Oct 4, 2017 at 10:45 AM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> A number of people are currently working on upgrading the ES support in
> Metron to 5.x (including the clients, and the mpack managed install).
>
> Would anyone have any objections to dropping formal support for 2.x as a
> result of this work? In theory the clients should be backward compatible
> against older data stores, so metron could be upgraded without needing an
> elastic upgrade.
>
> In practice, we would need to do pretty extensive testing and I wouldn’t
> want us to have to code around long term support on older clients if no-one
> in the community cares enough about the older ES. Do we think there is a
> case to be made for maintaining long term support for older clients?
>
> Simon
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Casey Stella <ce...@gmail.com>.
I agree, we shouldn't provide documentation for ES upgrade as ES has that
covered. I also agree that we should provide doc for ES mpack upgrade.
On Wed, Oct 4, 2017 at 3:47 PM, James Sirota <js...@apache.org> wrote:
> I am in favor of moving to 5.x and dropping support for 2.x. As Justin
> mentioned, Elastic have very good docs around cluster migrations and the
> procedure itself to upgrade from 2.x to 5.x is very simple.
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/restart-upgrade.html
>
> I don't agree that we should provide documentation for ES upgrade. I think
> pointing to elastic docs should be good enough. I do agree that we should
> provide documentation for the ES mpack upgrade, which we will.
>
> With us supporting 5.x I see little reason to be backwards compatible to
> 2.x
>
>
> 04.10.2017, 11:59, "Farrukh Naveed Anjum" <an...@gmail.com>:
>
> Its better to move to Elastic Search 5 or 6. As Elasticsearch 2.x is
> really pretty old.
>
> On Wed, Oct 4, 2017 at 9:45 PM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> A number of people are currently working on upgrading the ES support in
> Metron to 5.x (including the clients, and the mpack managed install).
>
> Would anyone have any objections to dropping formal support for 2.x as a
> result of this work? In theory the clients should be backward compatible
> against older data stores, so metron could be upgraded without needing an
> elastic upgrade.
>
> In practice, we would need to do pretty extensive testing and I wouldn’t
> want us to have to code around long term support on older clients if no-one
> in the community cares enough about the older ES. Do we think there is a
> case to be made for maintaining long term support for older clients?
>
> Simon
>
>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Michael Miklavcic <mi...@gmail.com>.
For reference, I want to share what our current ES client class
dependencies look like. This is part of what I intend to cleanup as part of
this effort. Note that I've diagramed both the prod as well as test sides
of the source code. Ideally, we'd have a single point of contact with ES to
ease future upgrades.
Mike
On Wed, Oct 4, 2017 at 1:57 PM, Otto Fowler <ot...@gmail.com> wrote:
> If we support indexing through extensions down the road we can add in
> support for older versions or other back ends as well.
>
>
> On October 4, 2017 at 15:47:35, James Sirota (jsirota@apache.org) wrote:
>
> I am in favor of moving to 5.x and dropping support for 2.x. As Justin
> mentioned, Elastic have very good docs around cluster migrations and the
> procedure itself to upgrade from 2.x to 5.x is very simple.
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/restart-upgrade.html
>
> I don't agree that we should provide documentation for ES upgrade. I think
> pointing to elastic docs should be good enough. I do agree that we should
> provide documentation for the ES mpack upgrade, which we will.
>
> With us supporting 5.x I see little reason to be backwards compatible to
> 2.x
>
>
> 04.10.2017, 11:59, "Farrukh Naveed Anjum" <an...@gmail.com>:
>
> Its better to move to Elastic Search 5 or 6. As Elasticsearch 2.x is
> really pretty old.
>
> On Wed, Oct 4, 2017 at 9:45 PM, Simon Elliston Ball <
> simon@simonellistonball.com> wrote:
>
> A number of people are currently working on upgrading the ES support in
> Metron to 5.x (including the clients, and the mpack managed install).
>
> Would anyone have any objections to dropping formal support for 2.x as a
> result of this work? In theory the clients should be backward compatible
> against older data stores, so metron could be upgraded without needing an
> elastic upgrade.
>
> In practice, we would need to do pretty extensive testing and I wouldn’t
> want us to have to code around long term support on older clients if no-one
> in the community cares enough about the older ES. Do we think there is a
> case to be made for maintaining long term support for older clients?
>
> Simon
>
>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
>
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
>
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Otto Fowler <ot...@gmail.com>.
If we support indexing through extensions down the road we can add in
support for older versions or other back ends as well.
On October 4, 2017 at 15:47:35, James Sirota (jsirota@apache.org) wrote:
I am in favor of moving to 5.x and dropping support for 2.x. As Justin
mentioned, Elastic have very good docs around cluster migrations and the
procedure itself to upgrade from 2.x to 5.x is very simple.
https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-upgrade.html
I don't agree that we should provide documentation for ES upgrade. I think
pointing to elastic docs should be good enough. I do agree that we should
provide documentation for the ES mpack upgrade, which we will.
With us supporting 5.x I see little reason to be backwards compatible to 2.x
04.10.2017, 11:59, "Farrukh Naveed Anjum" <an...@gmail.com>:
Its better to move to Elastic Search 5 or 6. As Elasticsearch 2.x is really
pretty old.
On Wed, Oct 4, 2017 at 9:45 PM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
A number of people are currently working on upgrading the ES support in
Metron to 5.x (including the clients, and the mpack managed install).
Would anyone have any objections to dropping formal support for 2.x as a
result of this work? In theory the clients should be backward compatible
against older data stores, so metron could be upgraded without needing an
elastic upgrade.
In practice, we would need to do pretty extensive testing and I wouldn’t
want us to have to code around long term support on older clients if no-one
in the community cares enough about the older ES. Do we think there is a
case to be made for maintaining long term support for older clients?
Simon
--
With Regards
Farrukh Naveed Anjum
-------------------
Thank you,
James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org
Re: [DISCUSS] Dropping support for elastic 2.x
Posted by Farrukh Naveed Anjum <an...@gmail.com>.
Its better to move to Elastic Search 5 or 6. As Elasticsearch 2.x is really
pretty old.
On Wed, Oct 4, 2017 at 9:45 PM, Simon Elliston Ball <
simon@simonellistonball.com> wrote:
> A number of people are currently working on upgrading the ES support in
> Metron to 5.x (including the clients, and the mpack managed install).
>
> Would anyone have any objections to dropping formal support for 2.x as a
> result of this work? In theory the clients should be backward compatible
> against older data stores, so metron could be upgraded without needing an
> elastic upgrade.
>
> In practice, we would need to do pretty extensive testing and I wouldn’t
> want us to have to code around long term support on older clients if no-one
> in the community cares enough about the older ES. Do we think there is a
> case to be made for maintaining long term support for older clients?
>
> Simon
--
With Regards
Farrukh Naveed Anjum