You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Rene Cordier <rc...@apache.org> on 2022/06/15 09:37:24 UTC
Upgrade regarding ElasticSearch
Hello James community!
I would like to start a discussion regarding the upgrade of
ElasticSearch, hoping we can reach a consensus, as I spent quite a lot
of time on this already.
As you know, the version 7.10 has reached EOL already, so we need to
migrate from it.
Thus a while ago I started a very painful migration to ES 8.2 here:
https://github.com/apache/james-project/pull/1018
Before being rightfully reminded by Matthieu Baechler that ES 7.10 is
the last OSI-compliant version of ElasticSearch, before you know they
switched to a new license that's not really open source anymore...
OpenSearch is indeed a fork of ES 7.10 using the Apache License, which
is definitely more in favor for adoption for the migration than ES 8. On
that, I totally agree.
From then, with Benoit Tellier, we did our little extra research then.
If we want to migrate from ES to OpenSearch, there is a few options on
the table actually:
- solution 1: not modifying the ES7 code. Well it's possible, but you
can only use versions 1.x of OpenSearch (1.3.3 atm). However, from 2.x
version of OpenSearch, the support for es7 client has been dropped in
favor of their own clients.
- solution 2: using the java high level rest client from OpenSearch
(https://opensearch.org/docs/latest/clients/java-rest-high-level/): That
client is a basic fork of the java high level rest client from ES. As
this client has been dropped in upper version of ES for a new client
(that you can see in the PR I did before:
https://github.com/apache/james-project/pull/1018), the fork is thus
identical.
Benoit did a little POC on it and it seems you only need to change the
imports and it works with OpenSearch 2.0 without issues (also said here
in their doc:
https://opensearch.org/docs/latest/clients/java-rest-high-level/#migrating-to-the-opensearch-java-high-level-rest-client)
- solution 3: using the new java client
(https://opensearch.org/docs/latest/clients/java/). That client has been
forked from the new java client from ES probably at its beginnings,
before the change of license. In the POC I did here:
https://github.com/apache/james-project/pull/1051, you can see the
structure is very similar to the java client from ES, but with obviously
some changes or bugs as the fork went its way since when from the
original one. That migration is complicated honestly, but because I did
the one to ES then the remaining work is minimal as proven in the POC.
Just a few issues though encountered... (in the POC you can see them)
I think solution 1 is IMO, not an option, as we probably want to migrate
to the latest version of OpenSearch as we are at it now.
Solution 2 is very easy (replace dependencies and imports... nothing
more) and allows to use OpenSearch 2.0.
I would say let's go with it if I didn't invest so much time migrating
to the new java client. Because this is the issue actually. Amazon
states that the java client is supposed to replace the high level one at
some point (like on forums, or the page of the github project
(https://github.com/opensearch-project/opensearch-java). It's a bit
blurry on really when the high level client would be dropped but I
wouldn't be surprised to see it on next major upgrade for example.
So at some point we will have to migrate the client eventually... do we
try to do it now (solution 3) or do we do things simple for now
(solution 2) and keep the work done under the hood for the day the
migration is necessary? (cause a big chunk of it has been done I think).
I'm honestly fine either way, but would love to hear what the community
has to say on the topic.
Sorry it was long! But I hope I gave all the keys necessary to
understand our options here regarding this migration.
Cheers,
Rene.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Upgrade regarding ElasticSearch
Posted by Benoit TELLIER <bt...@apache.org>.
> Sorry it was long! But I hope I gave all the keys necessary to
understand our options here regarding this migration.
Thanks for writing such a great content!
Personally I prefer solution 2: the existing issues will be tackled soon
by OpenSearch team and we will leverage a nice driver that don't bring
the hell of unrelated dependencies on our classpath.
Regards,
Benoit
On 15/06/2022 16:37, Rene Cordier wrote:
> Hello James community!
>
> I would like to start a discussion regarding the upgrade of
> ElasticSearch, hoping we can reach a consensus, as I spent quite a lot
> of time on this already.
>
> As you know, the version 7.10 has reached EOL already, so we need to
> migrate from it.
>
> Thus a while ago I started a very painful migration to ES 8.2 here:
> https://github.com/apache/james-project/pull/1018
>
> Before being rightfully reminded by Matthieu Baechler that ES 7.10 is
> the last OSI-compliant version of ElasticSearch, before you know they
> switched to a new license that's not really open source anymore...
>
> OpenSearch is indeed a fork of ES 7.10 using the Apache License, which
> is definitely more in favor for adoption for the migration than ES 8.
> On that, I totally agree.
>
> From then, with Benoit Tellier, we did our little extra research then.
>
> If we want to migrate from ES to OpenSearch, there is a few options on
> the table actually:
>
> - solution 1: not modifying the ES7 code. Well it's possible, but you
> can only use versions 1.x of OpenSearch (1.3.3 atm). However, from 2.x
> version of OpenSearch, the support for es7 client has been dropped in
> favor of their own clients.
>
> - solution 2: using the java high level rest client from OpenSearch
> (https://opensearch.org/docs/latest/clients/java-rest-high-level/):
> That client is a basic fork of the java high level rest client from
> ES. As this client has been dropped in upper version of ES for a new
> client (that you can see in the PR I did before:
> https://github.com/apache/james-project/pull/1018), the fork is thus
> identical.
> Benoit did a little POC on it and it seems you only need to change the
> imports and it works with OpenSearch 2.0 without issues (also said
> here in their doc:
> https://opensearch.org/docs/latest/clients/java-rest-high-level/#migrating-to-the-opensearch-java-high-level-rest-client)
>
> - solution 3: using the new java client
> (https://opensearch.org/docs/latest/clients/java/). That client has
> been forked from the new java client from ES probably at its
> beginnings, before the change of license. In the POC I did here:
> https://github.com/apache/james-project/pull/1051, you can see the
> structure is very similar to the java client from ES, but with
> obviously some changes or bugs as the fork went its way since when
> from the original one. That migration is complicated honestly, but
> because I did the one to ES then the remaining work is minimal as
> proven in the POC. Just a few issues though encountered... (in the POC
> you can see them)
>
> I think solution 1 is IMO, not an option, as we probably want to
> migrate to the latest version of OpenSearch as we are at it now.
>
> Solution 2 is very easy (replace dependencies and imports... nothing
> more) and allows to use OpenSearch 2.0.
>
> I would say let's go with it if I didn't invest so much time migrating
> to the new java client. Because this is the issue actually. Amazon
> states that the java client is supposed to replace the high level one
> at some point (like on forums, or the page of the github project
> (https://github.com/opensearch-project/opensearch-java). It's a bit
> blurry on really when the high level client would be dropped but I
> wouldn't be surprised to see it on next major upgrade for example.
>
> So at some point we will have to migrate the client eventually... do
> we try to do it now (solution 3) or do we do things simple for now
> (solution 2) and keep the work done under the hood for the day the
> migration is necessary? (cause a big chunk of it has been done I think).
>
> I'm honestly fine either way, but would love to hear what the
> community has to say on the topic.
>
> Sorry it was long! But I hope I gave all the keys necessary to
> understand our options here regarding this migration.
>
> Cheers,
> Rene.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Upgrade regarding ElasticSearch
Posted by Benoit TELLIER <bt...@apache.org>.
Hello,
Answers inlined.
On 17/06/2022 03:28, Matthieu Baechler wrote:
> Hi René,
>
> Thank you for starting this thread.
>
> On Wed, 2022-06-15 at 16:37 +0700, Rene Cordier wrote:
>> Hello James community!
>>
>> I would like to start a discussion regarding the upgrade of
>> ElasticSearch, hoping we can reach a consensus, as I spent quite a lot
>> of time on this already.
>>
>> As you know, the version 7.10 has reached EOL already, so we need to
>> migrate from it.
>>
>> Thus a while ago I started a very painful migration to ES 8.2 here:
>> https://github.com/apache/james-project/pull/1018
>>
>> Before being rightfully reminded by Matthieu Baechler that ES 7.10 is
>> the last OSI-compliant version of ElasticSearch, before you know they
>> switched to a new license that's not really open source anymore...
>>
>> OpenSearch is indeed a fork of ES 7.10 using the Apache License, which
>> is definitely more in favor for adoption for the migration than ES 8. On
>> that, I totally agree.
>>
>> From then, with Benoit Tellier, we did our little extra research then.
>>
>> If we want to migrate from ES to OpenSearch, there is a few options on
>> the table actually:
>>
>> - solution 1: not modifying the ES7 code. Well it's possible, but you
>> can only use versions 1.x of OpenSearch (1.3.3 atm). However, from 2.x
>> version of OpenSearch, the support for es7 client has been dropped in
>> favor of their own clients.
> The best part for this solution is: it should not require anything else
> than changing the docker image reference.
The downside is we keep a crappy client that will not be maintained on
the long run.
This strategy is defeated as soon as you upgrade the clients to the 2.x
release line.
Also both OpenSearch and ElasticSearch started migrating away of the
high level rest client (that brings all Open/ElasticSearch dependencies
in!!!) in favor of a pure HTTP+JSON client.
-> We would benefit from such a migration
-> We might be forced to move out of the high level rest clients anyway.
Finally I could not find information on support for OpenSearch 1.x
release line thus I am not confident promoting this option.
TL/DR: I am reluctant to Option 1.
>> - solution 2: using the java high level rest client from OpenSearch
>> (https://opensearch.org/docs/latest/clients/java-rest-high-level/): That
>> client is a basic fork of the java high level rest client from ES. As
>> this client has been dropped in upper version of ES for a new client
>> (that you can see in the PR I did before:
>> https://github.com/apache/james-project/pull/1018), the fork is thus
>> identical.
>> Benoit did a little POC on it and it seems you only need to change the
>> imports and it works with OpenSearch 2.0 without issues (also said here
>> in their doc:
>> https://opensearch.org/docs/latest/clients/java-rest-high-level/#migrating-to-the-opensearch-java-high-level-rest-client)
>>
>> - solution 3: using the new java client
>> (https://opensearch.org/docs/latest/clients/java/). That client has been
>> forked from the new java client from ES probably at its beginnings,
>> before the change of license. In the POC I did here:
>> https://github.com/apache/james-project/pull/1051, you can see the
>> structure is very similar to the java client from ES, but with obviously
>> some changes or bugs as the fork went its way since when from the
>> original one. That migration is complicated honestly, but because I did
>> the one to ES then the remaining work is minimal as proven in the POC.
>> Just a few issues though encountered... (in the POC you can see them)
>>
>> I think solution 1 is IMO, not an option, as we probably want to migrate
>> to the latest version of OpenSearch as we are at it now.
>>
>> Solution 2 is very easy (replace dependencies and imports... nothing
>> more) and allows to use OpenSearch 2.0.
>>
>> I would say let's go with it if I didn't invest so much time migrating
>> to the new java client. Because this is the issue actually. Amazon
>> states that the java client is supposed to replace the high level one at
>> some point (like on forums, or the page of the github project
>> (https://github.com/opensearch-project/opensearch-java). It's a bit
>> blurry on really when the high level client would be dropped but I
>> wouldn't be surprised to see it on next major upgrade for example.
>>
>> So at some point we will have to migrate the client eventually... do we
>> try to do it now (solution 3) or do we do things simple for now
>> (solution 2) and keep the work done under the hood for the day the
>> migration is necessary? (cause a big chunk of it has been done I think).
>>
>> I'm honestly fine either way, but would love to hear what the community
>> has to say on the topic.
>>
>> Sorry it was long! But I hope I gave all the keys necessary to
>> understand our options here regarding this migration.
>>
> My point is: upgrading to ES 8 was triggered by "ES 7 is EOL".
>
> OpenSearch 1.x is basically a supported ES 7 and the code for ES 7 is
> supposed to work perfectly with OpenSearch 1.x.
>
> Instead of deleting the support for ES 7, we could just keep it and run
> tests against OpenSearch.
>
> Investment is very low and people don't have to switch to non-
> opensource ES 8 if they don't want to.
>
> If ever somebody has interest in migrating to OpenSearch 2, it can be
> migrated at this point.
>
> I would personally postpone the migration and go for solution 1.
That's the red pill, the short term solution that will eventually get
nothing sorted.
We already invested 30 man days on the topic. Which is huge.
We have everything needed to apply all solutions easily.
I don't see us, nor anybody in the community takkle such a bit effort
anytime soon, if we don't conclude it now.
I do believe that people at Elastic wants users to choose their side,
and the SSPL license in the drivers itself is a strong enough no-go to
ban long term support of ElasticSearch in any ASF project.
I would really enjoy we get this problem sorted while it is still fresh.
Regards,
Benoit
>
> Regards,
>
> -- Matthieu
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Upgrade regarding ElasticSearch
Posted by Rene Cordier <rc...@apache.org>.
Hi,
Overall no problem, maybe I wasn't clear enough on my initial mail, or
it was too long to ingest.
But it seems we are all on the same page now so all good :)
Regards,
Rene.
On 17/06/2022 14:07, Matthieu Baechler wrote:
> Hi,
>
> On Fri, 2022-06-17 at 10:19 +0700, Rene Cordier wrote:
>
>
> [...]
>
>>
>>>
>>> Instead of deleting the support for ES 7, we could just keep it and run
>>> tests against OpenSearch.
>>>
>>> Investment is very low and people don't have to switch to non-
>>> opensource ES 8 if they don't want to.
>>
>> I'm sorry you are mixing up things... It's not question anymore to move
>> up to ES 8 at all,
>>
>
> Sorry, I must have missed that part from your email.
>
>
>> none of the 3 solutions I proposed above are implying
>> this.
>>
>
> Well, I was thinking that, given you already have a ES 8 impl almost
> done, you would like to keep it. It looks like I was wrong.
>
>> We agree it's better to just switch to OpenSearch and not continue
>> with ES versions after 7.10.
>>
>
> Definitely something I missed in the discussion.
>
>>>
>>> If ever somebody has interest in migrating to OpenSearch 2, it can be
>>> migrated at this point.
>>
>> Well as said as well... I did spend a lot of time on it personally to
>> migrate to their new java client, that looks very similar to the one on
>> es8 (that's why the POC is starting from the ES work:
>> https://github.com/apache/james-project/pull/1051... but I can clean it
>> up to make all the es 8 stuff disappear?)
>>
>> However I remain an issue with the sort to which we had a workaround
>> with Benoit (but I personally don't like it) and I proposed a fix on
>> their client:
>> https://github.com/opensearch-project/opensearch-java/pull/169 .
>>
>> But maybe a bit early to fully migrate to that yet...
>>
>>>
>>> I would personally postpone the migration and go for solution 1.
>>
>> I understand why not solution 3... but what about solution 2? It's just
>> change the one dependency es client to opensearch high level one and
>> change all the imports and... nothing else. Effort needed is not much
>> more time consuming IMO? And you can use OpenSearch 2. Would like to
>> know what makes you afraid on this?
>>>
>
> To be honest, if you tackle this issue (I mean, if you have time to
> spend on the task), I don't really care how, we have a good test suite,
> we can always change later if needed.
>
> Sorry for my previous email, I got the main assumption wrong.
>
> Regards,
>
> -- Matthieu
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Upgrade regarding ElasticSearch
Posted by Matthieu Baechler <ma...@apache.org>.
Hi,
On Fri, 2022-06-17 at 10:19 +0700, Rene Cordier wrote:
[...]
>
> >
> > Instead of deleting the support for ES 7, we could just keep it and run
> > tests against OpenSearch.
> >
> > Investment is very low and people don't have to switch to non-
> > opensource ES 8 if they don't want to.
>
> I'm sorry you are mixing up things... It's not question anymore to move
> up to ES 8 at all,
>
Sorry, I must have missed that part from your email.
> none of the 3 solutions I proposed above are implying
> this.
>
Well, I was thinking that, given you already have a ES 8 impl almost
done, you would like to keep it. It looks like I was wrong.
> We agree it's better to just switch to OpenSearch and not continue
> with ES versions after 7.10.
>
Definitely something I missed in the discussion.
> >
> > If ever somebody has interest in migrating to OpenSearch 2, it can be
> > migrated at this point.
>
> Well as said as well... I did spend a lot of time on it personally to
> migrate to their new java client, that looks very similar to the one on
> es8 (that's why the POC is starting from the ES work:
> https://github.com/apache/james-project/pull/1051... but I can clean it
> up to make all the es 8 stuff disappear?)
>
> However I remain an issue with the sort to which we had a workaround
> with Benoit (but I personally don't like it) and I proposed a fix on
> their client:
> https://github.com/opensearch-project/opensearch-java/pull/169 .
>
> But maybe a bit early to fully migrate to that yet...
>
> >
> > I would personally postpone the migration and go for solution 1.
>
> I understand why not solution 3... but what about solution 2? It's just
> change the one dependency es client to opensearch high level one and
> change all the imports and... nothing else. Effort needed is not much
> more time consuming IMO? And you can use OpenSearch 2. Would like to
> know what makes you afraid on this?
> >
To be honest, if you tackle this issue (I mean, if you have time to
spend on the task), I don't really care how, we have a good test suite,
we can always change later if needed.
Sorry for my previous email, I got the main assumption wrong.
Regards,
-- Matthieu
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Upgrade regarding ElasticSearch
Posted by Rene Cordier <rc...@apache.org>.
Hi Matthieu, thanks for joining the discussion.
Answers inline...
On 17/06/2022 03:28, Matthieu Baechler wrote:
> Hi René,
>
> Thank you for starting this thread.
>
> On Wed, 2022-06-15 at 16:37 +0700, Rene Cordier wrote:
>> Hello James community!
>>
>> I would like to start a discussion regarding the upgrade of
>> ElasticSearch, hoping we can reach a consensus, as I spent quite a lot
>> of time on this already.
>>
>> As you know, the version 7.10 has reached EOL already, so we need to
>> migrate from it.
>>
>> Thus a while ago I started a very painful migration to ES 8.2 here:
>> https://github.com/apache/james-project/pull/1018
>>
>> Before being rightfully reminded by Matthieu Baechler that ES 7.10 is
>> the last OSI-compliant version of ElasticSearch, before you know they
>> switched to a new license that's not really open source anymore...
>>
>> OpenSearch is indeed a fork of ES 7.10 using the Apache License, which
>> is definitely more in favor for adoption for the migration than ES 8. On
>> that, I totally agree.
>>
>> From then, with Benoit Tellier, we did our little extra research then.
>>
>> If we want to migrate from ES to OpenSearch, there is a few options on
>> the table actually:
>>
>> - solution 1: not modifying the ES7 code. Well it's possible, but you
>> can only use versions 1.x of OpenSearch (1.3.3 atm). However, from 2.x
>> version of OpenSearch, the support for es7 client has been dropped in
>> favor of their own clients.
>
> The best part for this solution is: it should not require anything else
> than changing the docker image reference.
>
>>
>> - solution 2: using the java high level rest client from OpenSearch
>> (https://opensearch.org/docs/latest/clients/java-rest-high-level/): That
>> client is a basic fork of the java high level rest client from ES. As
>> this client has been dropped in upper version of ES for a new client
>> (that you can see in the PR I did before:
>> https://github.com/apache/james-project/pull/1018), the fork is thus
>> identical.
>> Benoit did a little POC on it and it seems you only need to change the
>> imports and it works with OpenSearch 2.0 without issues (also said here
>> in their doc:
>> https://opensearch.org/docs/latest/clients/java-rest-high-level/#migrating-to-the-opensearch-java-high-level-rest-client)
>>
>> - solution 3: using the new java client
>> (https://opensearch.org/docs/latest/clients/java/). That client has been
>> forked from the new java client from ES probably at its beginnings,
>> before the change of license. In the POC I did here:
>> https://github.com/apache/james-project/pull/1051, you can see the
>> structure is very similar to the java client from ES, but with obviously
>> some changes or bugs as the fork went its way since when from the
>> original one. That migration is complicated honestly, but because I did
>> the one to ES then the remaining work is minimal as proven in the POC.
>> Just a few issues though encountered... (in the POC you can see them)
>>
>> I think solution 1 is IMO, not an option, as we probably want to migrate
>> to the latest version of OpenSearch as we are at it now.
>>
>> Solution 2 is very easy (replace dependencies and imports... nothing
>> more) and allows to use OpenSearch 2.0.
>>
>> I would say let's go with it if I didn't invest so much time migrating
>> to the new java client. Because this is the issue actually. Amazon
>> states that the java client is supposed to replace the high level one at
>> some point (like on forums, or the page of the github project
>> (https://github.com/opensearch-project/opensearch-java). It's a bit
>> blurry on really when the high level client would be dropped but I
>> wouldn't be surprised to see it on next major upgrade for example.
>>
>> So at some point we will have to migrate the client eventually... do we
>> try to do it now (solution 3) or do we do things simple for now
>> (solution 2) and keep the work done under the hood for the day the
>> migration is necessary? (cause a big chunk of it has been done I think).
>>
>> I'm honestly fine either way, but would love to hear what the community
>> has to say on the topic.
>>
>> Sorry it was long! But I hope I gave all the keys necessary to
>> understand our options here regarding this migration.
>>
>
> My point is: upgrading to ES 8 was triggered by "ES 7 is EOL
>
> OpenSearch 1.x is basically a supported ES 7 and the code for ES 7 is
> supposed to work perfectly with OpenSearch 1.x.
Yes it does.
>
> Instead of deleting the support for ES 7, we could just keep it and run
> tests against OpenSearch.
>
> Investment is very low and people don't have to switch to non-
> opensource ES 8 if they don't want to.
I'm sorry you are mixing up things... It's not question anymore to move
up to ES 8 at all, none of the 3 solutions I proposed above are implying
this. We agree it's better to just switch to OpenSearch and not continue
with ES versions after 7.10.
>
> If ever somebody has interest in migrating to OpenSearch 2, it can be
> migrated at this point.
Well as said as well... I did spend a lot of time on it personally to
migrate to their new java client, that looks very similar to the one on
es8 (that's why the POC is starting from the ES work:
https://github.com/apache/james-project/pull/1051... but I can clean it
up to make all the es 8 stuff disappear?)
However I remain an issue with the sort to which we had a workaround
with Benoit (but I personally don't like it) and I proposed a fix on
their client:
https://github.com/opensearch-project/opensearch-java/pull/169 .
But maybe a bit early to fully migrate to that yet...
>
> I would personally postpone the migration and go for solution 1.
I understand why not solution 3... but what about solution 2? It's just
change the one dependency es client to opensearch high level one and
change all the imports and... nothing else. Effort needed is not much
more time consuming IMO? And you can use OpenSearch 2. Would like to
know what makes you afraid on this?
>
> Regards,
>
> -- Matthieu
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
Cheers,
Rene.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Upgrade regarding ElasticSearch
Posted by Matthieu Baechler <ma...@apache.org>.
Hi René,
Thank you for starting this thread.
On Wed, 2022-06-15 at 16:37 +0700, Rene Cordier wrote:
> Hello James community!
>
> I would like to start a discussion regarding the upgrade of
> ElasticSearch, hoping we can reach a consensus, as I spent quite a lot
> of time on this already.
>
> As you know, the version 7.10 has reached EOL already, so we need to
> migrate from it.
>
> Thus a while ago I started a very painful migration to ES 8.2 here:
> https://github.com/apache/james-project/pull/1018
>
> Before being rightfully reminded by Matthieu Baechler that ES 7.10 is
> the last OSI-compliant version of ElasticSearch, before you know they
> switched to a new license that's not really open source anymore...
>
> OpenSearch is indeed a fork of ES 7.10 using the Apache License, which
> is definitely more in favor for adoption for the migration than ES 8. On
> that, I totally agree.
>
> From then, with Benoit Tellier, we did our little extra research then.
>
> If we want to migrate from ES to OpenSearch, there is a few options on
> the table actually:
>
> - solution 1: not modifying the ES7 code. Well it's possible, but you
> can only use versions 1.x of OpenSearch (1.3.3 atm). However, from 2.x
> version of OpenSearch, the support for es7 client has been dropped in
> favor of their own clients.
The best part for this solution is: it should not require anything else
than changing the docker image reference.
>
> - solution 2: using the java high level rest client from OpenSearch
> (https://opensearch.org/docs/latest/clients/java-rest-high-level/): That
> client is a basic fork of the java high level rest client from ES. As
> this client has been dropped in upper version of ES for a new client
> (that you can see in the PR I did before:
> https://github.com/apache/james-project/pull/1018), the fork is thus
> identical.
> Benoit did a little POC on it and it seems you only need to change the
> imports and it works with OpenSearch 2.0 without issues (also said here
> in their doc:
> https://opensearch.org/docs/latest/clients/java-rest-high-level/#migrating-to-the-opensearch-java-high-level-rest-client)
>
> - solution 3: using the new java client
> (https://opensearch.org/docs/latest/clients/java/). That client has been
> forked from the new java client from ES probably at its beginnings,
> before the change of license. In the POC I did here:
> https://github.com/apache/james-project/pull/1051, you can see the
> structure is very similar to the java client from ES, but with obviously
> some changes or bugs as the fork went its way since when from the
> original one. That migration is complicated honestly, but because I did
> the one to ES then the remaining work is minimal as proven in the POC.
> Just a few issues though encountered... (in the POC you can see them)
>
> I think solution 1 is IMO, not an option, as we probably want to migrate
> to the latest version of OpenSearch as we are at it now.
>
> Solution 2 is very easy (replace dependencies and imports... nothing
> more) and allows to use OpenSearch 2.0.
>
> I would say let's go with it if I didn't invest so much time migrating
> to the new java client. Because this is the issue actually. Amazon
> states that the java client is supposed to replace the high level one at
> some point (like on forums, or the page of the github project
> (https://github.com/opensearch-project/opensearch-java). It's a bit
> blurry on really when the high level client would be dropped but I
> wouldn't be surprised to see it on next major upgrade for example.
>
> So at some point we will have to migrate the client eventually... do we
> try to do it now (solution 3) or do we do things simple for now
> (solution 2) and keep the work done under the hood for the day the
> migration is necessary? (cause a big chunk of it has been done I think).
>
> I'm honestly fine either way, but would love to hear what the community
> has to say on the topic.
>
> Sorry it was long! But I hope I gave all the keys necessary to
> understand our options here regarding this migration.
>
My point is: upgrading to ES 8 was triggered by "ES 7 is EOL".
OpenSearch 1.x is basically a supported ES 7 and the code for ES 7 is
supposed to work perfectly with OpenSearch 1.x.
Instead of deleting the support for ES 7, we could just keep it and run
tests against OpenSearch.
Investment is very low and people don't have to switch to non-
opensource ES 8 if they don't want to.
If ever somebody has interest in migrating to OpenSearch 2, it can be
migrated at this point.
I would personally postpone the migration and go for solution 1.
Regards,
-- Matthieu
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org