You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Niels Basjes <Ni...@basjes.nl> on 2022/08/11 14:45:00 UTC
[flink-connector-elasticsearch] Supporting ElasticSearch 8 & Dropping ES 6 ?
Hi,
I was building something that puts data in ES and found that the current
connector is not compatible with an ES 8 cluster.
I found that
- The currently employed `RestHighLevelClient` has been deprecated by
Elastic .
- There is an `ApiCompatibilityMode` you can use in the latest ES 7
RestHighLevelClient to connect to an ES 8 cluster (works on my machine).
- Between the 7.10 (used now) and the 7.17 (latest) they have introduced in
addition to the ApiCompatibilityMode also several breaking API changes by
changing the packages of several classes we use.
- If you update to the latest 7x then the
`flink-connector-elasticsearch-base` can no longer be shared between ES 6
and ES 7 simply because of these package changes.
- In 2022 there was still an ES 6 release but according to
https://www.elastic.co/support/eol it is now EOL.
So I now see several ways forward:
1) We drop ES6, update ES7 and use this compatibility mode as a quick
solution.
2) We keep ES6, duplicating the `flink-connector-elasticsearch-base` (into
a 6.x and 7.x version with only these few classes being different), update
ES7 and use this compatibility mode as a quick solution.
Later on someone should then create a connector based on the new
ElasticSearch 8 Java client API.
See also:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/master/migrate-hlrc.html
My question is
- Which do we do?
- I vote for the "We drop ES6, update ES7 and use this compatibility
mode"
- Where do we do it?
- We have these connectors in both the 'flink' and
'flink-connector-elasticsearch' repos. Do we update only the new one? Or
both?
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [flink-connector-elasticsearch] Supporting ElasticSearch 8 & Dropping ES 6 ?
Posted by Niels Basjes <Ni...@basjes.nl>.
Ok, that's a bit bigger than I thought.
On Tue, Aug 30, 2022 at 10:03 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi Niels,
>
> With regards to ES support, we should follow the supported version that ES
> supports [1]. So if we add support for ES8, we drop support for ES6.
>
> With regards to the used clients, we can't upgrade to the latest version of
> `RestHighLevelClient`. That's because Elasticsearch has changed the license
> from version 7.11 onwards to the Apache-incompatible SSPL license [2]. What
> needs to happen is that the ES connector is refactored to use the new
> Elasticsearch Java Client [3] which is still available under Apache 2.0
> license in order to add support for ES8 clusters.
>
> For now, we need to make Elasticsearch changes to both the Flink main repo
> and the external one. Hopefully we can drop the ES connector from Flink's
> repo with Flink 1.17.
>
> Best regards,
>
> Martijn
>
> [1] https://www.elastic.co/support/eol
> [2] https://www.elastic.co/pricing/faq/licensing
> [3] https://github.com/elastic/elasticsearch-java
>
> Op do 11 aug. 2022 om 16:45 schreef Niels Basjes <Ni...@basjes.nl>:
>
> > Hi,
> >
> > I was building something that puts data in ES and found that the current
> > connector is not compatible with an ES 8 cluster.
> >
> > I found that
> > - The currently employed `RestHighLevelClient` has been deprecated by
> > Elastic .
> > - There is an `ApiCompatibilityMode` you can use in the latest ES 7
> > RestHighLevelClient to connect to an ES 8 cluster (works on my machine).
> > - Between the 7.10 (used now) and the 7.17 (latest) they have introduced
> in
> > addition to the ApiCompatibilityMode also several breaking API changes by
> > changing the packages of several classes we use.
> > - If you update to the latest 7x then the
> > `flink-connector-elasticsearch-base` can no longer be shared between ES 6
> > and ES 7 simply because of these package changes.
> > - In 2022 there was still an ES 6 release but according to
> > https://www.elastic.co/support/eol it is now EOL.
> >
> > So I now see several ways forward:
> > 1) We drop ES6, update ES7 and use this compatibility mode as a quick
> > solution.
> > 2) We keep ES6, duplicating the `flink-connector-elasticsearch-base`
> (into
> > a 6.x and 7.x version with only these few classes being different),
> update
> > ES7 and use this compatibility mode as a quick solution.
> >
> > Later on someone should then create a connector based on the new
> > ElasticSearch 8 Java client API.
> >
> > See also:
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/master/migrate-hlrc.html
> >
> > My question is
> > - Which do we do?
> > - I vote for the "We drop ES6, update ES7 and use this compatibility
> > mode"
> > - Where do we do it?
> > - We have these connectors in both the 'flink' and
> > 'flink-connector-elasticsearch' repos. Do we update only the new one? Or
> > both?
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>
--
Best regards / Met vriendelijke groeten,
Niels Basjes
Re: [flink-connector-elasticsearch] Supporting ElasticSearch 8 & Dropping ES 6 ?
Posted by Martijn Visser <ma...@apache.org>.
Hi Niels,
With regards to ES support, we should follow the supported version that ES
supports [1]. So if we add support for ES8, we drop support for ES6.
With regards to the used clients, we can't upgrade to the latest version of
`RestHighLevelClient`. That's because Elasticsearch has changed the license
from version 7.11 onwards to the Apache-incompatible SSPL license [2]. What
needs to happen is that the ES connector is refactored to use the new
Elasticsearch Java Client [3] which is still available under Apache 2.0
license in order to add support for ES8 clusters.
For now, we need to make Elasticsearch changes to both the Flink main repo
and the external one. Hopefully we can drop the ES connector from Flink's
repo with Flink 1.17.
Best regards,
Martijn
[1] https://www.elastic.co/support/eol
[2] https://www.elastic.co/pricing/faq/licensing
[3] https://github.com/elastic/elasticsearch-java
Op do 11 aug. 2022 om 16:45 schreef Niels Basjes <Ni...@basjes.nl>:
> Hi,
>
> I was building something that puts data in ES and found that the current
> connector is not compatible with an ES 8 cluster.
>
> I found that
> - The currently employed `RestHighLevelClient` has been deprecated by
> Elastic .
> - There is an `ApiCompatibilityMode` you can use in the latest ES 7
> RestHighLevelClient to connect to an ES 8 cluster (works on my machine).
> - Between the 7.10 (used now) and the 7.17 (latest) they have introduced in
> addition to the ApiCompatibilityMode also several breaking API changes by
> changing the packages of several classes we use.
> - If you update to the latest 7x then the
> `flink-connector-elasticsearch-base` can no longer be shared between ES 6
> and ES 7 simply because of these package changes.
> - In 2022 there was still an ES 6 release but according to
> https://www.elastic.co/support/eol it is now EOL.
>
> So I now see several ways forward:
> 1) We drop ES6, update ES7 and use this compatibility mode as a quick
> solution.
> 2) We keep ES6, duplicating the `flink-connector-elasticsearch-base` (into
> a 6.x and 7.x version with only these few classes being different), update
> ES7 and use this compatibility mode as a quick solution.
>
> Later on someone should then create a connector based on the new
> ElasticSearch 8 Java client API.
>
> See also:
>
> https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
>
> https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/master/migrate-hlrc.html
>
> My question is
> - Which do we do?
> - I vote for the "We drop ES6, update ES7 and use this compatibility
> mode"
> - Where do we do it?
> - We have these connectors in both the 'flink' and
> 'flink-connector-elasticsearch' repos. Do we update only the new one? Or
> both?
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>