Re: Elasticsearch

[Sean Busbey <sb...@apple.com.INVALID>]

tl;dr: I agree we should remove the ES module for 1.10 and include a prominent release note.

It looks like the most recent version of the server artifacts we could rely on using without falling afoul of a licensing issue is v7.10.2. It has ~2 months of support time it looks like?

The ES upgrade docs[1] claim that a given major version of ES can only read indices from 1 major version behind. So I’d say if our client change requires a server upgrade by more than 1 major version that’s the equivalent of breaking change.

The high level client’s compatibility statement says it needs to be on the same major version as the ES server[2]. The high level client looks to follow the same license as the server. So we could get to v7.10.2 while using it, but there isn’t really a sustainable ongoing path.

The low level client is ALv2 licensed in current releases AFIACT[3]. Its docs don’t include any information about what version servers it is expected to work with, so probably we should assume a similar situation as the hight level client.

So I think that means we have no way to update the client without causing an incompatible change.

[1]: 
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
and
https://www.elastic.co/guide/en/elastic-stack/7.10/upgrading-elastic-stack.html#prepare-to-upgrade <https://www.elastic.co/guide/en/elastic-stack/7.10/upgrading-elastic-stack.html#prepare-to-upgrade>

[2]:
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.10/java-rest-high-compatibility.html <https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.10/java-rest-high-compatibility.html>

[3]:
https://github.com/elastic/elasticsearch/blob/v8.1.2/client/rest/build.gradle#L33
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.10/_license.html

> On Mar 29, 2022, at 3:05 AM, Ralph Goers <ra...@dslextreme.com> wrote:
> 
> I think addressing Elasticsearch may be the last major hurdle before 1.10.0 can be released.
> 
> Flume currently uses Elasticsearch 0.90.1. https://www.elastic.co/support/eol#maintenance-tables shows that version 1.0.x reached end of life in 2015 so obviously the version Flume is using has been unsupported from even before that.
> 
> The major goal of the 1.10.0 release has been to upgrade all the dependencies so that we aren’t using any that have known security vulnerabilities. From looking at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=elasticsearch I am quite sure there is at least one CVE that applies to 0.90.1.
> 
> Furthermore, this release is so old that I find it hard to imagine that anyone could still be using it.
> 
> In addition, flume-ng-elasticsearch-sink is including org.elasticsearch:elasticsearch as an optional dependency. I suspect at the time that there was no Java client. We really should have a required dependency on the Java client and the Elasticsearch dependency should be a test dependency, not optional.
> 
> I see 4 choices here for the 1.10.0 release:
> 
> 1. Do nothing. This is not ideal since the component is practically useless as it exists and has security vulnerabilities.
> 2. Drop the module in 1.10.0. This is obviously not backward compatible but any upgrade is going to break compatibility. We can defer re-implementing it until 2.0.
> 3. Upgrade to a supported version of the Java client (which I believe has a license that is compatible with the ASF). Again, this is not backward compatible. It would need to use ElasticSearch for testing. The latest versions of ElasticSearch use a license which is Category X so we will need to include something in the NOTICE file and in the user’s guide warning about the ElasticSearch license if the component is used. 
> 4. Upgrade to the latest version of https://opensearch.org/docs/latest/clients/java/. This would use OpenSearch instead of Elasticsearch and AFAIK would be incompatible with Elasticsearch. 
> 
> Of these options, my recommendation is to go with option 2. Once we modularize things in 2.0 we can implement support for both Elasticsearch and OpenSearch. We could also support Solr if desired.
> 
> Thoughts?  
> 
> Ralph