You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "pjfanning (via GitHub)" <gi...@apache.org> on 2026/09/09 13:16:54 UTC
[PR] Elasticsearch: support Elasticsearch 8 and 9 [pekko-connectors]
pjfanning opened a new pull request, #1934:
URL: https://github.com/apache/pekko-connectors/pull/1934
### Motivation
The connector only knew about `ApiVersion.V5` and `V7`, so users on Elasticsearch 8 or 9 had to declare `V7` and hope. Neither server line was covered by any test.
### Modification
**Main source**
- Added `V8` and `V9` to `ApiVersion`, with matching `ElasticsearchParams.V8` / `V9` factories that take an index name only (mapping types are gone).
- Both dispatch to the existing V7 REST implementation in `ElasticsearchSimpleFlowStage` and `ElasticsearchSourceStage`. That implementation is already type-less: `RestBulkApiV7` never emits `_type`, the search endpoint is `/<index>/_search`, and hit parsing reads only `_id`, `_source` and `_version` — all of which survive the removal of mapping types.
**Tests**
- New `elasticsearch8` (8.19.21, port 9205) and `elasticsearch9` (9.5.3, port 9206) compose services, with `xpack.security.enabled=false` and a 512m heap so the two extra nodes do not blow up the CI footprint; both started by the `elasticsearch` CI job.
- Both Scala and Java `constructElasticsearchParams` helpers learn the new versions.
- `ElasticsearchSpec` runs the full `ElasticsearchConnectorBehaviour` suite against each new server, and `ElasticsearchParameterizedTest` covers ports 9205/9206.
- `createStrictMapping` declares mappings without the `_doc` type and without `include_type_name` for V8/V9 — Elasticsearch 8 removed both.
- The two strict-mapping error assertions now match the stable part of the reason instead of the trailing `within [_doc]`, which is server-internal and varies by major version.
**Docs**
- `elasticsearch.md` documents `V8`/`V9` in both settings tables and in the supported-API-versions section.
No existing signature changed; `elasticsearch/mimaReportBinaryIssues` is clean.
### Result
Elasticsearch 8 and 9 users can name their actual server version, and both lines are exercised by the full connector behaviour suite in Scala and by the parameterized test in Java.
`V8` and `V9` behave identically today — they are separate constants so the setting keeps its meaning if the two lines diverge, the same way `OpensearchApiVersion.V1` maps onto the V7 shape.
### Tests
- `sbt "elasticsearch/Test/compile"` — success
- `sbt "++3.3.8 elasticsearch/Test/compile"` — success (Scala 3)
- `sbt "elasticsearch/mimaReportBinaryIssues"` — success, no issues
- `sbt "elasticsearch/scalafmtAll" "elasticsearch/javafmtAll"` — success
- Integration run not done locally (no Docker on this machine); relying on the `elasticsearch` CI job to exercise 8.19.21 and 9.5.3.
### References
None — Elasticsearch 8 and 9 server support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Elasticsearch: support Elasticsearch 8 and 9 [pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #1934:
URL: https://github.com/apache/pekko-connectors/pull/1934#issuecomment-5603249818
Merged the test-cleanup work from #1935 into this branch, so this PR is now self-contained and does not need #1935 to land first.
Why it was needed: CI runs `sbt "+elasticsearch/test"`, so the whole suite runs **twice** against the same servers — once on Scala 2.13.18, once on 3.3.8. The previous run's first pass was fully green; the second pass failed 14 Scala and 2 Java tests purely on state the first pass left behind, because `DELETE /_all` is refused from Elasticsearch 8 on (`action.destructive_requires_name` defaults to true) and the old cleanup discarded the response. The failures were all leftover-state signatures — `Object is missing required member 'price'` (Scala reading Java-written documents), `version conflict, document already exists (current version [2])`, `expected: <1> but was: <2>`.
The merged fix lists indices via `_cat/indices` and deletes them by explicit name, and checks both responses so a cleanup that stops working fails loudly.
One thing the automatic merge got wrong and I corrected: it left the V8 and V9 cleanup on the `_all` path — the two servers where it does not work — while V5 and V7 moved to the helper. All four now go through `deleteAllIndices`.
Verified locally: compile on 2.13.18 and 3.3.8 (no warnings), `elasticsearch/mimaReportBinaryIssues` clean, scalafmt/javafmt clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Elasticsearch: support Elasticsearch 8 and 9 [pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning merged PR #1934:
URL: https://github.com/apache/pekko-connectors/pull/1934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org
Re: [PR] Elasticsearch: support Elasticsearch 8 and 9 [pekko-connectors]
Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on PR #1934:
URL: https://github.com/apache/pekko-connectors/pull/1934#issuecomment-5727729095
@Philippus this PR is bigger because the extra elasticsearch servers mean that we need to test other API versions and the biggest issue was having all the extra tests running in parallel has caused tests to clash with each other so the tests needed to be modified to avoid clashing.
I have merged the OpenSearch PR and rebased this one to include that.
The MIMA checks failed because of an OOM. A recent upgrade to sbt-mima-plugin seems to have led to this taking a lot more heap memory to run. I am rerunning that part of the build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org