You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Chris Sampson (Jira)" <ji...@apache.org> on 2024/05/07 22:13:00 UTC
[jira] [Commented] (NIFI-10948) QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer Elasticsearch versions)
[ https://issues.apache.org/jira/browse/NIFI-10948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844461#comment-17844461 ]
Chris Sampson commented on NIFI-10948:
--------------------------------------
I tested the REST API based Elasticsearch components of NiFi against AWS OpenSearch after upgrading the Elasticsearch (low-level) REST Client library to the latest version (8.13.3) as part of NIFI-11686.
Aside from the lack of Point in Time queries (an Elastic XPack feature that provides an improved pagination approach over the older Scroll functionality), things seemed to work without issue.
Document types have been deprecated in Elasticsearch since long before AWS forked the libraries to create OpenSearch. Use of OS without Types works fine if seems, and I imagine would be the recommended approach for OS use as it is in Elasticsearch (ES 8 no longer supports Types at all, NiFi allows for Types but discourages their use).
So I'd suggest this isn't something that will be fixed in the NiFi 1 branch, and works fine in the NiFi 2 branch for both OS & ES.
Work to create OS specific implementations could be raised as separate tickets to include:
* use of the AWS OpenSearch client libraries (rather than Elastic libraries)
* Allowing AWS-style auth, e.g. assume IAM roles (hopefully via the existing NiFi AWS controller services), rather than simple BASIC auth (username & password)
> QueryElasticsearchHTTP doesnt work on Opensearch (and possibly newer Elasticsearch versions)
> --------------------------------------------------------------------------------------------
>
> Key: NIFI-10948
> URL: https://issues.apache.org/jira/browse/NIFI-10948
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 1.15.3
> Environment: Nifi 1.15.3 on centos machine
> Reporter: David Vassallo
> Priority: Minor
> Attachments: image-2022-12-05-16-34-20-513.png
>
>
> The QueryElasticsearchHTTP processor will return a "Null Pointer Exception" when running agasint Opensearch and probably newer versions of Elasticsearch. I tracked down the null pointer to this line:
>
> [https://github.com/apache/nifi/blob/0ebc6d31489e975dcbbe078fa572332ef8ffa9e2/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/QueryElasticsearchHttp.java#L418]
>
> The processor is looking for the "_type" field which is no longer in the right place. In opensearch v 1.x and above (possibly Elasticsearch too, but I dont have the means to test this), the field has moved to be a child of the "@metadata" field:
> !image-2022-12-05-16-34-20-513.png!
>
> If you try to circumvent this issue by populating the "Type" property of the processor, you will get an HTTP 400 Bad Message response since the processor builds a URL with the "size" and "from" arguments as part of the URL which are not supported by Opensearch/Elasticsearch in more recent versions, so you are forced to leave this property empty but then run into the null problem described above
--
This message was sent by Atlassian Jira
(v8.20.10#820010)