You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Chris Sampson (Jira)" <ji...@apache.org> on 2022/11/11 15:49:00 UTC
[jira] [Commented] (NIFI-7449) ScrollElasticsearchHttp processor fails against ES 7.6 with error 400 Field expansion matches too many fields, limit: 1024
[ https://issues.apache.org/jira/browse/NIFI-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632433#comment-17632433 ]
Chris Sampson commented on NIFI-7449:
-------------------------------------
The ScrollElasticsearchHttp processor is now deprecated.
Recommendation would be to use the newer SearchElasticsearch or PaginatedJsonQueryElasticsearch processors instead as they implement the same functionality but using better APIs and the Elasticsearch REST Client libraries.
All queries through the ElasticSearchClient Controller Service use POST rather than GET, so this error should be avoided.
> ScrollElasticsearchHttp processor fails against ES 7.6 with error 400 Field expansion matches too many fields, limit: 1024
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-7449
> URL: https://issues.apache.org/jira/browse/NIFI-7449
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.9.0, 1.9.2
> Reporter: Aquilino Viveiros
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Using Elasticsearch 7.6 (probably true for any 7.x version) and using the processor to scroll over the data Nifi receives a 400 error from ES with the following content => *Field expansion matches too many fields, limit: 1024*
>
> Precondition: Index documents and mapping must have fields over the limit above.
>
> When initiating the scroll, the Nifi processor (1) uses a GET with a query parameter as follow:
> *[http://localhost:9200/index/_search?q=type:*]*
> Which causes the error above. Removing the q parameter eliminates the error
>
> Few ideas to fix it
> 1) Make query parameter optional (keep GET)
> 2) Refactor existing code to use POST instead of GET.
> 3) Introduce a different property for a Query to be used as a POST (does not trigger the error above)
>
> Tested on
> Elasticsearch 7.6
> Nifi 1.9.0 and 1.9.2
>
> 1) [https://github.com/apache/nifi/blob/9608fe33fa4a3f39ee6ddca54184ccc85ef1a163/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ScrollElasticsearchHttp.java#L284]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)