[jira] [Commented] (NIFI-5172) PutElasticSearchRecord does to fail individual recods

["Joseph Percivall (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-5172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700967#comment-16700967 ] 

Joseph Percivall commented on NIFI-5172:
----------------------------------------

I ran into this issue today and I assumed this to be a "bug" given that it differs from the logic in for all the other record based processors and the PutElasticsearchHttp processor most of the core logic was copied from(including the comment stating the "correct" logic[1]). 

Addressing [~mike.thomsen]'s comment, at least on the HTTP endpoint, ES will tell you which one failed as that's the logic currently in PutElasticsearchHttp[2].

Talking with [~mattyb149], the logic for adding it would be something along the lines of a merging of the error handling of PutElasticsearchHttp and with the record creation of SplitRecord[3].

I would argue that this is higher than a minor priority as behaves opposite what we've trained users to expect and potentially leads to duplicate data being ingested.

[1] https://issues.apache.org/jira/browse/NIFI-5172
[2] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttp.java#L345
[3] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/SplitRecord.java#L144

> PutElasticSearchRecord does to fail individual recods
> -----------------------------------------------------
>
>                 Key: NIFI-5172
>                 URL: https://issues.apache.org/jira/browse/NIFI-5172
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.6.0
>            Reporter: Juan C. Sequeiros
>            Priority: Minor
>
> My observation and not sure if working as expected but when I send my output from MergeRecord ( set to 10000 ) max number of records and one of those records has an invalid timestamp "bogusdata" value ES rejects it, rightly so since on ES we have a schema template more granular and is expecting timestamp as type "date".
>  
> From USER forums Matt Burgess:
> Yes the current behavior is to move the entire input flowfile to
> failure if any errors occur. Some other record-aware processors create
> separate flow files for failed and successful records, but
> PutElasticsearchHttpRecord does not (yet) do that. Please feel free to
> write a Jira for this improvement.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)