[jira] [Comment Edited] (NIFI-7990) PutElasticsearch/RecordHttp processors should support Elasticsearch Data Streams

["Chris Sampson (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239798#comment-17239798 ] 

Chris Sampson edited comment on NIFI-7990 at 11/27/20, 6:31 PM:
----------------------------------------------------------------

Seems sensible to address NIFI-7374 at the same time - the mapped {{id}} field could be retained within the Record in the same way the mapped {{@timestamp}} field could be retained.

By default, the existing behaviour of removing the mapped field should be kept.


was (Author: chris s):
Seems sensible to address NIFI-7474 at the same time - the mapped {{id}} field could be retained within the Record in the same way the mapped {{@timestamp}} field could be retained.

By default, the existing behaviour of removing the mapped field should be kept.

> PutElasticsearch/RecordHttp processors should support Elasticsearch Data Streams
> --------------------------------------------------------------------------------
>
>                 Key: NIFI-7990
>                 URL: https://issues.apache.org/jira/browse/NIFI-7990
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.11.4, 1.12.1
>            Reporter: Chris Sampson
>            Assignee: Chris Sampson
>            Priority: Minor
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> PUT Elasticsearch should support the new [Elasticsearch Data Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html#add-documents-to-a-data-stream] (new in Elasticsearch 7.9).
> NIFI-7474 will allow these processors to submit {{create}} operations via the _bulk API (which is a large part of the requirement).
> However, Data Streams require an {{@timestamp}} field to be provided in each document, but this field name is illegal in [Avro schemas|http://avro.apache.org/docs/1.8.2/spec.html#names] due to the leading {{@}}. The Record-based processors should therefore allow for the injection of this field into the JSON being sent to Elasticsearch - this could be based upon an existing field within the FlowFile and be identified by a property on the processor (e.g. like the {{_id}} field can be specified using Record Path).
> Optionally, the processor allow for the field used as the {{@timestamp}} field to be removed from the data being sent to Elasticsearch (i.e. rename the existing field *or* duplicate it depending upon property settings). Such field transformation should also take the timestamp format settings into account(e.g. if a {{Long}} epoch millisecond value is to be converted to a formatted date/time {{String}}).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)