You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jean-Sebastien Vachon <js...@brizodata.com> on 2018/07/16 18:35:50 UTC
Send data to ElasticSearch
Hi all,
I'm sending the result of my flow to ElasticSearch for indexing but ES fails to recognize my timestamp field for a date based field.
What is the recommended approach to fix this? I thought about adding some steps to my flow that would check if a mapping exists for my index and the send the mappings to ES if required. In case where the Mapping do not exists, ES will return something like:
{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "my_index",
"index_uuid": "_na_",
"index": "my_index"
}
],
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id": "my_index",
"index_uuid": "_na_",
"index": "my_index"
},
"status": 404
}
So, If I fetch the response and look out for "status:404", I would know whether or not I need to create the mappings. In this case, I would call a PutElasticSearchHttp with the proper mappings and everything should be fine. However, this seems like a lot of trouble for an operation that will most likely failed only once.
Are there any other ways of dealing with this?
Thanks
--
Jean-Sébastien Vachon
vachonjs@gmail.com<ma...@brizodata.com>
jsvachon@brizodata.com<ma...@brizodata.com>
www.brizodata.com<http://www.brizodata.com/>
RE: Send data to ElasticSearch
Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
Thanks all. I will look into these
From: Matt Burgess <ma...@gmail.com>
Sent: July 16, 2018 3:42 PM
To: users@nifi.apache.org
Subject: Re: Send data to ElasticSearch
There’s a PutElasticsearchHttpRecord that should give you the best of both worlds, the Record API to convert data types and using the REST API.
On Jul 16, 2018, at 2:59 PM, Mike Thomsen <mi...@gmail.com>> wrote:
With PutElasticsearchHttp, you'll have to define the field in advance as a date one because it's not a record-aware processor (the Record API has data types that can solve this problem). Once you've defined your date fields, an ISO8601 date string will suffice. If you have more than one index you'll need to write to, and the date field names are the same, I'd recommend writing an ElasticSearch index template to apply the rule to all indexes that match the supplied name pattern.
On Mon, Jul 16, 2018 at 2:36 PM Jean-Sebastien Vachon <js...@brizodata.com>> wrote:
Hi all,
I’m sending the result of my flow to ElasticSearch for indexing but ES fails to recognize my timestamp field for a date based field.
What is the recommended approach to fix this? I thought about adding some steps to my flow that would check if a mapping exists for my index and the send the mappings to ES if required. In case where the Mapping do not exists, ES will return something like:
{
"error": {
"root_cause": [
{
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id<http://resource.id>": "my_index",
"index_uuid": "_na_",
"index": "my_index"
}
],
"type": "index_not_found_exception",
"reason": "no such index",
"resource.type": "index_or_alias",
"resource.id<http://resource.id>": "my_index",
"index_uuid": "_na_",
"index": "my_index"
},
"status": 404
}
So, If I fetch the response and look out for “status:404”, I would know whether or not I need to create the mappings. In this case, I would call a PutElasticSearchHttp with the proper mappings and everything should be fine. However, this seems like a lot of trouble for an operation that will most likely failed only once.
Are there any other ways of dealing with this?
Thanks
--
Jean-Sébastien Vachon
vachonjs@gmail.com<ma...@brizodata.com>
jsvachon@brizodata.com<ma...@brizodata.com>
www.brizodata.com<http://www.brizodata.com/>
Re: Send data to ElasticSearch
Posted by Matt Burgess <ma...@gmail.com>.
There’s a PutElasticsearchHttpRecord that should give you the best of both worlds, the Record API to convert data types and using the REST API.
> On Jul 16, 2018, at 2:59 PM, Mike Thomsen <mi...@gmail.com> wrote:
>
> With PutElasticsearchHttp, you'll have to define the field in advance as a date one because it's not a record-aware processor (the Record API has data types that can solve this problem). Once you've defined your date fields, an ISO8601 date string will suffice. If you have more than one index you'll need to write to, and the date field names are the same, I'd recommend writing an ElasticSearch index template to apply the rule to all indexes that match the supplied name pattern.
>
>> On Mon, Jul 16, 2018 at 2:36 PM Jean-Sebastien Vachon <js...@brizodata.com> wrote:
>> Hi all,
>>
>>
>>
>> I’m sending the result of my flow to ElasticSearch for indexing but ES fails to recognize my timestamp field for a date based field.
>>
>> What is the recommended approach to fix this? I thought about adding some steps to my flow that would check if a mapping exists for my index and the send the mappings to ES if required. In case where the Mapping do not exists, ES will return something like:
>>
>>
>>
>> {
>>
>> "error": {
>>
>> "root_cause": [
>>
>> {
>>
>> "type": "index_not_found_exception",
>>
>> "reason": "no such index",
>>
>> "resource.type": "index_or_alias",
>>
>> "resource.id": "my_index",
>>
>> "index_uuid": "_na_",
>>
>> "index": "my_index"
>>
>> }
>>
>> ],
>>
>> "type": "index_not_found_exception",
>>
>> "reason": "no such index",
>>
>> "resource.type": "index_or_alias",
>>
>> "resource.id": "my_index",
>>
>> "index_uuid": "_na_",
>>
>> "index": "my_index"
>>
>> },
>>
>> "status": 404
>>
>> }
>>
>>
>>
>> So, If I fetch the response and look out for “status:404”, I would know whether or not I need to create the mappings. In this case, I would call a PutElasticSearchHttp with the proper mappings and everything should be fine. However, this seems like a lot of trouble for an operation that will most likely failed only once.
>>
>>
>>
>> Are there any other ways of dealing with this?
>>
>>
>>
>> Thanks
>>
>> --
>>
>> Jean-Sébastien Vachon
>>
>>
>> vachonjs@gmail.com
>>
>> jsvachon@brizodata.com
>>
>> www.brizodata.com
>>
>>
Re: Send data to ElasticSearch
Posted by Mike Thomsen <mi...@gmail.com>.
Z should only be added to your date strings if it's UTC/Zulu. If, say, your
time zone is PST or EST, you are going to have inaccurate dates if you
don't specify the correct numeric offset for the timezone and just put a Z
on there.
On Mon, Jul 16, 2018 at 3:35 PM Vijay Chhipa <vc...@apple.com> wrote:
> For value of the date field, ensure that you have 'Z' at the end to
> indicate the timezone, even if your timezone is UTC. I ran into a similar
> issue working with ES HTTP API from Python
>
>
>
> On Jul 16, 2018, at 1:59 PM, Mike Thomsen <mi...@gmail.com> wrote:
>
> ISO8601
>
>
>
Re: Send data to ElasticSearch
Posted by Vijay Chhipa <vc...@apple.com>.
For value of the date field, ensure that you have 'Z' at the end to indicate the timezone, even if your timezone is UTC. I ran into a similar issue working with ES HTTP API from Python
> On Jul 16, 2018, at 1:59 PM, Mike Thomsen <mi...@gmail.com> wrote:
>
> ISO8601
Re: Send data to ElasticSearch
Posted by Mike Thomsen <mi...@gmail.com>.
With PutElasticsearchHttp, you'll have to define the field in advance as a
date one because it's not a record-aware processor (the Record API has data
types that can solve this problem). Once you've defined your date fields,
an ISO8601 date string will suffice. If you have more than one index you'll
need to write to, and the date field names are the same, I'd recommend
writing an ElasticSearch index template to apply the rule to all indexes
that match the supplied name pattern.
On Mon, Jul 16, 2018 at 2:36 PM Jean-Sebastien Vachon <
jsvachon@brizodata.com> wrote:
> Hi all,
>
>
>
> I’m sending the result of my flow to ElasticSearch for indexing but ES
> fails to recognize my timestamp field for a date based field.
>
> What is the recommended approach to fix this? I thought about adding some
> steps to my flow that would check if a mapping exists for my index and the
> send the mappings to ES if required. In case where the Mapping do not
> exists, ES will return something like:
>
>
>
> {
>
> "error": {
>
> "root_cause": [
>
> {
>
> "type": "index_not_found_exception",
>
> "reason": "no such index",
>
> "resource.type": "index_or_alias",
>
> "resource.id": "my_index",
>
> "index_uuid": "_na_",
>
> "index": "my_index"
>
> }
>
> ],
>
> "type": "index_not_found_exception",
>
> "reason": "no such index",
>
> "resource.type": "index_or_alias",
>
> "resource.id": "my_index",
>
> "index_uuid": "_na_",
>
> "index": "my_index"
>
> },
>
> "status": 404
>
> }
>
>
>
> So, If I fetch the response and look out for “status:404”, I would know
> whether or not I need to create the mappings. In this case, I would call a
> PutElasticSearchHttp with the proper mappings and everything should be
> fine. However, this seems like a lot of trouble for an operation that will
> most likely failed only once.
>
>
>
> Are there any other ways of dealing with this?
>
>
>
> Thanks
>
> --
>
> Jean-Sébastien Vachon
>
> vachonjs@gmail.com <js...@brizodata.com>
>
> jsvachon@brizodata.com
>
> www.brizodata.com
>
>
>