Attributes To JSON question

[DAVID SMITH <da...@btinternet.com.INVALID>]

Hi
I am using the attributesToJSON processor to create a JSON flowfile which I then send to Elastic, I have noticed that some of my attributes which ar numbers such as file size always come out of the attributesToJSON processor as string values (ie with double quotes around them), therefore when I send my JSON string to Elastic it indexes all the fields as Strings and therefore won't let me do a cumulative count of all filesizes.I have tried to cast my filesize toNumber just before the attributes go into the attributesToJSON processor and it makes no difference, do you know how I can correct this?
Many thanksDave

Re: Attributes To JSON question

[Chris Sampson <ch...@naimuri.com.INVALID>]

You could probably use a JOLT Transform on the NiFi side (although I’m no JOLT expert).

From an Elasticsearch point of view, if you create an index template [1] for the destination index with mappings to store the fields as numbers, I think Elasticsearch will do the conversion for you (pretty sure I’m doing this in places).


[1] https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html <https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html>


Cheers,

-- 
Chris Sampson

t: +44 7867 843675
e: chris.sampson@naimuri.com

> On 3 Jun 2020, at 14:12, DAVID SMITH <da...@btinternet.com.INVALID> wrote:
> 
> Hi
> I am using the attributesToJSON processor to create a JSON flowfile which I then send to Elastic, I have noticed that some of my attributes which ar numbers such as file size always come out of the attributesToJSON processor as string values (ie with double quotes around them), therefore when I send my JSON string to Elastic it indexes all the fields as Strings and therefore won't let me do a cumulative count of all filesizes.I have tried to cast my filesize toNumber just before the attributes go into the attributesToJSON processor and it makes no difference, do you know how I can correct this?
> Many thanksDave

Re: Attributes To JSON question

[Mark Payne <ma...@hotmail.com>]

David,

Attributes are always Strings. I don’t think you can change how they are represented in AttributesToJSON.
My recommendation would be to not use that processor and instead use ReplaceText with a Replacement Strategy as Always. Then you can build the JSON to look however you want by doing something like:

{
  “abc”: “xyz”,
  “filename": “${filename}”,
  “size”: ${fileSize},
  “otherAttribute”: “${someAttribute}”
}

Thanks
-Mark



> On Jun 3, 2020, at 9:12 AM, DAVID SMITH <da...@btinternet.com.INVALID> wrote:
> 
> Hi
> I am using the attributesToJSON processor to create a JSON flowfile which I then send to Elastic, I have noticed that some of my attributes which ar numbers such as file size always come out of the attributesToJSON processor as string values (ie with double quotes around them), therefore when I send my JSON string to Elastic it indexes all the fields as Strings and therefore won't let me do a cumulative count of all filesizes.I have tried to cast my filesize toNumber just before the attributes go into the attributesToJSON processor and it makes no difference, do you know how I can correct this?
> Many thanksDave