How to find average of two lines in NiFi?

[prabhu Mahendran <pr...@gmail.com>]

i need to find average of two values in seperate lines.

My Csv file loo like this.,

Name,ID,Marks
Mahi,1,90
Mahi,1,90


Andy,2,100
Andy,2,100
Now i need to store that average of 2 marks in database. "Average" column
should add two marks and divide with 2 and store that result in SQL query

Table:

Name,ID,Average
Mahi,2,90
Andy,2,100
Is this possible to find average of two value's in seperate rows using NiFi?

Re: How to find average of two lines in NiFi?

[Andy LoPresto <al...@apache.org>]

Prabhu,

I don’t think NiFi is the right tool to do the complete task (fine for CSV parsing & splitting), but you should look at writing a script or using Spark, etc. to do the data operations. Any NiFi-native solution will be brittle and won’t scale well.

That said, I did answer how you could do this in NiFi only on your StackOverflow question [1].

However, you could certainly use a combination of SplitText processors to get the proper data into individual flowfiles (i.e. all Mahi rows in one, all Andy rows in another). Once you have a record that looks like:

Andy,1,85
Andy,1,95
you can use ExtractText with regular expressions to get 85 and 95 into attributes marks.1 and marks.2 (a good example of where scaling will break down -- doing this with 2 rows is easy; doing this with 100k is ridiculous). You can then use UpdateAttribute with the Expression Language to calculate the average of those two attributes (convert toNumber() first) and populate a third attribute marks.average (either through chaining plus() and divide() functions or with the math advanced operation <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#math> (uses Java Reflection)). Once you have the desired result in an attribute, use ReplaceText to update the flowfile content, and MergeContent to merge the individual flowfiles back into a single instance.




[1] http://stackoverflow.com/a/42381999/70465


Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Feb 21, 2017, at 5:20 AM, prabhu Mahendran <pr...@gmail.com> wrote:
> 
> i need to find average of two values in seperate lines.
> 
> My Csv file loo like this.,
> 
> Name,ID,Marks
> Mahi,1,90
> Mahi,1,90
> 
> 
> Andy,2,100
> Andy,2,100
> Now i need to store that average of 2 marks in database. "Average" column should add two marks and divide with 2 and store that result in SQL query
> 
> Table:
> 
> Name,ID,Average
> Mahi,2,90
> Andy,2,100
> Is this possible to find average of two value's in seperate rows using NiFi?
>