You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Chris Herrera <ch...@gmail.com> on 2017/07/07 14:18:36 UTC

Interest in a time series simulator processor for NiFi

Hi All,

I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.

Regards,
Chris 

[1] - https://github.com/cetic/TSimulus

Aw: Re: Interest in a time series simulator processor for NiFi

Posted by Uwe Geercken <uw...@web.de>.
Just my two cents...

I have created a processor that generates CSV data.
 
You can see an example here: http://datamelt.weebly.com/blog/nifi-processor-generatedata
 
It generates data based on word lists, regular expressions or purely random. Date data can be generated in a way that the columns generated make sense. So e.g. a real date 2017-07-08 is generated randomly and you can have columns for year, month, day, weekday, week of year, etc that corespond to this date.
 
And when you use word lists, it will generate data for catagories such as airlines, seasons, car manufacturers, etc.
 
Hope this helps,
 
Uwe
 

Gesendet: Freitag, 07. Juli 2017 um 17:21 Uhr
Von: "Lee Laim (leelaim)" <le...@micron.com>
An: "dev@nifi.apache.org" <de...@nifi.apache.org>
Betreff: Re: Interest in a time series simulator processor for NiFi
Yes. It will be very useful in testing flows that will run in non-perfect environments. This will be much more elegant than using several generate flow files in parallel with different settings.

Thanks!
Lee

leelaim@micron.com



> On Jul 7, 2017, at 8:44 AM, Matt Burgess <ma...@gmail.com> wrote:
>
> Chris,
>
> This sounds great! IMO Realistic data generation in all forms is a great addition, looking forward to your contribution!
>
> Regards,
> Matt
>
>
>> On Jul 7, 2017, at 10:18 AM, Chris Herrera <ch...@gmail.com> wrote:
>>
>> Hi All,
>>
>> I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.
>>
>> Regards,
>> Chris
>>
>> [1] - https://github.com/cetic/TSimulus[https://github.com/cetic/TSimulus]

Re: Interest in a time series simulator processor for NiFi

Posted by "Lee Laim (leelaim)" <le...@micron.com>.
Yes.   It will be very useful in testing flows that will run in non-perfect environments.   This will be much more elegant than using several generate flow files in parallel with different settings.  

Thanks!
Lee

leelaim@micron.com



> On Jul 7, 2017, at 8:44 AM, Matt Burgess <ma...@gmail.com> wrote:
> 
> Chris,
> 
> This sounds great! IMO Realistic data generation in all forms is a great addition, looking forward to your contribution!
> 
> Regards,
> Matt
> 
> 
>> On Jul 7, 2017, at 10:18 AM, Chris Herrera <ch...@gmail.com> wrote:
>> 
>> Hi All,
>> 
>> I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.
>> 
>> Regards,
>> Chris 
>> 
>> [1] - https://github.com/cetic/TSimulus

Re: Interest in a time series simulator processor for NiFi

Posted by Matt Burgess <ma...@gmail.com>.
Chris,

This sounds great! IMO Realistic data generation in all forms is a great addition, looking forward to your contribution!

Regards,
Matt


> On Jul 7, 2017, at 10:18 AM, Chris Herrera <ch...@gmail.com> wrote:
> 
> Hi All,
> 
> I am trying to gauge interest in a processor I have written that generates realistic time series data. I used the excellent GenerateFlowFile processor for a long time for load testing, etc..., however, I needed something that mirrored more the semantics of a sensor, and more importantly generated data like one. I have wrapped another Apache 2 licensed project TSimulus[1] that generates realistic time series data driven by a JSON configuration file. The idea is that this can be used to simulate a more IoT-like scenario. More than anything I am just trying to see if this is something that has value outside of my little world.
> 
> Regards,
> Chris 
> 
> [1] - https://github.com/cetic/TSimulus