You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Esa Heikkinen <es...@student.tut.fi> on 2018/02/08 15:28:12 UTC
CEP for time series in csv-file
Hi
I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and "attributes" of event.
I'd want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP 5) writes results
Do you have any hints or examples how to do that ?
By the way, what kind of time stamp should be in csv-file ?
RE: CEP for time series in csv-file
Posted by Esa Heikkinen <es...@student.tut.fi>.
Hi
I have little problem with this example about Scala maven. I created new project by command: “mvn archetype:generate …” as stated in the instructions.
What is the correct build command ? “mvn package –Pbuild-jar” in same directory with pom.xml ?
For example, it does not generate WordCount.jar. I only found target/art-1.0-SNAPSHOT.jar.
What is the correct run command of flink for WordCount in this case ?
Can I run it without starting of Flink (start-local.sh) and browser-interface (in port 8081)?
What did I do wrong ?
From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: Saturday, February 10, 2018 1:07 PM
To: Esa Heikkinen <es...@student.tut.fi>
Cc: Timo Walther <tw...@apache.org>; user@flink.apache.org
Subject: Re: CEP for time series in csv-file
Hi,
I'm not aware of any example project that ticks all your requirements. As you said, too many combinations...
Flink uses Maven. So, most examples provided by committers will be based on Maven.
I would recommend to start with the Scala Maven quickstart archetypes [1].
Next, I'd implement a source function that reads your data from a single file. I've given some pointers in a previous mail.
Once you can read the data, I'd add the CEP library.
Best, Fabian
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/scala_api_quickstart.html#maven
2018-02-09 15:35 GMT+01:00 Esa Heikkinen <es...@student.tut.fi>>:
Hi
Thanks for the hints, but I am still very interested about simple working example with combination: sbt-project, scala, csv-file reading and cep processing. I have did not exactly find something like that. It would help me a lot.
It takes lot of time to learn and test many possible code combinations.. Too many “moving” parts..
For example “huge” amount of different “imports” and where I can find how use them and so on ?
I did not find strict “reference” guide. For example for readCsvFile(). Or should I look it from code ?
By the way what is better to use maven or sbt ? It seems most of examples use maven, but I haven’t got maven to work properly (yet) ..
Best Regards
Esa
From: Timo Walther [mailto:twalthr@apache.org<ma...@apache.org>]
Sent: Thursday, February 8, 2018 7:23 PM
To: user@flink.apache.org<ma...@flink.apache.org>
Subject: Re: CEP for time series in csv-file
You can also take a look at the Flink training from data Artisans and the code examples there. They also use CEP and basically read also from a file:
http://training.data-artisans.com/exercises/CEP.html
Regards,
Timo
Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:
Hi Esa,
I think the best place to start is the documentation available at the flink website.
Some pointers are the following:
CEP documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update
Cheers,
Kostas
On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <es...@student.tut.fi>> wrote:
Hi
I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.
I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP 5) writes results
Do you have any hints or examples how to do that ?
By the way, what kind of time stamp should be in csv-file ?
Re: CEP for time series in csv-file
Posted by Fabian Hueske <fh...@gmail.com>.
Hi,
I'm not aware of any example project that ticks all your requirements. As
you said, too many combinations...
Flink uses Maven. So, most examples provided by committers will be based on
Maven.
I would recommend to start with the Scala Maven quickstart archetypes [1].
Next, I'd implement a source function that reads your data from a single
file. I've given some pointers in a previous mail.
Once you can read the data, I'd add the CEP library.
Best, Fabian
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/scala_api_quickstart.html#maven
2018-02-09 15:35 GMT+01:00 Esa Heikkinen <es...@student.tut.fi>:
> Hi
>
>
>
> Thanks for the hints, but I am still very interested about simple working
> example with combination: sbt-project, scala, csv-file reading and cep
> processing. I have did not exactly find something like that. It would help
> me a lot.
>
>
>
> It takes lot of time to learn and test many possible code combinations..
> Too many “moving” parts..
>
> For example “huge” amount of different “imports” and where I can find how
> use them and so on ?
>
> I did not find strict “reference” guide. For example for readCsvFile(). Or
> should I look it from code ?
>
>
>
> By the way what is better to use maven or sbt ? It seems most of examples
> use maven, but I haven’t got maven to work properly (yet) ..
>
>
>
> Best Regards
>
> Esa
>
>
>
> *From:* Timo Walther [mailto:twalthr@apache.org]
> *Sent:* Thursday, February 8, 2018 7:23 PM
> *To:* user@flink.apache.org
> *Subject:* Re: CEP for time series in csv-file
>
>
>
> You can also take a look at the Flink training from data Artisans and the
> code examples there. They also use CEP and basically read also from a file:
>
> http://training.data-artisans.com/exercises/CEP.html
>
> Regards,
> Timo
>
>
> Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:
>
> Hi Esa,
>
>
>
> I think the best place to start is the documentation available at the
> flink website.
>
>
>
> Some pointers are the following:
>
>
>
> CEP documentation: https://ci.apache.org/projects/flink/
> flink-docs-release-1.4/dev/libs/cep.html
>
>
>
> Blog post with CEP example: https://data-artisans.com/blog/complex-
> event-processing-flink-cep-update
>
>
>
> Cheers,
>
> Kostas
>
>
>
> On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <es...@student.tut.fi>
> wrote:
>
>
>
> Hi
>
>
>
> I have cvs-file(s) that contain an event in every row and first column is
> time stamp of event. Rest of columns are data and “attributes” of event.
>
>
>
> I’d want to write simple Scala code that: 1) reads data of csv-file 2)
> converts data of csv-file compatible for CEP 3) sets pattern for CEP 4)
> Runs CEP 5) writes results
>
>
>
> Do you have any hints or examples how to do that ?
>
>
>
> By the way, what kind of time stamp should be in csv-file ?
>
>
>
>
>
RE: CEP for time series in csv-file
Posted by Esa Heikkinen <es...@student.tut.fi>.
Hi
Thanks for the hints, but I am still very interested about simple working example with combination: sbt-project, scala, csv-file reading and cep processing. I have did not exactly find something like that. It would help me a lot.
It takes lot of time to learn and test many possible code combinations.. Too many “moving” parts..
For example “huge” amount of different “imports” and where I can find how use them and so on ?
I did not find strict “reference” guide. For example for readCsvFile(). Or should I look it from code ?
By the way what is better to use maven or sbt ? It seems most of examples use maven, but I haven’t got maven to work properly (yet) ..
Best Regards
Esa
From: Timo Walther [mailto:twalthr@apache.org]
Sent: Thursday, February 8, 2018 7:23 PM
To: user@flink.apache.org
Subject: Re: CEP for time series in csv-file
You can also take a look at the Flink training from data Artisans and the code examples there. They also use CEP and basically read also from a file:
http://training.data-artisans.com/exercises/CEP.html
Regards,
Timo
Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:
Hi Esa,
I think the best place to start is the documentation available at the flink website.
Some pointers are the following:
CEP documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update
Cheers,
Kostas
On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <es...@student.tut.fi>> wrote:
Hi
I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.
I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP 5) writes results
Do you have any hints or examples how to do that ?
By the way, what kind of time stamp should be in csv-file ?
Re: CEP for time series in csv-file
Posted by Timo Walther <tw...@apache.org>.
You can also take a look at the Flink training from data Artisans and
the code examples there. They also use CEP and basically read also from
a file:
http://training.data-artisans.com/exercises/CEP.html
Regards,
Timo
Am 2/8/18 um 6:09 PM schrieb Kostas Kloudas:
> Hi Esa,
>
> I think the best place to start is the documentation available at the
> flink website.
>
> Some pointers are the following:
>
> CEP documentation:
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html
>
> Blog post with CEP example:
> https://data-artisans.com/blog/complex-event-processing-flink-cep-update
>
> Cheers,
> Kostas
>
>> On Feb 8, 2018, at 4:28 PM, Esa Heikkinen
>> <esa.heikkinen@student.tut.fi <ma...@student.tut.fi>>
>> wrote:
>>
>> Hi
>> I have cvs-file(s) that contain an event in every row and first
>> column is time stamp of event. Rest of columns are data and
>> “attributes” of event.
>> I’d want to write simple Scala code that: 1) reads data of csv-file
>> 2) converts data of csv-file compatible for CEP 3) sets pattern for
>> CEP 4) Runs CEP 5) writes results
>> Do you have any hints or examples how to do that ?
>> By the way, what kind of time stamp should be in csv-file ?
>
Re: CEP for time series in csv-file
Posted by Kostas Kloudas <k....@data-artisans.com>.
Hi Esa,
I think the best place to start is the documentation available at the flink website.
Some pointers are the following:
CEP documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/libs/cep.html>
Blog post with CEP example: https://data-artisans.com/blog/complex-event-processing-flink-cep-update <https://data-artisans.com/blog/complex-event-processing-flink-cep-update>
Cheers,
Kostas
> On Feb 8, 2018, at 4:28 PM, Esa Heikkinen <es...@student.tut.fi> wrote:
>
> Hi
>
> I have cvs-file(s) that contain an event in every row and first column is time stamp of event. Rest of columns are data and “attributes” of event.
>
> I’d want to write simple Scala code that: 1) reads data of csv-file 2) converts data of csv-file compatible for CEP 3) sets pattern for CEP 4) Runs CEP 5) writes results
>
> Do you have any hints or examples how to do that ?
>
> By the way, what kind of time stamp should be in csv-file ?