You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "Branham, Jeremy [IT]" <Je...@sprint.com> on 2017/07/07 20:29:55 UTC
Anomaly Detection with Flink-ML
Hello -
I'm working on an anomaly detector for some time series monitoring data.
I've setup an example project with Flink that reads from Kafka to get the monitoring data.
Unfortunately, I'm not sure what to do next.
The goal is to perform some clustering on the metric values that Flink is receiving and detect when the value[s] are anomalous.
I've got a DataStream<MetricDefinition> that I think needs to go through some pre-processing, like transforming it into a vector, but I'm not sure how to proceed.
The pojo[MetricDefinition] looks like this -
https://github.com/savantly-net/metric-schema/blob/master/src/main/java/net/savantly/metrics/schema/MetricDefinition.java
Can anyone point me in the right direction?
Thanks!
Jeremy D. Branham
Technology Architect - Sprint
O: +1 (972) 405-2970 | M: +1 (817) 791-1627
Jeremy.D.Branham@Sprint.com<ma...@Sprint.com>
#gettingbettereveryday
________________________________
This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
Re: Anomaly Detection with Flink-ML
Posted by "Branham, Jeremy [IT]" <Je...@sprint.com>.
Great information!
Thanks Jonas!
------ Original message------
From: Jonas Gröger
Date: Fri, Jul 7, 2017 4:12 PM
To: user@flink.apache.org;
Cc:
Subject:Re: Anomaly Detection with Flink-ML
Hello Jeremy,
it looks like what you are looking for is map (1 in, 1 out) / flatmap (1 in,
0-n out) for preprocessing on a single element basis as well as windows for
looking at related MetricDefinition elements calculating some result.
I suggest you look into Windows
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fdev%2Fwindows.html&data=02%7C01%7CJeremy.D.Branham%40sprint.com%7C66a475a8840f40aee51808d4c57cbe82%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636350587286223292&sdata=NoSCU2%2F3hcpLpwUCHdlFYrzA2dNraUuSMWlRUJ3H9vI%3D&reserved=0)
and basic transformations
(https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-release-1.3%2Fdev%2Fdatastream_api.html%23datastream-transformations&data=02%7C01%7CJeremy.D.Branham%40sprint.com%7C66a475a8840f40aee51808d4c57cbe82%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636350587286223292&sdata=KHnHRq9MSKTsRQJV4oUIAHJMHoxLlfaWWDCFqKsO95Y%3D&reserved=0).
Regards,
Jonas
--
View this message in context: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache-flink-user-mailing-list-archive.2336050.n4.nabble.com%2FAnomaly-Detection-with-Flink-ML-tp14149p14151.html&data=02%7C01%7CJeremy.D.Branham%40sprint.com%7C66a475a8840f40aee51808d4c57cbe82%7C4f8bc0acbd784bf5b55f1b31301d9adf%7C0%7C0%7C636350587286223292&sdata=2PRn8Gb7vOAoCT7r91J4ewhTaVauSEXK380vZp4YV2Q%3D&reserved=0
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
________________________________
This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
Re: Anomaly Detection with Flink-ML
Posted by Jonas Gröger <jo...@huntun.de>.
Hello Jeremy,
it looks like what you are looking for is map (1 in, 1 out) / flatmap (1 in,
0-n out) for preprocessing on a single element basis as well as windows for
looking at related MetricDefinition elements calculating some result.
I suggest you look into Windows
(https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/windows.html)
and basic transformations
(https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/datastream_api.html#datastream-transformations).
Regards,
Jonas
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Anomaly-Detection-with-Flink-ML-tp14149p14151.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.