Metron and Edge Analytics

[Julian Feinauer <j....@pragmaticminds.de>]

Hi Metron-People,

I asked the following question on the Apache NiFi list and Otto Fowler approached me and introduced me to the Apache Metron Projekt.
I would be pleased to discuss together with you guys what you do and what we are looking for.

Best Julian

PS.: See my original message below

I’m from the incubating plc4x project [1] and I am looking for a framework which is suitable for the management of IoT Datastreams and do some edge computing.
As nifi is often times mentioned in relation with IoT I tried to find out what nifi realy does and how it would fit with our ideas (and also the MiNiFi Project seems to fit into this).

From what I understood from the Docs and some Videos NiFi looks for me a bit like Apache Camel [2] as it is able to (dynamically) integrate different systems and manage the dataflow between them. So what I did not get exactly I how the payloads are managed between these Endpoints and how much of processing Nifi does itself and how much it delegates to other components (like e.g. Service Activater in EIP).

What I am looking for is a framework which does some analysis of data streams coming from controllers that, e.g., control machines or robots. chrisdutz already prepared the first version of an NiFi Endpoint in th Plc4x Repo so we are already able to stream these datasets to NiFi. Whats unclear to me is how we could tackle some of the questions like “how long was this bit set” or “notify me when this signal is below a certain threshold for more than 30s” or so.
Is this in the scope of NiFi or is NiFi more of an integration / data-flow layer which is absolutely agnostic of these processing blocks?

I hope my questions are not too dumb or I’m not missing NiFis core too much with my current knowledge.
I would be happy for some answers or some ideas about how to approach the questions stated above by some experienced users.

Best
Julian

[1] http://plc4x.incubator.apache.org/
[2] https://camel.apache.org/
[3] https://github.com/apache/incubator-plc4x/tree/master/integrations/apache-nifi

Re: Metron and Edge Analytics

[Michael Miklavcic <mi...@gmail.com>]

Hi Julian,

Welcome, and thanks for reaching out! If you're looking for processing at
the edge, then I think you're on the right track with NiFi and MiNiFi.
Where Metron would be of potential use to you is if you're looking to
perform additional analytics, ie as you stated things like "how long bit
set... notifiy me signal below threshold". It's a cybersecurity-first
platform, however the core platform is really something you could use as a
general purpose streaming analytics platform. We readily accept input from
sources such as NiFi via Kafka topics that we then send through a series of
real-time streaming processing steps, ending finally with indexing to a doc
store such as Elasticsearch or Solr. Very simply put:

   - parsing - puts data in a format consumable by the Metron ecosystem
   - enrichment - enrich data with lookups or other customizable
   transformations, e.g. whois info, geo tagging
   - indexing (Elasticsearch or Solr, and HDFS)

There is also support for performing outlier analysis using our profiler,
running ML models via model as a service, and more. For a sampling of some
of our capabilities, check out our blog tutorials here -
https://cwiki.apache.org/confluence/display/METRON/2016/06/22/Metron+Tutorial+-+Fundamentals+Part+7%3A+Dashboarding+with+Kibana.
I linked the last entry for the TOC, of course start with part 1. You might
also have a look at some of the use cases we have write-ups for here -
https://github.com/apache/metron/tree/master/use-cases. Some background on
the profiler can be found here -
https://github.com/apache/metron/blob/master/metron-analytics/metron-profiler-common/README.md
.

Best,
Mike


On Mon, Oct 1, 2018 at 8:30 AM Julian Feinauer <j....@pragmaticminds.de>
wrote:

> Hi Metron-People,
>
>
>
> I asked the following question on the Apache NiFi list and Otto Fowler
> approached me and introduced me to the Apache Metron Projekt.
>
> I would be pleased to discuss together with you guys what you do and what
> we are looking for.
>
>
>
> Best Julian
>
>
>
> PS.: See my original message below
>
>
>
> I’m from the incubating plc4x project [1] and I am looking for a framework
> which is suitable for the management of IoT Datastreams and do some edge
> computing.
>
> As nifi is often times mentioned in relation with IoT I tried to find out
> what nifi realy does and how it would fit with our ideas (and also the
> MiNiFi Project seems to fit into this).
>
>
>
> From what I understood from the Docs and some Videos NiFi looks for me a
> bit like Apache Camel [2] as it is able to (dynamically) integrate
> different systems and manage the dataflow between them. So what I did not
> get exactly I how the payloads are managed between these Endpoints and how
> much of processing Nifi does itself and how much it delegates to other
> components (like e.g. Service Activater in EIP).
>
>
>
> What I am looking for is a framework which does some analysis of data
> streams coming from controllers that, e.g., control machines or robots.
> chrisdutz already prepared the first version of an NiFi Endpoint in th
> Plc4x Repo so we are already able to stream these datasets to NiFi. Whats
> unclear to me is how we could tackle some of the questions like “how long
> was this bit set” or “notify me when this signal is below a certain
> threshold for more than 30s” or so.
>
> Is this in the scope of NiFi or is NiFi more of an integration / data-flow
> layer which is absolutely agnostic of these processing blocks?
>
>
>
> I hope my questions are not too dumb or I’m not missing NiFis core too
> much with my current knowledge.
>
> I would be happy for some answers or some ideas about how to approach the
> questions stated above by some experienced users.
>
>
>
> Best
>
> Julian
>
>
>
> [1] http://plc4x.incubator.apache.org/
>
> [2] https://camel.apache.org/
>
> [3]
> https://github.com/apache/incubator-plc4x/tree/master/integrations/apache-nifi
>
>
>