Re: [DISCUSS] How are you using in Metron?

[Michael Miklavcic <mi...@gmail.com>]

Hi Sanket, thanks for sharing!

Can you elaborate a bit more on your experience and challenges with model
deployment?

> We have another interesting use case where we kind of started prototyping
Metron – financial fraud. Although it might sound a very different and
unrelated domain, the “technical architectural pattern” is astonishingly
similar.

TBH, you could probably view Metron even more broadly than that.
Fundamentally, it's a streaming analytics platform with some emphasis on
cybersecurity to keep things a bit more focused. But I see absolutely no
reason why you couldn't replace terminology like "sensor/parser" with
something more generalized such as "data source." We get data into the
system, normalize it, provide hooks for enhancing (enriching) that data via
a variety of sources including machine learning models, and flag records
and provide a highly configurable method to score them. I mean, why not use
this for genomics? Or dynamic live traffic adjustments? Or stock trading?
Etc...


On Wed, Nov 27, 2019, 4:01 PM Sanket Sharma <sa...@dukstra.com>
wrote:

> Hi,
>
>
>
> Thank you for starting a great discussion! We started exploring Metron in
> June this for networking monitoring. We are piloting it with an objective
> of replacing Splunk in certain or perhaps all scenarios. We’re looking at
> about 2 TB of data per day.
>
>    1. Features we are currently considering:
>       1. Enrichments
>       2. Streaming enhancements: We are using Spark to do some
>       enrichments but need to explore this further.
>       3. Profiler: Not using it at the moment
>       4. Pcap: Not using it at the moment.
>       5. Flatfile summarizer: Not using it at the moment.
>       6. MaaS: IMHO this needs serious usability enhancements, especially
>       for data scientists. Deploying models seems like a common issue that most
>       data scientist struggle with (at least in our area, unless they have
>       serious python/engineering skills.).
>       7. Meta alerts: Not using it at the moment
>       8. Parser aggregation: Limited use
>       9. Config UI: Using it extensively to configure sensors and rules.
>       10. Alert UI: Using it extensive to view alerts.
>       11. Elastic search: Using it extensively to index alerts and other
>       data.
>       12. Stellar: Not using it at the moment, except for creating rules
>       with scores in the config UI.
>       13. Stellar REPL: Not using it at all
>       14. REST API: Not using it explicitly.
>       15. Other?
>    2. Many features around usability can be improved:
>       1. Model deployment can reconsidered as a whole.
>       2. Ability to compare models
>       3. Config UI field configuration could be improved
>       4. General ease of use/deployment, documentation
>       5. Templates for common use cases
>       6. Reports – we just can’t do without reporting in the enterprise ☺
>    3. Alerts UI, Stellar and pipelines I suppose.
>    4. I would love to contribute ☺, just in the middle of a big
>    relocation. Hopefully, I will be able to resume and join the community in
>    next 2-3 months.
>
>
>
> We have another interesting use case where we kind of started prototyping
> Metron – financial fraud. Although it might sound a very different and
> unrelated domain, the “technical architectural pattern” is astonishingly
> similar. We receive streaming and batch data from various channels over
> kafka, gets enriched and the based on certain rules we assign a score to
> it. It then makes it to the alert UI where investigators can further
> examine the transactions. This is obviously an oversimplification, but I
> hope you get the idea.
>
>
>
> I was thinking of proposing a fork or perhaps a different “flavour” of
> metron that caters for finance domain and can be built as a separate
> project, although not sure how to go about it. Is that something the
> community/project owners might be interested in considering or supporting?
>
>
>
> Best regards,
>
> Sanket
>
>
>
> *From: *Michael Miklavcic <mi...@gmail.com>
> *Reply to: *"user@metron.apache.org" <us...@metron.apache.org>
> *Date: *Thursday, 17 October 2019 at 18:22
> *To: *"dev@metron.apache.org" <de...@metron.apache.org>, "
> user@metron.apache.org" <us...@metron.apache.org>
> *Subject: *[DISCUSS] How are you using in Metron?
>
>
>
> I'd like to kick off a discussion to get a sense of how the broader
> community is currently using Metron.
>
> 1.       What features are you using or seriously considering? e.g.
>
> 1.       enrichments
>
> 2.       streaming enrichments
>
> 3.       profiler
>
> 4.       pcap
>
> 5.       flatfile summarizer
>
> 6.       MaaS
>
> 7.       Meta alerts
>
> 8.       parser aggregation
>
> 9.       config UI
>
> 10.   alert UI
>
> 11.   solr, ES
>
> 12.   Stellar
>
> 13.   Stellar REPL
>
> 14.   REST API
>
> 15.   other?
>
> 2.       What features would you like to see added or improved?
>
> 3.       What features do you consider to be core to Metron as a platform?
>
> 4.       If you're using Metron, but not an active community contributor,
> what would it take to get you more involved in the project?
>
> We are close to finishing up a feature branch around upgrading to HDP 3.1,
> and subsequently on the doorstep of a 1.0 release. This is a huge milestone
> for the project. I think it's time to take some lessons learned over the
> past several years and consider what the next phase of Metron will be.
> Whether you've participated in community discussions before or not, we'd
> love to hear from you.
>
>
>
> Best,
>
> Mike Miklavcic
>
> PMC Apache Metron
>