You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Sanket Sharma <sa...@dukstra.com> on 2019/11/27 23:01:12 UTC
Re: [DISCUSS] How are you using in Metron?
Hi,
Thank you for starting a great discussion! We started exploring Metron in June this for networking monitoring. We are piloting it with an objective of replacing Splunk in certain or perhaps all scenarios. We’re looking at about 2 TB of data per day.
1. Features we are currently considering:
* Enrichments
* Streaming enhancements: We are using Spark to do some enrichments but need to explore this further.
* Profiler: Not using it at the moment
* Pcap: Not using it at the moment.
* Flatfile summarizer: Not using it at the moment.
* MaaS: IMHO this needs serious usability enhancements, especially for data scientists. Deploying models seems like a common issue that most data scientist struggle with (at least in our area, unless they have serious python/engineering skills.).
* Meta alerts: Not using it at the moment
* Parser aggregation: Limited use
* Config UI: Using it extensively to configure sensors and rules.
* Alert UI: Using it extensive to view alerts.
* Elastic search: Using it extensively to index alerts and other data.
* Stellar: Not using it at the moment, except for creating rules with scores in the config UI.
* Stellar REPL: Not using it at all
* REST API: Not using it explicitly.
* Other?
2. Many features around usability can be improved:
* Model deployment can reconsidered as a whole.
* Ability to compare models
* Config UI field configuration could be improved
* General ease of use/deployment, documentation
* Templates for common use cases
* Reports – we just can’t do without reporting in the enterprise ☺
3. Alerts UI, Stellar and pipelines I suppose.
4. I would love to contribute ☺, just in the middle of a big relocation. Hopefully, I will be able to resume and join the community in next 2-3 months.
We have another interesting use case where we kind of started prototyping Metron – financial fraud. Although it might sound a very different and unrelated domain, the “technical architectural pattern” is astonishingly similar. We receive streaming and batch data from various channels over kafka, gets enriched and the based on certain rules we assign a score to it. It then makes it to the alert UI where investigators can further examine the transactions. This is obviously an oversimplification, but I hope you get the idea.
I was thinking of proposing a fork or perhaps a different “flavour” of metron that caters for finance domain and can be built as a separate project, although not sure how to go about it. Is that something the community/project owners might be interested in considering or supporting?
Best regards,
Sanket
From: Michael Miklavcic <mi...@gmail.com>
Reply to: "user@metron.apache.org" <us...@metron.apache.org>
Date: Thursday, 17 October 2019 at 18:22
To: "dev@metron.apache.org" <de...@metron.apache.org>, "user@metron.apache.org" <us...@metron.apache.org>
Subject: [DISCUSS] How are you using in Metron?
I'd like to kick off a discussion to get a sense of how the broader community is currently using Metron.
1. What features are you using or seriously considering? e.g.
1. enrichments
2. streaming enrichments
3. profiler
4. pcap
5. flatfile summarizer
6. MaaS
7. Meta alerts
8. parser aggregation
9. config UI
10. alert UI
11. solr, ES
12. Stellar
13. Stellar REPL
14. REST API
15. other?
2. What features would you like to see added or improved?
3. What features do you consider to be core to Metron as a platform?
4. If you're using Metron, but not an active community contributor, what would it take to get you more involved in the project?
We are close to finishing up a feature branch around upgrading to HDP 3.1, and subsequently on the doorstep of a 1.0 release. This is a huge milestone for the project. I think it's time to take some lessons learned over the past several years and consider what the next phase of Metron will be. Whether you've participated in community discussions before or not, we'd love to hear from you.
Best,
Mike Miklavcic
PMC Apache Metron
Re: [DISCUSS] How are you using in Metron?
Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Sanket, thanks for sharing!
Can you elaborate a bit more on your experience and challenges with model
deployment?
> We have another interesting use case where we kind of started prototyping
Metron – financial fraud. Although it might sound a very different and
unrelated domain, the “technical architectural pattern” is astonishingly
similar.
TBH, you could probably view Metron even more broadly than that.
Fundamentally, it's a streaming analytics platform with some emphasis on
cybersecurity to keep things a bit more focused. But I see absolutely no
reason why you couldn't replace terminology like "sensor/parser" with
something more generalized such as "data source." We get data into the
system, normalize it, provide hooks for enhancing (enriching) that data via
a variety of sources including machine learning models, and flag records
and provide a highly configurable method to score them. I mean, why not use
this for genomics? Or dynamic live traffic adjustments? Or stock trading?
Etc...
On Wed, Nov 27, 2019, 4:01 PM Sanket Sharma <sa...@dukstra.com>
wrote:
> Hi,
>
>
>
> Thank you for starting a great discussion! We started exploring Metron in
> June this for networking monitoring. We are piloting it with an objective
> of replacing Splunk in certain or perhaps all scenarios. We’re looking at
> about 2 TB of data per day.
>
> 1. Features we are currently considering:
> 1. Enrichments
> 2. Streaming enhancements: We are using Spark to do some
> enrichments but need to explore this further.
> 3. Profiler: Not using it at the moment
> 4. Pcap: Not using it at the moment.
> 5. Flatfile summarizer: Not using it at the moment.
> 6. MaaS: IMHO this needs serious usability enhancements, especially
> for data scientists. Deploying models seems like a common issue that most
> data scientist struggle with (at least in our area, unless they have
> serious python/engineering skills.).
> 7. Meta alerts: Not using it at the moment
> 8. Parser aggregation: Limited use
> 9. Config UI: Using it extensively to configure sensors and rules.
> 10. Alert UI: Using it extensive to view alerts.
> 11. Elastic search: Using it extensively to index alerts and other
> data.
> 12. Stellar: Not using it at the moment, except for creating rules
> with scores in the config UI.
> 13. Stellar REPL: Not using it at all
> 14. REST API: Not using it explicitly.
> 15. Other?
> 2. Many features around usability can be improved:
> 1. Model deployment can reconsidered as a whole.
> 2. Ability to compare models
> 3. Config UI field configuration could be improved
> 4. General ease of use/deployment, documentation
> 5. Templates for common use cases
> 6. Reports – we just can’t do without reporting in the enterprise ☺
> 3. Alerts UI, Stellar and pipelines I suppose.
> 4. I would love to contribute ☺, just in the middle of a big
> relocation. Hopefully, I will be able to resume and join the community in
> next 2-3 months.
>
>
>
> We have another interesting use case where we kind of started prototyping
> Metron – financial fraud. Although it might sound a very different and
> unrelated domain, the “technical architectural pattern” is astonishingly
> similar. We receive streaming and batch data from various channels over
> kafka, gets enriched and the based on certain rules we assign a score to
> it. It then makes it to the alert UI where investigators can further
> examine the transactions. This is obviously an oversimplification, but I
> hope you get the idea.
>
>
>
> I was thinking of proposing a fork or perhaps a different “flavour” of
> metron that caters for finance domain and can be built as a separate
> project, although not sure how to go about it. Is that something the
> community/project owners might be interested in considering or supporting?
>
>
>
> Best regards,
>
> Sanket
>
>
>
> *From: *Michael Miklavcic <mi...@gmail.com>
> *Reply to: *"user@metron.apache.org" <us...@metron.apache.org>
> *Date: *Thursday, 17 October 2019 at 18:22
> *To: *"dev@metron.apache.org" <de...@metron.apache.org>, "
> user@metron.apache.org" <us...@metron.apache.org>
> *Subject: *[DISCUSS] How are you using in Metron?
>
>
>
> I'd like to kick off a discussion to get a sense of how the broader
> community is currently using Metron.
>
> 1. What features are you using or seriously considering? e.g.
>
> 1. enrichments
>
> 2. streaming enrichments
>
> 3. profiler
>
> 4. pcap
>
> 5. flatfile summarizer
>
> 6. MaaS
>
> 7. Meta alerts
>
> 8. parser aggregation
>
> 9. config UI
>
> 10. alert UI
>
> 11. solr, ES
>
> 12. Stellar
>
> 13. Stellar REPL
>
> 14. REST API
>
> 15. other?
>
> 2. What features would you like to see added or improved?
>
> 3. What features do you consider to be core to Metron as a platform?
>
> 4. If you're using Metron, but not an active community contributor,
> what would it take to get you more involved in the project?
>
> We are close to finishing up a feature branch around upgrading to HDP 3.1,
> and subsequently on the doorstep of a 1.0 release. This is a huge milestone
> for the project. I think it's time to take some lessons learned over the
> past several years and consider what the next phase of Metron will be.
> Whether you've participated in community discussions before or not, we'd
> love to hear from you.
>
>
>
> Best,
>
> Mike Miklavcic
>
> PMC Apache Metron
>
Re: [DISCUSS] How are you using in Metron?
Posted by Michael Miklavcic <mi...@gmail.com>.
Hi Sanket, thanks for sharing!
Can you elaborate a bit more on your experience and challenges with model
deployment?
> We have another interesting use case where we kind of started prototyping
Metron – financial fraud. Although it might sound a very different and
unrelated domain, the “technical architectural pattern” is astonishingly
similar.
TBH, you could probably view Metron even more broadly than that.
Fundamentally, it's a streaming analytics platform with some emphasis on
cybersecurity to keep things a bit more focused. But I see absolutely no
reason why you couldn't replace terminology like "sensor/parser" with
something more generalized such as "data source." We get data into the
system, normalize it, provide hooks for enhancing (enriching) that data via
a variety of sources including machine learning models, and flag records
and provide a highly configurable method to score them. I mean, why not use
this for genomics? Or dynamic live traffic adjustments? Or stock trading?
Etc...
On Wed, Nov 27, 2019, 4:01 PM Sanket Sharma <sa...@dukstra.com>
wrote:
> Hi,
>
>
>
> Thank you for starting a great discussion! We started exploring Metron in
> June this for networking monitoring. We are piloting it with an objective
> of replacing Splunk in certain or perhaps all scenarios. We’re looking at
> about 2 TB of data per day.
>
> 1. Features we are currently considering:
> 1. Enrichments
> 2. Streaming enhancements: We are using Spark to do some
> enrichments but need to explore this further.
> 3. Profiler: Not using it at the moment
> 4. Pcap: Not using it at the moment.
> 5. Flatfile summarizer: Not using it at the moment.
> 6. MaaS: IMHO this needs serious usability enhancements, especially
> for data scientists. Deploying models seems like a common issue that most
> data scientist struggle with (at least in our area, unless they have
> serious python/engineering skills.).
> 7. Meta alerts: Not using it at the moment
> 8. Parser aggregation: Limited use
> 9. Config UI: Using it extensively to configure sensors and rules.
> 10. Alert UI: Using it extensive to view alerts.
> 11. Elastic search: Using it extensively to index alerts and other
> data.
> 12. Stellar: Not using it at the moment, except for creating rules
> with scores in the config UI.
> 13. Stellar REPL: Not using it at all
> 14. REST API: Not using it explicitly.
> 15. Other?
> 2. Many features around usability can be improved:
> 1. Model deployment can reconsidered as a whole.
> 2. Ability to compare models
> 3. Config UI field configuration could be improved
> 4. General ease of use/deployment, documentation
> 5. Templates for common use cases
> 6. Reports – we just can’t do without reporting in the enterprise ☺
> 3. Alerts UI, Stellar and pipelines I suppose.
> 4. I would love to contribute ☺, just in the middle of a big
> relocation. Hopefully, I will be able to resume and join the community in
> next 2-3 months.
>
>
>
> We have another interesting use case where we kind of started prototyping
> Metron – financial fraud. Although it might sound a very different and
> unrelated domain, the “technical architectural pattern” is astonishingly
> similar. We receive streaming and batch data from various channels over
> kafka, gets enriched and the based on certain rules we assign a score to
> it. It then makes it to the alert UI where investigators can further
> examine the transactions. This is obviously an oversimplification, but I
> hope you get the idea.
>
>
>
> I was thinking of proposing a fork or perhaps a different “flavour” of
> metron that caters for finance domain and can be built as a separate
> project, although not sure how to go about it. Is that something the
> community/project owners might be interested in considering or supporting?
>
>
>
> Best regards,
>
> Sanket
>
>
>
> *From: *Michael Miklavcic <mi...@gmail.com>
> *Reply to: *"user@metron.apache.org" <us...@metron.apache.org>
> *Date: *Thursday, 17 October 2019 at 18:22
> *To: *"dev@metron.apache.org" <de...@metron.apache.org>, "
> user@metron.apache.org" <us...@metron.apache.org>
> *Subject: *[DISCUSS] How are you using in Metron?
>
>
>
> I'd like to kick off a discussion to get a sense of how the broader
> community is currently using Metron.
>
> 1. What features are you using or seriously considering? e.g.
>
> 1. enrichments
>
> 2. streaming enrichments
>
> 3. profiler
>
> 4. pcap
>
> 5. flatfile summarizer
>
> 6. MaaS
>
> 7. Meta alerts
>
> 8. parser aggregation
>
> 9. config UI
>
> 10. alert UI
>
> 11. solr, ES
>
> 12. Stellar
>
> 13. Stellar REPL
>
> 14. REST API
>
> 15. other?
>
> 2. What features would you like to see added or improved?
>
> 3. What features do you consider to be core to Metron as a platform?
>
> 4. If you're using Metron, but not an active community contributor,
> what would it take to get you more involved in the project?
>
> We are close to finishing up a feature branch around upgrading to HDP 3.1,
> and subsequently on the doorstep of a 1.0 release. This is a huge milestone
> for the project. I think it's time to take some lessons learned over the
> past several years and consider what the next phase of Metron will be.
> Whether you've participated in community discussions before or not, we'd
> love to hear from you.
>
>
>
> Best,
>
> Mike Miklavcic
>
> PMC Apache Metron
>