You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@metron.apache.org by Jack Roberts <jr...@turing.ac.uk> on 2021/02/02 09:18:25 UTC
Metron's Future / Alternatives
Hi Metron community,
I recently started to explore Metron as part of a research project I’m involved in, but I’ve just seen the unfortunate news that the project is being “moved to the Attic”. I’d be very grateful if anyone could help to clarify the following:
* Is there likely to be any continued development of Metron outside of Apache?
* Are there any alternatives to Metron that people in this community would recommend? In particular, we’re looking for something open source that we can deploy ourselves, and with the functionality to straightforwardly integrate our own machine learning models for anomaly detection/similar.
Many thanks and best wishes,
Jack
The Alan Turing Institute is a limited liability company, registered in England with registered number 09512457 with registered offices at British Library, 96 Euston Road, London, England, NW1 2DB. We are also a charity registered in England with charity number 1162533. DISCLAIMER: Although we have taken reasonable precautions to ensure the completeness and accuracy of this e-mail, transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or be incomplete. If you receive a suspicious or unexpected email from us, or purporting to have been sent on our behalf, particularly containing different bank details, please do not reply to the email, click on any links, open any attachments, or comply with any instructions contained within it. Our Transparency Notice found here - https://www.turing.ac.uk/transparency-notice sets out how and why we collect, store, use and share your personal data and it explains your rights and how to raise concerns with us.
Re: Metron's Future / Alternatives
Posted by Jack Roberts <jr...@turing.ac.uk>.
Hi Alex,
Thanks for your reply! We’re hoping to find and start using something on quite a short deadline so we’ll probably be looking for something that’s more established in the short-term, but I’d certainly be interested to check out what you’re working on when it’s public.
As for the models – the honest answer is I’m not sure yet, but it’s for research purposes so we want it to be as flexible as possible (i.e. we want to be able to plug in our own code and algorithms and not be tied to pre-existing templates).
Best wishes,
Jack
From: Alex Scammon <al...@alexscammon.com>
Date: Wednesday, 3 February 2021 at 00:08
To: user@metron.apache.org <us...@metron.apache.org>
Subject: Re: Metron's Future / Alternatives
Hey there Jack,
We were also disappointed that Metron was shuttered. But we've seen it as an opportunity to continue an internal project which builds on top of Metron. Our goal is to make our project open source as a potential successor to Metron.
We're maybe a month or two away from making it public, but we'd love some eyes on it before we take that step if you're interested in taking a look.
Since it builds on Metron, a lot of the core architecture remains familiar. Java, Kafka, Storm, etc.... Hopefully, that presents a familiar ecosystem for folks who are currently using Metron. For improvements, we focused on:
· Ensuring that simple configuration mistakes don't bring down the whole pipeline
· A git-based approval workflow for rules updates (approvals and an audit trail are important for us)
· An improved, modern-looking UI in Angular
· Easier installation steps
Let me know if you're interested in discussing more -- I'd be interested to hear whether there are any particulars about the models you're running that we should take into consideration.
Cheers,
Alex Scammon
Head of Open Source Development
G-Research
gresearch.co.uk<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgresearch.co.uk%2F&data=04%7C01%7Cjroberts%40turing.ac.uk%7Cf4b382b60efa417500b708d8c7d7d0fa%7C4395f4a7e4554f958a9f1fbaef6384f9%7C0%7C0%7C637479077046081220%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=k%2FO3q4Tzankm95UvuclT8tAM0VB9v0aPUvBABsUqvmY%3D&reserved=0>
On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts <jr...@turing.ac.uk>> wrote:
Hi Metron community,
I recently started to explore Metron as part of a research project I’m involved in, but I’ve just seen the unfortunate news that the project is being “moved to the Attic”. I’d be very grateful if anyone could help to clarify the following:
* Is there likely to be any continued development of Metron outside of Apache?
* Are there any alternatives to Metron that people in this community would recommend? In particular, we’re looking for something open source that we can deploy ourselves, and with the functionality to straightforwardly integrate our own machine learning models for anomaly detection/similar.
Many thanks and best wishes,
Jack
The Alan Turing Institute is a limited liability company, registered in England with registered number 09512457 with registered offices at British Library, 96 Euston Road, London, England, NW1 2DB. We are also a charity registered in England with charity number 1162533. DISCLAIMER: Although we have taken reasonable precautions to ensure the completeness and accuracy of this e-mail, transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or be incomplete. If you receive a suspicious or unexpected email from us, or purporting to have been sent on our behalf, particularly containing different bank details, please do not reply to the email, click on any links, open any attachments, or comply with any instructions contained within it. Our Transparency Notice found here - https://www.turing.ac.uk/transparency-notice<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.turing.ac.uk%2Ftransparency-notice&data=04%7C01%7Cjroberts%40turing.ac.uk%7Cf4b382b60efa417500b708d8c7d7d0fa%7C4395f4a7e4554f958a9f1fbaef6384f9%7C0%7C0%7C637479077046081220%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WjY03b5b6%2BUjnmYESbvfsDaFdAVtwU1Iuhsm66oOC9w%3D&reserved=0> sets out how and why we collect, store, use and share your personal data and it explains your rights and how to raise concerns with us.
The Alan Turing Institute is a limited liability company, registered in England with registered number 09512457 with registered offices at British Library, 96 Euston Road, London, England, NW1 2DB. We are also a charity registered in England with charity number 1162533. DISCLAIMER: Although we have taken reasonable precautions to ensure the completeness and accuracy of this e-mail, transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or be incomplete. If you receive a suspicious or unexpected email from us, or purporting to have been sent on our behalf, particularly containing different bank details, please do not reply to the email, click on any links, open any attachments, or comply with any instructions contained within it. Our Transparency Notice found here - https://www.turing.ac.uk/transparency-notice sets out how and why we collect, store, use and share your personal data and it explains your rights and how to raise concerns with us.
Re: Metron's Future / Alternatives
Posted by updates on tube <ab...@gmail.com>.
Okay, so is it starting as a new architecture?
On 2021/02/03 01:28:21, Simon Elliston Ball <si...@simonellistonball.com> wrote:
> We’re also replacing much of the functionality of Metron with a series of
> Apache Flink based streaming components in a number of installations. It
> makes for a composable approach, and borrows from elements of the Metron
> architecture, while using more efficient formats like Apache Avro instead
> of JSON to reduce Kafka consumption and increase performance. Using Flink
> also allows for more efficient aggregation and sql based rules.
>
> It’s much more of a custom solution than the generic project Metron took
> on, but seems to be working well for many of the same log related use cases.
>
> Simon
>
> On Wed, 3 Feb 2021 at 00:08, Alex Scammon <al...@alexscammon.com> wrote:
>
> > Hey there Jack,
> >
> > We were also disappointed that Metron was shuttered. But we've seen it as
> > an opportunity to continue an internal project which builds on top of
> > Metron. Our goal is to make our project open source as a potential
> > successor to Metron.
> >
> > We're maybe a month or two away from making it public, but we'd love some
> > eyes on it before we take that step if you're interested in taking a look.
> >
> > Since it builds on Metron, a lot of the core architecture remains
> > familiar. Java, Kafka, Storm, etc.... Hopefully, that presents a familiar
> > ecosystem for folks who are currently using Metron. For improvements, we
> > focused on:
> >
> > - Ensuring that simple configuration mistakes don't bring down the
> > whole pipeline
> > - A git-based approval workflow for rules updates (approvals and an
> > audit trail are important for us)
> > - An improved, modern-looking UI in Angular
> > - Easier installation steps
> >
> > Let me know if you're interested in discussing more -- I'd be interested
> > to hear whether there are any particulars about the models you're running
> > that we should take into consideration.
> >
> > Cheers,
> >
> > Alex Scammon
> > Head of Open Source Development
> > G-Research
> > gresearch.co.uk
> >
> >
> > On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts <jr...@turing.ac.uk> wrote:
> >
> >> Hi Metron community,
> >>
> >>
> >>
> >> I recently started to explore Metron as part of a research project I’m
> >> involved in, but I’ve just seen the unfortunate news that the project is
> >> being “moved to the Attic”. I’d be very grateful if anyone could help to
> >> clarify the following:
> >>
> >>
> >>
> >> - Is there likely to be any continued development of Metron outside
> >> of Apache?
> >> - Are there any alternatives to Metron that people in this community
> >> would recommend? In particular, we’re looking for something open source
> >> that we can deploy ourselves, and with the functionality to
> >> straightforwardly integrate our own machine learning models for anomaly
> >> detection/similar.
> >>
> >>
> >>
> >> Many thanks and best wishes,
> >>
> >> Jack
> >> The Alan Turing Institute is a limited liability company, registered in
> >> England with registered number 09512457 with registered offices at British
> >> Library, 96 Euston Road, London, England, NW1 2DB
> >> <https://www.google.com/maps/search/96+Euston+Road,+London,+England,+NW1+2DB?entry=gmail&source=g>.
> >> We are also a charity registered in England with charity number 1162533.
> >> DISCLAIMER: Although we have taken reasonable precautions to ensure the
> >> completeness and accuracy of this e-mail, transmission cannot be guaranteed
> >> to be secure or error-free as information could be intercepted, corrupted,
> >> lost, destroyed, arrive late or be incomplete. If you receive a suspicious
> >> or unexpected email from us, or purporting to have been sent on our behalf,
> >> particularly containing different bank details, please do not reply to the
> >> email, click on any links, open any attachments, or comply with any
> >> instructions contained within it. Our Transparency Notice found here -
> >> https://www.turing.ac.uk/transparency-notice sets out how and why we
> >> collect, store, use and share your personal data and it explains your
> >> rights and how to raise concerns with us.
> >>
> > --
> --
> simon elliston ball
> @sireb
>
Re: Metron's Future / Alternatives
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
We’re also replacing much of the functionality of Metron with a series of
Apache Flink based streaming components in a number of installations. It
makes for a composable approach, and borrows from elements of the Metron
architecture, while using more efficient formats like Apache Avro instead
of JSON to reduce Kafka consumption and increase performance. Using Flink
also allows for more efficient aggregation and sql based rules.
It’s much more of a custom solution than the generic project Metron took
on, but seems to be working well for many of the same log related use cases.
Simon
On Wed, 3 Feb 2021 at 00:08, Alex Scammon <al...@alexscammon.com> wrote:
> Hey there Jack,
>
> We were also disappointed that Metron was shuttered. But we've seen it as
> an opportunity to continue an internal project which builds on top of
> Metron. Our goal is to make our project open source as a potential
> successor to Metron.
>
> We're maybe a month or two away from making it public, but we'd love some
> eyes on it before we take that step if you're interested in taking a look.
>
> Since it builds on Metron, a lot of the core architecture remains
> familiar. Java, Kafka, Storm, etc.... Hopefully, that presents a familiar
> ecosystem for folks who are currently using Metron. For improvements, we
> focused on:
>
> - Ensuring that simple configuration mistakes don't bring down the
> whole pipeline
> - A git-based approval workflow for rules updates (approvals and an
> audit trail are important for us)
> - An improved, modern-looking UI in Angular
> - Easier installation steps
>
> Let me know if you're interested in discussing more -- I'd be interested
> to hear whether there are any particulars about the models you're running
> that we should take into consideration.
>
> Cheers,
>
> Alex Scammon
> Head of Open Source Development
> G-Research
> gresearch.co.uk
>
>
> On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts <jr...@turing.ac.uk> wrote:
>
>> Hi Metron community,
>>
>>
>>
>> I recently started to explore Metron as part of a research project I’m
>> involved in, but I’ve just seen the unfortunate news that the project is
>> being “moved to the Attic”. I’d be very grateful if anyone could help to
>> clarify the following:
>>
>>
>>
>> - Is there likely to be any continued development of Metron outside
>> of Apache?
>> - Are there any alternatives to Metron that people in this community
>> would recommend? In particular, we’re looking for something open source
>> that we can deploy ourselves, and with the functionality to
>> straightforwardly integrate our own machine learning models for anomaly
>> detection/similar.
>>
>>
>>
>> Many thanks and best wishes,
>>
>> Jack
>> The Alan Turing Institute is a limited liability company, registered in
>> England with registered number 09512457 with registered offices at British
>> Library, 96 Euston Road, London, England, NW1 2DB
>> <https://www.google.com/maps/search/96+Euston+Road,+London,+England,+NW1+2DB?entry=gmail&source=g>.
>> We are also a charity registered in England with charity number 1162533.
>> DISCLAIMER: Although we have taken reasonable precautions to ensure the
>> completeness and accuracy of this e-mail, transmission cannot be guaranteed
>> to be secure or error-free as information could be intercepted, corrupted,
>> lost, destroyed, arrive late or be incomplete. If you receive a suspicious
>> or unexpected email from us, or purporting to have been sent on our behalf,
>> particularly containing different bank details, please do not reply to the
>> email, click on any links, open any attachments, or comply with any
>> instructions contained within it. Our Transparency Notice found here -
>> https://www.turing.ac.uk/transparency-notice sets out how and why we
>> collect, store, use and share your personal data and it explains your
>> rights and how to raise concerns with us.
>>
> --
--
simon elliston ball
@sireb
Re: Metron's Future / Alternatives
Posted by Alex Scammon <al...@alexscammon.com>.
Hey there Jack,
We were also disappointed that Metron was shuttered. But we've seen it as
an opportunity to continue an internal project which builds on top of
Metron. Our goal is to make our project open source as a potential
successor to Metron.
We're maybe a month or two away from making it public, but we'd love some
eyes on it before we take that step if you're interested in taking a look.
Since it builds on Metron, a lot of the core architecture remains
familiar. Java, Kafka, Storm, etc.... Hopefully, that presents a familiar
ecosystem for folks who are currently using Metron. For improvements, we
focused on:
- Ensuring that simple configuration mistakes don't bring down the whole
pipeline
- A git-based approval workflow for rules updates (approvals and an
audit trail are important for us)
- An improved, modern-looking UI in Angular
- Easier installation steps
Let me know if you're interested in discussing more -- I'd be interested to
hear whether there are any particulars about the models you're running that
we should take into consideration.
Cheers,
Alex Scammon
Head of Open Source Development
G-Research
gresearch.co.uk
On Tue, Feb 2, 2021 at 1:18 AM Jack Roberts <jr...@turing.ac.uk> wrote:
> Hi Metron community,
>
>
>
> I recently started to explore Metron as part of a research project I’m
> involved in, but I’ve just seen the unfortunate news that the project is
> being “moved to the Attic”. I’d be very grateful if anyone could help to
> clarify the following:
>
>
>
> - Is there likely to be any continued development of Metron outside of
> Apache?
> - Are there any alternatives to Metron that people in this community
> would recommend? In particular, we’re looking for something open source
> that we can deploy ourselves, and with the functionality to
> straightforwardly integrate our own machine learning models for anomaly
> detection/similar.
>
>
>
> Many thanks and best wishes,
>
> Jack
> The Alan Turing Institute is a limited liability company, registered in
> England with registered number 09512457 with registered offices at British
> Library, 96 Euston Road, London, England, NW1 2DB. We are also a charity
> registered in England with charity number 1162533. DISCLAIMER: Although we
> have taken reasonable precautions to ensure the completeness and accuracy
> of this e-mail, transmission cannot be guaranteed to be secure or
> error-free as information could be intercepted, corrupted, lost, destroyed,
> arrive late or be incomplete. If you receive a suspicious or unexpected
> email from us, or purporting to have been sent on our behalf, particularly
> containing different bank details, please do not reply to the email, click
> on any links, open any attachments, or comply with any instructions
> contained within it. Our Transparency Notice found here -
> https://www.turing.ac.uk/transparency-notice sets out how and why we
> collect, store, use and share your personal data and it explains your
> rights and how to raise concerns with us.
>