You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Tyler Moore <tm...@goflyball.com> on 2016/12/30 18:26:42 UTC
Custom Storm Topologies
Happy Holidays Metron Devs!
Could anyone lend me some guidance on customizing the storm topologies in
metron? What I am am trying to accomplish:
1) Add a method to the threat intel joiner bolt that sends an http post
with the score of the threat to a remote rest api. This will conditionally
trigger notifications based on user settings in another database (the
backend processing logic is on another platform).
The score should be available within the JSONObject but I am not an expert
with storm and I am not completely understanding what conditions constitute
when the threat feed is considered an "alert" in metron. Please clarify.
2) How would I add an external dependency, my http rest java class, to the
metron maven build process? More specifically, if I was adding a custom
class that needed accessed by a bolt in storm, how would I add this in
maven as a dependency. I have limited experience with maven but, my
understanding is that I would add it to the pom.xml and recompile.
Although, the metron quick dev platform is built on a vm, would I need to
account for this? Please advise.
Regards,
Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com
Re: Custom Storm Topologies
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
After a visual review that is about what I was thinking. Like you
mentioned it could use some work - I added some comments to the JIRA
hitting on some suggested improvements.
Jon
On Thu, Jan 5, 2017 at 12:03 PM Tyler Moore <tm...@goflyball.com> wrote:
> I attached a sample function to the Jira, just calls a script from on the
> server by passing in the path. Certainly could use some work, but I wanted
> to make sure this was the right idea and get a consensus, let me know your
> thoughts.
>
> My thought process for this is that if we only allow script execution from
> a shell script that must be on the storm host, then the user is in a sense
> authenticated by their own user-defined access restrictions for that
> system.
>
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Phone: 248-909-2769 <(248)%20909-2769>
> Email: moore.tyler@goflyball.com
>
>
> On Tue, Jan 3, 2017 at 3:47 PM, Carolyn Duby <cd...@hortonworks.com>
> wrote:
>
> > Also please consider the security of the scripts and script injection
> > attacks. For example, we should probably restrict file access.
> >
> > Thanks
> > Carolyn
> >
> >
> >
> > On 1/3/17, 3:25 PM, "Otto Fowler" <ot...@gmail.com> wrote:
> >
> > >A script bolt would still allow them to write the script the way they
> want
> > >to, but would avoid having to write all the scaffolding.
> > >The matter then would be how to integrate that script bolt into the
> > >topologies.
> > >
> > >
> > >On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com)
> > wrote:
> > >
> > >Right, that definitely is more efficient, but part of the point here is
> to
> > >lower the barrier of entry to using Metron.
> > >
> > >It makes Metron's triage abilities more flexible and allows a user to
> > reuse
> > >existing code quickly and easily. Having this available for PoC,
> > >prototyping, and low volume environments or situations (only when threat
> > >score is 100, for instance) is important, as it lowers the barrier to
> > entry
> > >of migrating a company to a Metron environment.
> > >
> > >I see this as a tradeoff where I would prioritize ease of use over
> > >efficiency. There's nothing wrong with making both options available,
> at
> > >some point, and making their different use cases clear.
> > >
> > >Jon
> > >
> > >On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
> > >
> > >Well, yes :-)
> > >And clearly it should always be more efficient to write a custom bolt in
> > >Java than to invoke a script and manage it.
> > >
> > >--Matt
> > >
> > >From: Otto Fowler <ot...@gmail.com>
> > >Date: Tuesday, January 3, 2017 at 7:08 AM
> > >To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org
> >,
> > >Matt Foley <ma...@apache.org>
> > >Subject: Re: Custom Storm Topologies
> > >
> > >Wouldn’t that be a bolt?
> > >
> > >
> > >On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
> > >Should we consider a script calling capability that can launch a
> streaming
> > >script and keep it alive and fed, long-term, rather than launching the
> > >script anew every time the Stellar function is invoked? I’m thinking two
> > >basic rules: Write a line, read a line; and always have a timeout. Prob
> > >need a UID of some sort for a cache of running process objects.
> > >
> > >--Matt
> > >
> > >On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
> > >
> > >
> > >Inserting a script inline is ok for low throughput and prototyping but
> > once
> > >you get higher throughput (millions of events per second), it’s probably
> > >going to be a bottleneck.
> > >
> > >
> > >For Metron-571 you might want to consider a java based extension plugin
> > >similar to Eclipse plugins.
> > >
> > >Thanks
> > >Carolyn
> > >
> > >On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
> > >
> > >>Thanks Jon,
> > >>
> > >>I'll look over the tutorial and put something together for the
> SHELL_EXEC
> > >>stellar function.
> > >>I don't believe I have permissions to assign in Jira if you want to
> > assign
> > >>to me my username is devopsec.
> > >>I'll post back details and we can review security issues
> > >>
> > >>Regards,
> > >>
> > >>Tyler Moore
> > >>Software Engineer
> > >>Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> > >>Email: moore.tyler@goflyball.com
> > >>
> > >>
> > >>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> > >>
> > >>> Casey did a tutorial on how to add your own Stellar function here
> > >>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an
> > existing
> > >>> function that does this (current functions are listed here
> > >>> <https://github.com/apache/incubator-metron/tree/master/
> > >>> metron-platform/metron-common#stellar-core-functions>).
> > >>> I noticed that some of the Stellar function documentation was a bit
> > dated
> > >>> so I've opened a PR to update it here
> > >>> <https://github.com/apache/incubator-metron/pull/407>.
> > >>>
> > >>> As this is something I need as well, I'd be happy to assist you
> where I
> > >>> can. Perhaps you want to self-assign METRON-571
> > >>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
> > >>> security concerns with a SHELL_EXEC function because it could result
> in
> > >RCE
> > >>> - if that's the route you go I could probably help with a thorough
> > secure
> > >>> code review.
> > >>>
> > >>> Jon
> > >>>
> > >>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
> > >wrote:
> > >>>
> > >>> Thank you everyone for your suggestions,
> > >>>
> > >>> I believe that kicking off the function via stellar would be the
> > optimal
> > >>> solution. If anyone has an example of calling external code via
> stellar
> > >>> that would be very helpful. Thanks!
> > >>>
> > >>> Regards,
> > >>>
> > >>> Tyler Moore
> > >>> IT Specialist
> > >>> tyler.mathieu@yahoo.com
> > >>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> <(248)%20909-2769>
> > >>>
> > >>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com>
> > wrote:
> > >>> >
> > >>> > They are all extension points.
> > >>> >
> > >>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (
> > zeolla@gmail.com)
> > >>> wrote:
> > >>> >>
> > >>> >> Right but unless I'm missing something, both of those options are
> > more
> > >>> >> rigid and the MaaS service would have an unnecessary delay as
> > opposed
> > >to
> > >>> >> doing it entirely in Stellar. Unless there's a reason to do
> > otherwise
> > >>> that
> > >>> >> I'm missing, I would think doing this in Stellar gives you a more
> > >timely
> > >>> >> and (re)configurable end result.
> > >>> >>
> > >>> >> Jon
> > >>> >>
> > >>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com
> >
> > >>> wrote:
> > >>> >>>
> > >>> >>> I think there are a couple of things you can do here. There way
> to
> > >get
> > >>> >>> something else into the split is to have another adapter to split
> > to,
> > >>> which
> > >>> >>> is what I think you mean. You can also integrate with MaaS and
> > create
> > >>> a
> > >>> >>> service that you can call via STELLAR.
> > >>> >>>
> > >>> >>>
> > >>> >>>
> > >>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
> > >ottobackwards@gmail.com
> > >>> )
> > >>> >>> wrote:
> > >>> >>>
> > >>> >>> Or a Maas service?
> > >>> >>>
> > >>> >>>
> > >>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (
> > zeolla@gmail.com)
> > >>> >>> wrote:
> > >>> >>>
> > >>> >>> Depending on the details it sounds like a much simpler solution
> > would
> > >>> be
> > >>> >>> to
> > >>> >>> handle this in a Stellar function.
> > >>> >>>
> > >>> >>> Jon
> > >>> >>>
> > >>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
> > >wrote:
> > >>> >>>>
> > >>> >>>> Happy Holidays Metron Devs!
> > >>> >>>>
> > >>> >>>> Could anyone lend me some guidance on customizing the storm
> > >topologies
> > >>> >>> in
> > >>> >>>> metron? What I am am trying to accomplish:
> > >>> >>>>
> > >>> >>>> 1) Add a method to the threat intel joiner bolt that sends an
> http
> > >>> post
> > >>> >>>> with the score of the threat to a remote rest api. This will
> > >>> >>> conditionally
> > >>> >>>> trigger notifications based on user settings in another database
> > >(the
> > >>> >>>> backend processing logic is on another platform).
> > >>> >>>> The score should be available within the JSONObject but I am not
> > an
> > >>> >>> expert
> > >>> >>>> with storm and I am not completely understanding what conditions
> > >>> >>> constitute
> > >>> >>>> when the threat feed is considered an "alert" in metron. Please
> > >>> clarify.
> > >>> >>>>
> > >>> >>>> 2) How would I add an external dependency, my http rest java
> > class,
> > >to
> > >>> >>> the
> > >>> >>>> metron maven build process? More specifically, if I was adding a
> > >>> custom
> > >>> >>>> class that needed accessed by a bolt in storm, how would I add
> > this
> > >in
> > >>> >>>> maven as a dependency. I have limited experience with maven but,
> > my
> > >>> >>>> understanding is that I would add it to the pom.xml and
> > recompile.
> > >>> >>>> Although, the metron quick dev platform is built on a vm, would
> I
> > >need
> > >>> >>> to
> > >>> >>>> account for this? Please advise.
> > >>> >>>>
> > >>> >>>> Regards,
> > >>> >>>>
> > >>> >>>> Tyler Moore
> > >>> >>>>
> > >>> >>>>
> > >>> >>>> Software Engineer
> > >>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> <(248)%20909-2769>
> > >>> >>>> Email: moore.tyler@goflyball.com
> > >>> >>>>
> > >>> >>> --
> > >>> >>>
> > >>> >>> Jon
> > >>> >>>
> > >>> >>> Sent from my mobile device
> > >>> >>>
> > >>> >>> --
> > >>> >>
> > >>> >> Jon
> > >>> >>
> > >>> >> Sent from my mobile device
> > >>> >>
> > >>>
> > >>> --
> > >>>
> > >>> Jon
> > >>>
> > >>> Sent from my mobile device
> > >>>
> > >
> > >
> > >
> > >
> > >
> > >--
> > >
> > >Jon
> > >
> > >Sent from my mobile device
> >
>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by Tyler Moore <tm...@goflyball.com>.
I attached a sample function to the Jira, just calls a script from on the
server by passing in the path. Certainly could use some work, but I wanted
to make sure this was the right idea and get a consensus, let me know your
thoughts.
My thought process for this is that if we only allow script execution from
a shell script that must be on the storm host, then the user is in a sense
authenticated by their own user-defined access restrictions for that system.
Regards,
Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com
On Tue, Jan 3, 2017 at 3:47 PM, Carolyn Duby <cd...@hortonworks.com> wrote:
> Also please consider the security of the scripts and script injection
> attacks. For example, we should probably restrict file access.
>
> Thanks
> Carolyn
>
>
>
> On 1/3/17, 3:25 PM, "Otto Fowler" <ot...@gmail.com> wrote:
>
> >A script bolt would still allow them to write the script the way they want
> >to, but would avoid having to write all the scaffolding.
> >The matter then would be how to integrate that script bolt into the
> >topologies.
> >
> >
> >On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
> >
> >Right, that definitely is more efficient, but part of the point here is to
> >lower the barrier of entry to using Metron.
> >
> >It makes Metron's triage abilities more flexible and allows a user to
> reuse
> >existing code quickly and easily. Having this available for PoC,
> >prototyping, and low volume environments or situations (only when threat
> >score is 100, for instance) is important, as it lowers the barrier to
> entry
> >of migrating a company to a Metron environment.
> >
> >I see this as a tradeoff where I would prioritize ease of use over
> >efficiency. There's nothing wrong with making both options available, at
> >some point, and making their different use cases clear.
> >
> >Jon
> >
> >On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
> >
> >Well, yes :-)
> >And clearly it should always be more efficient to write a custom bolt in
> >Java than to invoke a script and manage it.
> >
> >--Matt
> >
> >From: Otto Fowler <ot...@gmail.com>
> >Date: Tuesday, January 3, 2017 at 7:08 AM
> >To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
> >Matt Foley <ma...@apache.org>
> >Subject: Re: Custom Storm Topologies
> >
> >Wouldn’t that be a bolt?
> >
> >
> >On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
> >Should we consider a script calling capability that can launch a streaming
> >script and keep it alive and fed, long-term, rather than launching the
> >script anew every time the Stellar function is invoked? I’m thinking two
> >basic rules: Write a line, read a line; and always have a timeout. Prob
> >need a UID of some sort for a cache of running process objects.
> >
> >--Matt
> >
> >On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
> >
> >
> >Inserting a script inline is ok for low throughput and prototyping but
> once
> >you get higher throughput (millions of events per second), it’s probably
> >going to be a bottleneck.
> >
> >
> >For Metron-571 you might want to consider a java based extension plugin
> >similar to Eclipse plugins.
> >
> >Thanks
> >Carolyn
> >
> >On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
> >
> >>Thanks Jon,
> >>
> >>I'll look over the tutorial and put something together for the SHELL_EXEC
> >>stellar function.
> >>I don't believe I have permissions to assign in Jira if you want to
> assign
> >>to me my username is devopsec.
> >>I'll post back details and we can review security issues
> >>
> >>Regards,
> >>
> >>Tyler Moore
> >>Software Engineer
> >>Phone: 248-909-2769 <(248)%20909-2769>
> >>Email: moore.tyler@goflyball.com
> >>
> >>
> >>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >>
> >>> Casey did a tutorial on how to add your own Stellar function here
> >>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an
> existing
> >>> function that does this (current functions are listed here
> >>> <https://github.com/apache/incubator-metron/tree/master/
> >>> metron-platform/metron-common#stellar-core-functions>).
> >>> I noticed that some of the Stellar function documentation was a bit
> dated
> >>> so I've opened a PR to update it here
> >>> <https://github.com/apache/incubator-metron/pull/407>.
> >>>
> >>> As this is something I need as well, I'd be happy to assist you where I
> >>> can. Perhaps you want to self-assign METRON-571
> >>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
> >>> security concerns with a SHELL_EXEC function because it could result in
> >RCE
> >>> - if that's the route you go I could probably help with a thorough
> secure
> >>> code review.
> >>>
> >>> Jon
> >>>
> >>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
> >wrote:
> >>>
> >>> Thank you everyone for your suggestions,
> >>>
> >>> I believe that kicking off the function via stellar would be the
> optimal
> >>> solution. If anyone has an example of calling external code via stellar
> >>> that would be very helpful. Thanks!
> >>>
> >>> Regards,
> >>>
> >>> Tyler Moore
> >>> IT Specialist
> >>> tyler.mathieu@yahoo.com
> >>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> >>>
> >>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com>
> wrote:
> >>> >
> >>> > They are all extension points.
> >>> >
> >>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (
> zeolla@gmail.com)
> >>> wrote:
> >>> >>
> >>> >> Right but unless I'm missing something, both of those options are
> more
> >>> >> rigid and the MaaS service would have an unnecessary delay as
> opposed
> >to
> >>> >> doing it entirely in Stellar. Unless there's a reason to do
> otherwise
> >>> that
> >>> >> I'm missing, I would think doing this in Stellar gives you a more
> >timely
> >>> >> and (re)configurable end result.
> >>> >>
> >>> >> Jon
> >>> >>
> >>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
> >>> wrote:
> >>> >>>
> >>> >>> I think there are a couple of things you can do here. There way to
> >get
> >>> >>> something else into the split is to have another adapter to split
> to,
> >>> which
> >>> >>> is what I think you mean. You can also integrate with MaaS and
> create
> >>> a
> >>> >>> service that you can call via STELLAR.
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
> >ottobackwards@gmail.com
> >>> )
> >>> >>> wrote:
> >>> >>>
> >>> >>> Or a Maas service?
> >>> >>>
> >>> >>>
> >>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (
> zeolla@gmail.com)
> >>> >>> wrote:
> >>> >>>
> >>> >>> Depending on the details it sounds like a much simpler solution
> would
> >>> be
> >>> >>> to
> >>> >>> handle this in a Stellar function.
> >>> >>>
> >>> >>> Jon
> >>> >>>
> >>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
> >wrote:
> >>> >>>>
> >>> >>>> Happy Holidays Metron Devs!
> >>> >>>>
> >>> >>>> Could anyone lend me some guidance on customizing the storm
> >topologies
> >>> >>> in
> >>> >>>> metron? What I am am trying to accomplish:
> >>> >>>>
> >>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
> >>> post
> >>> >>>> with the score of the threat to a remote rest api. This will
> >>> >>> conditionally
> >>> >>>> trigger notifications based on user settings in another database
> >(the
> >>> >>>> backend processing logic is on another platform).
> >>> >>>> The score should be available within the JSONObject but I am not
> an
> >>> >>> expert
> >>> >>>> with storm and I am not completely understanding what conditions
> >>> >>> constitute
> >>> >>>> when the threat feed is considered an "alert" in metron. Please
> >>> clarify.
> >>> >>>>
> >>> >>>> 2) How would I add an external dependency, my http rest java
> class,
> >to
> >>> >>> the
> >>> >>>> metron maven build process? More specifically, if I was adding a
> >>> custom
> >>> >>>> class that needed accessed by a bolt in storm, how would I add
> this
> >in
> >>> >>>> maven as a dependency. I have limited experience with maven but,
> my
> >>> >>>> understanding is that I would add it to the pom.xml and
> recompile.
> >>> >>>> Although, the metron quick dev platform is built on a vm, would I
> >need
> >>> >>> to
> >>> >>>> account for this? Please advise.
> >>> >>>>
> >>> >>>> Regards,
> >>> >>>>
> >>> >>>> Tyler Moore
> >>> >>>>
> >>> >>>>
> >>> >>>> Software Engineer
> >>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> >>> >>>> Email: moore.tyler@goflyball.com
> >>> >>>>
> >>> >>> --
> >>> >>>
> >>> >>> Jon
> >>> >>>
> >>> >>> Sent from my mobile device
> >>> >>>
> >>> >>> --
> >>> >>
> >>> >> Jon
> >>> >>
> >>> >> Sent from my mobile device
> >>> >>
> >>>
> >>> --
> >>>
> >>> Jon
> >>>
> >>> Sent from my mobile device
> >>>
> >
> >
> >
> >
> >
> >--
> >
> >Jon
> >
> >Sent from my mobile device
>
Re: Custom Storm Topologies
Posted by Carolyn Duby <cd...@hortonworks.com>.
Also please consider the security of the scripts and script injection attacks. For example, we should probably restrict file access.
Thanks
Carolyn
On 1/3/17, 3:25 PM, "Otto Fowler" <ot...@gmail.com> wrote:
>A script bolt would still allow them to write the script the way they want
>to, but would avoid having to write all the scaffolding.
>The matter then would be how to integrate that script bolt into the
>topologies.
>
>
>On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com) wrote:
>
>Right, that definitely is more efficient, but part of the point here is to
>lower the barrier of entry to using Metron.
>
>It makes Metron's triage abilities more flexible and allows a user to reuse
>existing code quickly and easily. Having this available for PoC,
>prototyping, and low volume environments or situations (only when threat
>score is 100, for instance) is important, as it lowers the barrier to entry
>of migrating a company to a Metron environment.
>
>I see this as a tradeoff where I would prioritize ease of use over
>efficiency. There's nothing wrong with making both options available, at
>some point, and making their different use cases clear.
>
>Jon
>
>On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
>
>Well, yes :-)
>And clearly it should always be more efficient to write a custom bolt in
>Java than to invoke a script and manage it.
>
>--Matt
>
>From: Otto Fowler <ot...@gmail.com>
>Date: Tuesday, January 3, 2017 at 7:08 AM
>To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
>Matt Foley <ma...@apache.org>
>Subject: Re: Custom Storm Topologies
>
>Wouldn’t that be a bolt?
>
>
>On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
>Should we consider a script calling capability that can launch a streaming
>script and keep it alive and fed, long-term, rather than launching the
>script anew every time the Stellar function is invoked? I’m thinking two
>basic rules: Write a line, read a line; and always have a timeout. Prob
>need a UID of some sort for a cache of running process objects.
>
>--Matt
>
>On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
>
>
>Inserting a script inline is ok for low throughput and prototyping but once
>you get higher throughput (millions of events per second), it’s probably
>going to be a bottleneck.
>
>
>For Metron-571 you might want to consider a java based extension plugin
>similar to Eclipse plugins.
>
>Thanks
>Carolyn
>
>On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>
>>Thanks Jon,
>>
>>I'll look over the tutorial and put something together for the SHELL_EXEC
>>stellar function.
>>I don't believe I have permissions to assign in Jira if you want to assign
>>to me my username is devopsec.
>>I'll post back details and we can review security issues
>>
>>Regards,
>>
>>Tyler Moore
>>Software Engineer
>>Phone: 248-909-2769 <(248)%20909-2769>
>>Email: moore.tyler@goflyball.com
>>
>>
>>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>>
>>> Casey did a tutorial on how to add your own Stellar function here
>>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>>> function that does this (current functions are listed here
>>> <https://github.com/apache/incubator-metron/tree/master/
>>> metron-platform/metron-common#stellar-core-functions>).
>>> I noticed that some of the Stellar function documentation was a bit dated
>>> so I've opened a PR to update it here
>>> <https://github.com/apache/incubator-metron/pull/407>.
>>>
>>> As this is something I need as well, I'd be happy to assist you where I
>>> can. Perhaps you want to self-assign METRON-571
>>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>>> security concerns with a SHELL_EXEC function because it could result in
>RCE
>>> - if that's the route you go I could probably help with a thorough secure
>>> code review.
>>>
>>> Jon
>>>
>>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
>wrote:
>>>
>>> Thank you everyone for your suggestions,
>>>
>>> I believe that kicking off the function via stellar would be the optimal
>>> solution. If anyone has an example of calling external code via stellar
>>> that would be very helpful. Thanks!
>>>
>>> Regards,
>>>
>>> Tyler Moore
>>> IT Specialist
>>> tyler.mathieu@yahoo.com
>>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>>
>>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>>> >
>>> > They are all extension points.
>>> >
>>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>>> wrote:
>>> >>
>>> >> Right but unless I'm missing something, both of those options are more
>>> >> rigid and the MaaS service would have an unnecessary delay as opposed
>to
>>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>>> that
>>> >> I'm missing, I would think doing this in Stellar gives you a more
>timely
>>> >> and (re)configurable end result.
>>> >>
>>> >> Jon
>>> >>
>>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> I think there are a couple of things you can do here. There way to
>get
>>> >>> something else into the split is to have another adapter to split to,
>>> which
>>> >>> is what I think you mean. You can also integrate with MaaS and create
>>> a
>>> >>> service that you can call via STELLAR.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
>ottobackwards@gmail.com
>>> )
>>> >>> wrote:
>>> >>>
>>> >>> Or a Maas service?
>>> >>>
>>> >>>
>>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>>> >>> wrote:
>>> >>>
>>> >>> Depending on the details it sounds like a much simpler solution would
>>> be
>>> >>> to
>>> >>> handle this in a Stellar function.
>>> >>>
>>> >>> Jon
>>> >>>
>>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
>wrote:
>>> >>>>
>>> >>>> Happy Holidays Metron Devs!
>>> >>>>
>>> >>>> Could anyone lend me some guidance on customizing the storm
>topologies
>>> >>> in
>>> >>>> metron? What I am am trying to accomplish:
>>> >>>>
>>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>>> post
>>> >>>> with the score of the threat to a remote rest api. This will
>>> >>> conditionally
>>> >>>> trigger notifications based on user settings in another database
>(the
>>> >>>> backend processing logic is on another platform).
>>> >>>> The score should be available within the JSONObject but I am not an
>>> >>> expert
>>> >>>> with storm and I am not completely understanding what conditions
>>> >>> constitute
>>> >>>> when the threat feed is considered an "alert" in metron. Please
>>> clarify.
>>> >>>>
>>> >>>> 2) How would I add an external dependency, my http rest java class,
>to
>>> >>> the
>>> >>>> metron maven build process? More specifically, if I was adding a
>>> custom
>>> >>>> class that needed accessed by a bolt in storm, how would I add this
>in
>>> >>>> maven as a dependency. I have limited experience with maven but, my
>>> >>>> understanding is that I would add it to the pom.xml and recompile.
>>> >>>> Although, the metron quick dev platform is built on a vm, would I
>need
>>> >>> to
>>> >>>> account for this? Please advise.
>>> >>>>
>>> >>>> Regards,
>>> >>>>
>>> >>>> Tyler Moore
>>> >>>>
>>> >>>>
>>> >>>> Software Engineer
>>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>> >>>> Email: moore.tyler@goflyball.com
>>> >>>>
>>> >>> --
>>> >>>
>>> >>> Jon
>>> >>>
>>> >>> Sent from my mobile device
>>> >>>
>>> >>> --
>>> >>
>>> >> Jon
>>> >>
>>> >> Sent from my mobile device
>>> >>
>>>
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>
>
>
>
>
>--
>
>Jon
>
>Sent from my mobile device
Re: Custom Storm Topologies
Posted by Otto Fowler <ot...@gmail.com>.
A script bolt would still allow them to write the script the way they want
to, but would avoid having to write all the scaffolding.
The matter then would be how to integrate that script bolt into the
topologies.
On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com) wrote:
Right, that definitely is more efficient, but part of the point here is to
lower the barrier of entry to using Metron.
It makes Metron's triage abilities more flexible and allows a user to reuse
existing code quickly and easily. Having this available for PoC,
prototyping, and low volume environments or situations (only when threat
score is 100, for instance) is important, as it lowers the barrier to entry
of migrating a company to a Metron environment.
I see this as a tradeoff where I would prioritize ease of use over
efficiency. There's nothing wrong with making both options available, at
some point, and making their different use cases clear.
Jon
On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
Well, yes :-)
And clearly it should always be more efficient to write a custom bolt in
Java than to invoke a script and manage it.
--Matt
From: Otto Fowler <ot...@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
Matt Foley <ma...@apache.org>
Subject: Re: Custom Storm Topologies
Wouldn’t that be a bolt?
On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.
--Matt
On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.
For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.
Thanks
Carolyn
On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769 <(248)%20909-2769>
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Tyler Moore
>> >>>>
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Right, that definitely is more efficient, but part of the point here is to
lower the barrier of entry to using Metron.
It makes Metron's triage abilities more flexible and allows a user to reuse
existing code quickly and easily. Having this available for PoC,
prototyping, and low volume environments or situations (only when threat
score is 100, for instance) is important, as it lowers the barrier to entry
of migrating a company to a Metron environment.
I see this as a tradeoff where I would prioritize ease of use over
efficiency. There's nothing wrong with making both options available, at
some point, and making their different use cases clear.
Jon
On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
Well, yes :-)
And clearly it should always be more efficient to write a custom bolt in
Java than to invoke a script and manage it.
--Matt
From: Otto Fowler <ot...@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
Matt Foley <ma...@apache.org>
Subject: Re: Custom Storm Topologies
Wouldn’t that be a bolt?
On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.
--Matt
On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.
For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.
Thanks
Carolyn
On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769 <(248)%20909-2769>
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Tyler Moore
>> >>>>
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by Matt Foley <ma...@apache.org>.
Well, yes :-)
And clearly it should always be more efficient to write a custom bolt in Java than to invoke a script and manage it.
--Matt
From: Otto Fowler <ot...@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>, Matt Foley <ma...@apache.org>
Subject: Re: Custom Storm Topologies
Wouldn’t that be a bolt?
On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming script and keep it alive and fed, long-term, rather than launching the script anew every time the Stellar function is invoked? I’m thinking two basic rules: Write a line, read a line; and always have a timeout. Prob need a UID of some sort for a cache of running process objects.
--Matt
On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck.
For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins.
Thanks
Carolyn
On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database (the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class, to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Tyler Moore
>> >>>>
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
Re: Custom Storm Topologies
Posted by Otto Fowler <ot...@gmail.com>.
Wouldn’t that be a bolt?
On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.
--Matt
On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.
For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.
Thanks
Carolyn
On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit
dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough
secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com>
wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are
more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split
to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and
create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution
would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Tyler Moore
>> >>>>
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
Re: Custom Storm Topologies
Posted by Matt Foley <ma...@apache.org>.
Should we consider a script calling capability that can launch a streaming script and keep it alive and fed, long-term, rather than launching the script anew every time the Stellar function is invoked? I’m thinking two basic rules: Write a line, read a line; and always have a timeout. Prob need a UID of some sort for a cache of running process objects.
--Matt
On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck.
For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins.
Thanks
Carolyn
On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database (the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class, to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Tyler Moore
>> >>>>
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
Re: Custom Storm Topologies
Posted by Carolyn Duby <cd...@hortonworks.com>.
Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck.
For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins.
Thanks
Carolyn
On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database (the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class, to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> Regards,
>> >>>>
>> >>>> Tyler Moore
>> >>>>
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
Re: Custom Storm Topologies
Posted by Tyler Moore <tm...@goflyball.com>.
Thanks Jon,
I'll look over the tutorial and put something together for the SHELL_EXEC
stellar function.
I don't believe I have permissions to assign in Jira if you want to assign
to me my username is devopsec.
I'll post back details and we can review security issues
Regards,
Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com
On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
> Casey did a tutorial on how to add your own Stellar function here
> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
> function that does this (current functions are listed here
> <https://github.com/apache/incubator-metron/tree/master/
> metron-platform/metron-common#stellar-core-functions>).
> I noticed that some of the Stellar function documentation was a bit dated
> so I've opened a PR to update it here
> <https://github.com/apache/incubator-metron/pull/407>.
>
> As this is something I need as well, I'd be happy to assist you where I
> can. Perhaps you want to self-assign METRON-571
> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
> security concerns with a SHELL_EXEC function because it could result in RCE
> - if that's the route you go I could probably help with a thorough secure
> code review.
>
> Jon
>
> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
>
> Thank you everyone for your suggestions,
>
> I believe that kicking off the function via stellar would be the optimal
> solution. If anyone has an example of calling external code via stellar
> that would be very helpful. Thanks!
>
> Regards,
>
> Tyler Moore
> IT Specialist
> tyler.mathieu@yahoo.com
> 248-909-2769 <(248)%20909-2769>
>
> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
> >
> > They are all extension points.
> >
> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
> >>
> >> Right but unless I'm missing something, both of those options are more
> >> rigid and the MaaS service would have an unnecessary delay as opposed to
> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
> that
> >> I'm missing, I would think doing this in Stellar gives you a more timely
> >> and (re)configurable end result.
> >>
> >> Jon
> >>
> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
> wrote:
> >>>
> >>> I think there are a couple of things you can do here. There way to get
> >>> something else into the split is to have another adapter to split to,
> which
> >>> is what I think you mean. You can also integrate with MaaS and create
> a
> >>> service that you can call via STELLAR.
> >>>
> >>>
> >>>
> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
> )
> >>> wrote:
> >>>
> >>> Or a Maas service?
> >>>
> >>>
> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
> >>> wrote:
> >>>
> >>> Depending on the details it sounds like a much simpler solution would
> be
> >>> to
> >>> handle this in a Stellar function.
> >>>
> >>> Jon
> >>>
> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
> >>>>
> >>>> Happy Holidays Metron Devs!
> >>>>
> >>>> Could anyone lend me some guidance on customizing the storm topologies
> >>> in
> >>>> metron? What I am am trying to accomplish:
> >>>>
> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
> post
> >>>> with the score of the threat to a remote rest api. This will
> >>> conditionally
> >>>> trigger notifications based on user settings in another database (the
> >>>> backend processing logic is on another platform).
> >>>> The score should be available within the JSONObject but I am not an
> >>> expert
> >>>> with storm and I am not completely understanding what conditions
> >>> constitute
> >>>> when the threat feed is considered an "alert" in metron. Please
> clarify.
> >>>>
> >>>> 2) How would I add an external dependency, my http rest java class, to
> >>> the
> >>>> metron maven build process? More specifically, if I was adding a
> custom
> >>>> class that needed accessed by a bolt in storm, how would I add this in
> >>>> maven as a dependency. I have limited experience with maven but, my
> >>>> understanding is that I would add it to the pom.xml and recompile.
> >>>> Although, the metron quick dev platform is built on a vm, would I need
> >>> to
> >>>> account for this? Please advise.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Tyler Moore
> >>>>
> >>>>
> >>>> Software Engineer
> >>>> Phone: 248-909-2769 <(248)%20909-2769>
> >>>> Email: moore.tyler@goflyball.com
> >>>>
> >>> --
> >>>
> >>> Jon
> >>>
> >>> Sent from my mobile device
> >>>
> >>> --
> >>
> >> Jon
> >>
> >> Sent from my mobile device
> >>
>
> --
>
> Jon
>
> Sent from my mobile device
>
Re: Custom Storm Topologies
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Casey did a tutorial on how to add your own Stellar function here
<https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
function that does this (current functions are listed here
<https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#stellar-core-functions>).
I noticed that some of the Stellar function documentation was a bit dated
so I've opened a PR to update it here
<https://github.com/apache/incubator-metron/pull/407>.
As this is something I need as well, I'd be happy to assist you where I
can. Perhaps you want to self-assign METRON-571
<https://issues.apache.org/jira/browse/METRON-571>? I do have some
security concerns with a SHELL_EXEC function because it could result in RCE
- if that's the route you go I could probably help with a thorough secure
code review.
Jon
On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
Thank you everyone for your suggestions,
I believe that kicking off the function via stellar would be the optimal
solution. If anyone has an example of calling external code via stellar
that would be very helpful. Thanks!
Regards,
Tyler Moore
IT Specialist
tyler.mathieu@yahoo.com
248-909-2769 <(248)%20909-2769>
> On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>
> They are all extension points.
>
>> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
wrote:
>>
>> Right but unless I'm missing something, both of those options are more
>> rigid and the MaaS service would have an unnecessary delay as opposed to
>> doing it entirely in Stellar. Unless there's a reason to do otherwise
that
>> I'm missing, I would think doing this in Stellar gives you a more timely
>> and (re)configurable end result.
>>
>> Jon
>>
>>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com> wrote:
>>>
>>> I think there are a couple of things you can do here. There way to get
>>> something else into the split is to have another adapter to split to,
which
>>> is what I think you mean. You can also integrate with MaaS and create a
>>> service that you can call via STELLAR.
>>>
>>>
>>>
>>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com)
>>> wrote:
>>>
>>> Or a Maas service?
>>>
>>>
>>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>>> wrote:
>>>
>>> Depending on the details it sounds like a much simpler solution would be
>>> to
>>> handle this in a Stellar function.
>>>
>>> Jon
>>>
>>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>>>>
>>>> Happy Holidays Metron Devs!
>>>>
>>>> Could anyone lend me some guidance on customizing the storm topologies
>>> in
>>>> metron? What I am am trying to accomplish:
>>>>
>>>> 1) Add a method to the threat intel joiner bolt that sends an http post
>>>> with the score of the threat to a remote rest api. This will
>>> conditionally
>>>> trigger notifications based on user settings in another database (the
>>>> backend processing logic is on another platform).
>>>> The score should be available within the JSONObject but I am not an
>>> expert
>>>> with storm and I am not completely understanding what conditions
>>> constitute
>>>> when the threat feed is considered an "alert" in metron. Please
clarify.
>>>>
>>>> 2) How would I add an external dependency, my http rest java class, to
>>> the
>>>> metron maven build process? More specifically, if I was adding a custom
>>>> class that needed accessed by a bolt in storm, how would I add this in
>>>> maven as a dependency. I have limited experience with maven but, my
>>>> understanding is that I would add it to the pom.xml and recompile.
>>>> Although, the metron quick dev platform is built on a vm, would I need
>>> to
>>>> account for this? Please advise.
>>>>
>>>> Regards,
>>>>
>>>> Tyler Moore
>>>>
>>>>
>>>> Software Engineer
>>>> Phone: 248-909-2769 <(248)%20909-2769>
>>>> Email: moore.tyler@goflyball.com
>>>>
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by Tyler Moore <tm...@goflyball.com>.
Thank you everyone for your suggestions,
I believe that kicking off the function via stellar would be the optimal solution. If anyone has an example of calling external code via stellar that would be very helpful. Thanks!
Regards,
Tyler Moore
IT Specialist
tyler.mathieu@yahoo.com
248-909-2769
> On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>
> They are all extension points.
>
>> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com) wrote:
>>
>> Right but unless I'm missing something, both of those options are more
>> rigid and the MaaS service would have an unnecessary delay as opposed to
>> doing it entirely in Stellar. Unless there's a reason to do otherwise that
>> I'm missing, I would think doing this in Stellar gives you a more timely
>> and (re)configurable end result.
>>
>> Jon
>>
>>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com> wrote:
>>>
>>> I think there are a couple of things you can do here. There way to get
>>> something else into the split is to have another adapter to split to, which
>>> is what I think you mean. You can also integrate with MaaS and create a
>>> service that you can call via STELLAR.
>>>
>>>
>>>
>>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com)
>>> wrote:
>>>
>>> Or a Maas service?
>>>
>>>
>>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>>> wrote:
>>>
>>> Depending on the details it sounds like a much simpler solution would be
>>> to
>>> handle this in a Stellar function.
>>>
>>> Jon
>>>
>>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>>>>
>>>> Happy Holidays Metron Devs!
>>>>
>>>> Could anyone lend me some guidance on customizing the storm topologies
>>> in
>>>> metron? What I am am trying to accomplish:
>>>>
>>>> 1) Add a method to the threat intel joiner bolt that sends an http post
>>>> with the score of the threat to a remote rest api. This will
>>> conditionally
>>>> trigger notifications based on user settings in another database (the
>>>> backend processing logic is on another platform).
>>>> The score should be available within the JSONObject but I am not an
>>> expert
>>>> with storm and I am not completely understanding what conditions
>>> constitute
>>>> when the threat feed is considered an "alert" in metron. Please clarify.
>>>>
>>>> 2) How would I add an external dependency, my http rest java class, to
>>> the
>>>> metron maven build process? More specifically, if I was adding a custom
>>>> class that needed accessed by a bolt in storm, how would I add this in
>>>> maven as a dependency. I have limited experience with maven but, my
>>>> understanding is that I would add it to the pom.xml and recompile.
>>>> Although, the metron quick dev platform is built on a vm, would I need
>>> to
>>>> account for this? Please advise.
>>>>
>>>> Regards,
>>>>
>>>> Tyler Moore
>>>>
>>>>
>>>> Software Engineer
>>>> Phone: 248-909-2769
>>>> Email: moore.tyler@goflyball.com
>>>>
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
Re: Custom Storm Topologies
Posted by Otto Fowler <ot...@gmail.com>.
They are all extension points.
On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com) wrote:
> Right but unless I'm missing something, both of those options are more
> rigid and the MaaS service would have an unnecessary delay as opposed to
> doing it entirely in Stellar. Unless there's a reason to do otherwise that
> I'm missing, I would think doing this in Stellar gives you a more timely
> and (re)configurable end result.
>
> Jon
>
> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com> wrote:
>
>> I think there are a couple of things you can do here. There way to get
>> something else into the split is to have another adapter to split to, which
>> is what I think you mean. You can also integrate with MaaS and create a
>> service that you can call via STELLAR.
>>
>>
>>
>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com)
>> wrote:
>>
>> Or a Maas service?
>>
>>
>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>>
>> Depending on the details it sounds like a much simpler solution would be
>> to
>> handle this in a Stellar function.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>>
>> > Happy Holidays Metron Devs!
>> >
>> > Could anyone lend me some guidance on customizing the storm topologies
>> in
>> > metron? What I am am trying to accomplish:
>> >
>> > 1) Add a method to the threat intel joiner bolt that sends an http post
>> > with the score of the threat to a remote rest api. This will
>> conditionally
>> > trigger notifications based on user settings in another database (the
>> > backend processing logic is on another platform).
>> > The score should be available within the JSONObject but I am not an
>> expert
>> > with storm and I am not completely understanding what conditions
>> constitute
>> > when the threat feed is considered an "alert" in metron. Please clarify.
>> >
>> > 2) How would I add an external dependency, my http rest java class, to
>> the
>> > metron maven build process? More specifically, if I was adding a custom
>> > class that needed accessed by a bolt in storm, how would I add this in
>> > maven as a dependency. I have limited experience with maven but, my
>> > understanding is that I would add it to the pom.xml and recompile.
>> > Although, the metron quick dev platform is built on a vm, would I need
>> to
>> > account for this? Please advise.
>> >
>> > Regards,
>> >
>> > Tyler Moore
>> >
>> >
>> > Software Engineer
>> > Phone: 248-909-2769
>> > Email: moore.tyler@goflyball.com
>> >
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>
>> --
>
> Jon
>
> Sent from my mobile device
>
Re: Custom Storm Topologies
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Right but unless I'm missing something, both of those options are more
rigid and the MaaS service would have an unnecessary delay as opposed to
doing it entirely in Stellar. Unless there's a reason to do otherwise that
I'm missing, I would think doing this in Stellar gives you a more timely
and (re)configurable end result.
Jon
On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com> wrote:
> I think there are a couple of things you can do here. There way to get
> something else into the split is to have another adapter to split to, which
> is what I think you mean. You can also integrate with MaaS and create a
> service that you can call via STELLAR.
>
>
>
> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com)
> wrote:
>
> Or a Maas service?
>
>
> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> Depending on the details it sounds like a much simpler solution would be to
> handle this in a Stellar function.
>
> Jon
>
> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>
> > Happy Holidays Metron Devs!
> >
> > Could anyone lend me some guidance on customizing the storm topologies in
> > metron? What I am am trying to accomplish:
> >
> > 1) Add a method to the threat intel joiner bolt that sends an http post
> > with the score of the threat to a remote rest api. This will
> conditionally
> > trigger notifications based on user settings in another database (the
> > backend processing logic is on another platform).
> > The score should be available within the JSONObject but I am not an
> expert
> > with storm and I am not completely understanding what conditions
> constitute
> > when the threat feed is considered an "alert" in metron. Please clarify.
> >
> > 2) How would I add an external dependency, my http rest java class, to
> the
> > metron maven build process? More specifically, if I was adding a custom
> > class that needed accessed by a bolt in storm, how would I add this in
> > maven as a dependency. I have limited experience with maven but, my
> > understanding is that I would add it to the pom.xml and recompile.
> > Although, the metron quick dev platform is built on a vm, would I need to
> > account for this? Please advise.
> >
> > Regards,
> >
> > Tyler Moore
> >
> >
> > Software Engineer
> > Phone: 248-909-2769
> > Email: moore.tyler@goflyball.com
> >
> --
>
> Jon
>
> Sent from my mobile device
>
> --
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by Otto Fowler <ot...@gmail.com>.
I think there are a couple of things you can do here. There way to get
something else into the split is to have another adapter to split to, which
is what I think you mean. You can also integrate with MaaS and create a
service that you can call via STELLAR.
On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com)
wrote:
Or a Maas service?
On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com) wrote:
Depending on the details it sounds like a much simpler solution would be to
handle this in a Stellar function.
Jon
On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
> Happy Holidays Metron Devs!
>
> Could anyone lend me some guidance on customizing the storm topologies in
> metron? What I am am trying to accomplish:
>
> 1) Add a method to the threat intel joiner bolt that sends an http post
> with the score of the threat to a remote rest api. This will conditionally
> trigger notifications based on user settings in another database (the
> backend processing logic is on another platform).
> The score should be available within the JSONObject but I am not an expert
> with storm and I am not completely understanding what conditions
constitute
> when the threat feed is considered an "alert" in metron. Please clarify.
>
> 2) How would I add an external dependency, my http rest java class, to the
> metron maven build process? More specifically, if I was adding a custom
> class that needed accessed by a bolt in storm, how would I add this in
> maven as a dependency. I have limited experience with maven but, my
> understanding is that I would add it to the pom.xml and recompile.
> Although, the metron quick dev platform is built on a vm, would I need to
> account for this? Please advise.
>
> Regards,
>
> Tyler Moore
>
>
> Software Engineer
> Phone: 248-909-2769
> Email: moore.tyler@goflyball.com
>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by Simon Elliston Ball <si...@simonellistonball.com>.
Sounds like what you're trying to do is apply different threat scores based on a user identified at the parser or enrichment stage. What I would do is to add the scoring factor as an enrichment, and then make the final threat triage expression a combination of the rest result called by the enrichment, and whatever the threat would have been from other sources, or potentially just use enrichment calls the whole rest piece and keep the threat triage simple. This may not work if you absolutely need it to be post threat Intel, in which case, as others have suggested, a stellar function may be the answer.
Simon
> On 30 Dec 2016, at 20:08, Otto Fowler <ot...@gmail.com> wrote:
>
> Or a Maas service?
>
>
> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com) wrote:
>
> Depending on the details it sounds like a much simpler solution would be to
> handle this in a Stellar function.
>
> Jon
>
>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>>
>> Happy Holidays Metron Devs!
>>
>> Could anyone lend me some guidance on customizing the storm topologies in
>> metron? What I am am trying to accomplish:
>>
>> 1) Add a method to the threat intel joiner bolt that sends an http post
>> with the score of the threat to a remote rest api. This will conditionally
>> trigger notifications based on user settings in another database (the
>> backend processing logic is on another platform).
>> The score should be available within the JSONObject but I am not an expert
>> with storm and I am not completely understanding what conditions constitute
>> when the threat feed is considered an "alert" in metron. Please clarify.
>>
>> 2) How would I add an external dependency, my http rest java class, to the
>> metron maven build process? More specifically, if I was adding a custom
>> class that needed accessed by a bolt in storm, how would I add this in
>> maven as a dependency. I have limited experience with maven but, my
>> understanding is that I would add it to the pom.xml and recompile.
>> Although, the metron quick dev platform is built on a vm, would I need to
>> account for this? Please advise.
>>
>> Regards,
>>
>> Tyler Moore
>>
>>
>> Software Engineer
>> Phone: 248-909-2769
>> Email: moore.tyler@goflyball.com
>>
> --
>
> Jon
>
> Sent from my mobile device
Re: Custom Storm Topologies
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Would it be accurate to summarize what you're looking to do as - configure
Metron to take a mitigating action in response to a tuple meeting a
conditional? In your case you're looking to do an API call to another
system if, say, threat score is > 90 and a user was identified during
enrichment (for example), but logically this could be something as simple
as kicking off a script.
Just want to make sure I'm clear on what you're looking to do - if that's
correct, I actually have the exact same use case on my to do list and a
while back I opened METRON-571
<https://issues.apache.org/jira/browse/METRON-571> with the thought that a
first step in this direction would be to have Stellar handle the
conditional and kick off a script (providing it arguments pulled from the
tuple) which handles the API integration. Thanks,
Jon
On Fri, Dec 30, 2016 at 3:51 PM Tyler Moore <tm...@goflyball.com> wrote:
> It would be executed after threat intel / triage scoring.
>
> Could you give an example of either solution?
> I did look into using stellar functions but wasn't sure how to call a
> seperate method using stellar, how would I would I make the new method
> accessible using stellar functions?
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Phone: 248-909-2769 <(248)%20909-2769>
> Email: moore.tyler@goflyball.com
>
>
> On Fri, Dec 30, 2016 at 3:08 PM, Otto Fowler <ot...@gmail.com>
> wrote:
>
> > Or a Maas service?
> >
> >
> > On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
> > wrote:
> >
> > Depending on the details it sounds like a much simpler solution would be
> to
> > handle this in a Stellar function.
> >
> > Jon
> >
> > On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
> >
> > > Happy Holidays Metron Devs!
> > >
> > > Could anyone lend me some guidance on customizing the storm topologies
> in
> > > metron? What I am am trying to accomplish:
> > >
> > > 1) Add a method to the threat intel joiner bolt that sends an http post
> > > with the score of the threat to a remote rest api. This will
> > conditionally
> > > trigger notifications based on user settings in another database (the
> > > backend processing logic is on another platform).
> > > The score should be available within the JSONObject but I am not an
> > expert
> > > with storm and I am not completely understanding what conditions
> > constitute
> > > when the threat feed is considered an "alert" in metron. Please
> clarify.
> > >
> > > 2) How would I add an external dependency, my http rest java class, to
> > the
> > > metron maven build process? More specifically, if I was adding a custom
> > > class that needed accessed by a bolt in storm, how would I add this in
> > > maven as a dependency. I have limited experience with maven but, my
> > > understanding is that I would add it to the pom.xml and recompile.
> > > Although, the metron quick dev platform is built on a vm, would I need
> to
> > > account for this? Please advise.
> > >
> > > Regards,
> > >
> > > Tyler Moore
> > >
> > >
> > > Software Engineer
> > > Phone: 248-909-2769 <(248)%20909-2769>
> > > Email: moore.tyler@goflyball.com
> > >
> > --
> >
> > Jon
> >
> > Sent from my mobile device
> >
>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by Tyler Moore <tm...@goflyball.com>.
It would be executed after threat intel / triage scoring.
Could you give an example of either solution?
I did look into using stellar functions but wasn't sure how to call a
seperate method using stellar, how would I would I make the new method
accessible using stellar functions?
Regards,
Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com
On Fri, Dec 30, 2016 at 3:08 PM, Otto Fowler <ot...@gmail.com>
wrote:
> Or a Maas service?
>
>
> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
>
> Depending on the details it sounds like a much simpler solution would be to
> handle this in a Stellar function.
>
> Jon
>
> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>
> > Happy Holidays Metron Devs!
> >
> > Could anyone lend me some guidance on customizing the storm topologies in
> > metron? What I am am trying to accomplish:
> >
> > 1) Add a method to the threat intel joiner bolt that sends an http post
> > with the score of the threat to a remote rest api. This will
> conditionally
> > trigger notifications based on user settings in another database (the
> > backend processing logic is on another platform).
> > The score should be available within the JSONObject but I am not an
> expert
> > with storm and I am not completely understanding what conditions
> constitute
> > when the threat feed is considered an "alert" in metron. Please clarify.
> >
> > 2) How would I add an external dependency, my http rest java class, to
> the
> > metron maven build process? More specifically, if I was adding a custom
> > class that needed accessed by a bolt in storm, how would I add this in
> > maven as a dependency. I have limited experience with maven but, my
> > understanding is that I would add it to the pom.xml and recompile.
> > Although, the metron quick dev platform is built on a vm, would I need to
> > account for this? Please advise.
> >
> > Regards,
> >
> > Tyler Moore
> >
> >
> > Software Engineer
> > Phone: 248-909-2769
> > Email: moore.tyler@goflyball.com
> >
> --
>
> Jon
>
> Sent from my mobile device
>
Re: Custom Storm Topologies
Posted by Otto Fowler <ot...@gmail.com>.
Or a Maas service?
On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com) wrote:
Depending on the details it sounds like a much simpler solution would be to
handle this in a Stellar function.
Jon
On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
> Happy Holidays Metron Devs!
>
> Could anyone lend me some guidance on customizing the storm topologies in
> metron? What I am am trying to accomplish:
>
> 1) Add a method to the threat intel joiner bolt that sends an http post
> with the score of the threat to a remote rest api. This will conditionally
> trigger notifications based on user settings in another database (the
> backend processing logic is on another platform).
> The score should be available within the JSONObject but I am not an expert
> with storm and I am not completely understanding what conditions constitute
> when the threat feed is considered an "alert" in metron. Please clarify.
>
> 2) How would I add an external dependency, my http rest java class, to the
> metron maven build process? More specifically, if I was adding a custom
> class that needed accessed by a bolt in storm, how would I add this in
> maven as a dependency. I have limited experience with maven but, my
> understanding is that I would add it to the pom.xml and recompile.
> Although, the metron quick dev platform is built on a vm, would I need to
> account for this? Please advise.
>
> Regards,
>
> Tyler Moore
>
>
> Software Engineer
> Phone: 248-909-2769
> Email: moore.tyler@goflyball.com
>
--
Jon
Sent from my mobile device
Re: Custom Storm Topologies
Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Depending on the details it sounds like a much simpler solution would be to
handle this in a Stellar function.
Jon
On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
> Happy Holidays Metron Devs!
>
> Could anyone lend me some guidance on customizing the storm topologies in
> metron? What I am am trying to accomplish:
>
> 1) Add a method to the threat intel joiner bolt that sends an http post
> with the score of the threat to a remote rest api. This will conditionally
> trigger notifications based on user settings in another database (the
> backend processing logic is on another platform).
> The score should be available within the JSONObject but I am not an expert
> with storm and I am not completely understanding what conditions constitute
> when the threat feed is considered an "alert" in metron. Please clarify.
>
> 2) How would I add an external dependency, my http rest java class, to the
> metron maven build process? More specifically, if I was adding a custom
> class that needed accessed by a bolt in storm, how would I add this in
> maven as a dependency. I have limited experience with maven but, my
> understanding is that I would add it to the pom.xml and recompile.
> Although, the metron quick dev platform is built on a vm, would I need to
> account for this? Please advise.
>
> Regards,
>
> Tyler Moore
>
>
> Software Engineer
> Phone: 248-909-2769
> Email: moore.tyler@goflyball.com
>
--
Jon
Sent from my mobile device