You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by Carolyn Duby <cd...@hortonworks.com> on 2017/01/02 16:50:42 UTC

Re: Custom Storm Topologies

Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck.


For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins.

Thanks
Carolyn

On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:

>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can.  Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>?  I do have some
>> security concerns with a SHELL_EXEC function because it could result in RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to
>> >> doing it entirely in Stellar.  Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here.  There way to get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean.  You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database (the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class, to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> ​Regards,​
>> >>>>
>> >>>> Tyler Moore
>> >>>> ​​
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>

Re: Custom Storm Topologies

Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
After a visual review that is about what I was thinking.  Like you
mentioned it could use some work - I added some comments to the JIRA
hitting on some suggested improvements.

Jon

On Thu, Jan 5, 2017 at 12:03 PM Tyler Moore <tm...@goflyball.com> wrote:

> I attached a sample function to the Jira, just calls a script from on the
> server by passing in the path. Certainly could use some work, but I wanted
> to make sure this was the right idea and get a consensus, let me know your
> thoughts.
>
> My thought process for this is that if we only allow script execution from
> a shell script that must be on the storm host, then the user is in a sense
> authenticated by their own user-defined access restrictions for that
> system.
>
>
> Regards,
>
> Tyler Moore
> Software Engineer
> Phone: 248-909-2769 <(248)%20909-2769>
> Email: moore.tyler@goflyball.com
>
>
> On Tue, Jan 3, 2017 at 3:47 PM, Carolyn Duby <cd...@hortonworks.com>
> wrote:
>
> > Also please consider the security of the scripts and script injection
> > attacks.  For example, we should probably restrict file access.
> >
> > Thanks
> > Carolyn
> >
> >
> >
> > On 1/3/17, 3:25 PM, "Otto Fowler" <ot...@gmail.com> wrote:
> >
> > >A script bolt would still allow them to write the script the way they
> want
> > >to, but would avoid having to write all the scaffolding.
> > >The matter then would be how to integrate that script bolt into the
> > >topologies.
> > >
> > >
> > >On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com)
> > wrote:
> > >
> > >Right, that definitely is more efficient, but part of the point here is
> to
> > >lower the barrier of entry to using Metron.
> > >
> > >It makes Metron's triage abilities more flexible and allows a user to
> > reuse
> > >existing code quickly and easily.  Having this available for PoC,
> > >prototyping, and low volume environments or situations (only when threat
> > >score is 100, for instance) is important, as it lowers the barrier to
> > entry
> > >of migrating a company to a Metron environment.
> > >
> > >I see this as a tradeoff where I would prioritize ease of use over
> > >efficiency.  There's nothing wrong with making both options available,
> at
> > >some point, and making their different use cases clear.
> > >
> > >Jon
> > >
> > >On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
> > >
> > >Well, yes :-)
> > >And clearly it should always be more efficient to write a custom bolt in
> > >Java than to invoke a script and manage it.
> > >
> > >--Matt
> > >
> > >From: Otto Fowler <ot...@gmail.com>
> > >Date: Tuesday, January 3, 2017 at 7:08 AM
> > >To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org
> >,
> > >Matt Foley <ma...@apache.org>
> > >Subject: Re: Custom Storm Topologies
> > >
> > >Wouldn’t that be a bolt?
> > >
> > >
> > >On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
> > >Should we consider a script calling capability that can launch a
> streaming
> > >script and keep it alive and fed, long-term, rather than launching the
> > >script anew every time the Stellar function is invoked? I’m thinking two
> > >basic rules: Write a line, read a line; and always have a timeout. Prob
> > >need a UID of some sort for a cache of running process objects.
> > >
> > >--Matt
> > >
> > >On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
> > >
> > >
> > >Inserting a script inline is ok for low throughput and prototyping but
> > once
> > >you get higher throughput (millions of events per second), it’s probably
> > >going to be a bottleneck.
> > >
> > >
> > >For Metron-571 you might want to consider a java based extension plugin
> > >similar to Eclipse plugins.
> > >
> > >Thanks
> > >Carolyn
> > >
> > >On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
> > >
> > >>Thanks Jon,
> > >>
> > >>I'll look over the tutorial and put something together for the
> SHELL_EXEC
> > >>stellar function.
> > >>I don't believe I have permissions to assign in Jira if you want to
> > assign
> > >>to me my username is devopsec.
> > >>I'll post back details and we can review security issues
> > >>
> > >>Regards,
> > >>
> > >>Tyler Moore
> > >>Software Engineer
> > >>Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> > >>Email: moore.tyler@goflyball.com
> > >>
> > >>
> > >>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com>
> > wrote:
> > >>
> > >>> Casey did a tutorial on how to add your own Stellar function here
> > >>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an
> > existing
> > >>> function that does this (current functions are listed here
> > >>> <https://github.com/apache/incubator-metron/tree/master/
> > >>> metron-platform/metron-common#stellar-core-functions>).
> > >>> I noticed that some of the Stellar function documentation was a bit
> > dated
> > >>> so I've opened a PR to update it here
> > >>> <https://github.com/apache/incubator-metron/pull/407>.
> > >>>
> > >>> As this is something I need as well, I'd be happy to assist you
> where I
> > >>> can. Perhaps you want to self-assign METRON-571
> > >>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
> > >>> security concerns with a SHELL_EXEC function because it could result
> in
> > >RCE
> > >>> - if that's the route you go I could probably help with a thorough
> > secure
> > >>> code review.
> > >>>
> > >>> Jon
> > >>>
> > >>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
> > >wrote:
> > >>>
> > >>> Thank you everyone for your suggestions,
> > >>>
> > >>> I believe that kicking off the function via stellar would be the
> > optimal
> > >>> solution. If anyone has an example of calling external code via
> stellar
> > >>> that would be very helpful. Thanks!
> > >>>
> > >>> Regards,
> > >>>
> > >>> Tyler Moore
> > >>> IT Specialist
> > >>> tyler.mathieu@yahoo.com
> > >>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> <(248)%20909-2769>
> > >>>
> > >>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com>
> > wrote:
> > >>> >
> > >>> > They are all extension points.
> > >>> >
> > >>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (
> > zeolla@gmail.com)
> > >>> wrote:
> > >>> >>
> > >>> >> Right but unless I'm missing something, both of those options are
> > more
> > >>> >> rigid and the MaaS service would have an unnecessary delay as
> > opposed
> > >to
> > >>> >> doing it entirely in Stellar. Unless there's a reason to do
> > otherwise
> > >>> that
> > >>> >> I'm missing, I would think doing this in Stellar gives you a more
> > >timely
> > >>> >> and (re)configurable end result.
> > >>> >>
> > >>> >> Jon
> > >>> >>
> > >>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwards@gmail.com
> >
> > >>> wrote:
> > >>> >>>
> > >>> >>> I think there are a couple of things you can do here. There way
> to
> > >get
> > >>> >>> something else into the split is to have another adapter to split
> > to,
> > >>> which
> > >>> >>> is what I think you mean. You can also integrate with MaaS and
> > create
> > >>> a
> > >>> >>> service that you can call via STELLAR.
> > >>> >>>
> > >>> >>>
> > >>> >>>
> > >>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
> > >ottobackwards@gmail.com
> > >>> )
> > >>> >>> wrote:
> > >>> >>>
> > >>> >>> Or a Maas service?
> > >>> >>>
> > >>> >>>
> > >>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (
> > zeolla@gmail.com)
> > >>> >>> wrote:
> > >>> >>>
> > >>> >>> Depending on the details it sounds like a much simpler solution
> > would
> > >>> be
> > >>> >>> to
> > >>> >>> handle this in a Stellar function.
> > >>> >>>
> > >>> >>> Jon
> > >>> >>>
> > >>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
> > >wrote:
> > >>> >>>>
> > >>> >>>> Happy Holidays Metron Devs!
> > >>> >>>>
> > >>> >>>> Could anyone lend me some guidance on customizing the storm
> > >topologies
> > >>> >>> in
> > >>> >>>> metron? What I am am trying to accomplish:
> > >>> >>>>
> > >>> >>>> 1) Add a method to the threat intel joiner bolt that sends an
> http
> > >>> post
> > >>> >>>> with the score of the threat to a remote rest api. This will
> > >>> >>> conditionally
> > >>> >>>> trigger notifications based on user settings in another database
> > >(the
> > >>> >>>> backend processing logic is on another platform).
> > >>> >>>> The score should be available within the JSONObject but I am not
> > an
> > >>> >>> expert
> > >>> >>>> with storm and I am not completely understanding what conditions
> > >>> >>> constitute
> > >>> >>>> when the threat feed is considered an "alert" in metron. Please
> > >>> clarify.
> > >>> >>>>
> > >>> >>>> 2) How would I add an external dependency, my http rest java
> > class,
> > >to
> > >>> >>> the
> > >>> >>>> metron maven build process? More specifically, if I was adding a
> > >>> custom
> > >>> >>>> class that needed accessed by a bolt in storm, how would I add
> > this
> > >in
> > >>> >>>> maven as a dependency. I have limited experience with maven but,
> > my
> > >>> >>>> understanding is that I would add it to the pom.xml ​and
> > recompile.
> > >>> >>>> Although, the metron quick dev platform is built on a vm, would
> I
> > >need
> > >>> >>> to
> > >>> >>>> account for this? Please advise.
> > >>> >>>>
> > >>> >>>> ​Regards,​
> > >>> >>>>
> > >>> >>>> Tyler Moore
> > >>> >>>> ​​
> > >>> >>>>
> > >>> >>>> Software Engineer
> > >>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> <(248)%20909-2769>
> > >>> >>>> Email: moore.tyler@goflyball.com
> > >>> >>>>
> > >>> >>> --
> > >>> >>>
> > >>> >>> Jon
> > >>> >>>
> > >>> >>> Sent from my mobile device
> > >>> >>>
> > >>> >>> --
> > >>> >>
> > >>> >> Jon
> > >>> >>
> > >>> >> Sent from my mobile device
> > >>> >>
> > >>>
> > >>> --
> > >>>
> > >>> Jon
> > >>>
> > >>> Sent from my mobile device
> > >>>
> > >
> > >
> > >
> > >
> > >
> > >--
> > >
> > >Jon
> > >
> > >Sent from my mobile device
> >
>
-- 

Jon

Sent from my mobile device

Re: Custom Storm Topologies

Posted by Tyler Moore <tm...@goflyball.com>.
I attached a sample function to the Jira, just calls a script from on the
server by passing in the path. Certainly could use some work, but I wanted
to make sure this was the right idea and get a consensus, let me know your
thoughts.

My thought process for this is that if we only allow script execution from
a shell script that must be on the storm host, then the user is in a sense
authenticated by their own user-defined access restrictions for that system.


Regards,

Tyler Moore
Software Engineer
Phone: 248-909-2769
Email: moore.tyler@goflyball.com


On Tue, Jan 3, 2017 at 3:47 PM, Carolyn Duby <cd...@hortonworks.com> wrote:

> Also please consider the security of the scripts and script injection
> attacks.  For example, we should probably restrict file access.
>
> Thanks
> Carolyn
>
>
>
> On 1/3/17, 3:25 PM, "Otto Fowler" <ot...@gmail.com> wrote:
>
> >A script bolt would still allow them to write the script the way they want
> >to, but would avoid having to write all the scaffolding.
> >The matter then would be how to integrate that script bolt into the
> >topologies.
> >
> >
> >On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com)
> wrote:
> >
> >Right, that definitely is more efficient, but part of the point here is to
> >lower the barrier of entry to using Metron.
> >
> >It makes Metron's triage abilities more flexible and allows a user to
> reuse
> >existing code quickly and easily.  Having this available for PoC,
> >prototyping, and low volume environments or situations (only when threat
> >score is 100, for instance) is important, as it lowers the barrier to
> entry
> >of migrating a company to a Metron environment.
> >
> >I see this as a tradeoff where I would prioritize ease of use over
> >efficiency.  There's nothing wrong with making both options available, at
> >some point, and making their different use cases clear.
> >
> >Jon
> >
> >On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
> >
> >Well, yes :-)
> >And clearly it should always be more efficient to write a custom bolt in
> >Java than to invoke a script and manage it.
> >
> >--Matt
> >
> >From: Otto Fowler <ot...@gmail.com>
> >Date: Tuesday, January 3, 2017 at 7:08 AM
> >To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
> >Matt Foley <ma...@apache.org>
> >Subject: Re: Custom Storm Topologies
> >
> >Wouldn’t that be a bolt?
> >
> >
> >On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
> >Should we consider a script calling capability that can launch a streaming
> >script and keep it alive and fed, long-term, rather than launching the
> >script anew every time the Stellar function is invoked? I’m thinking two
> >basic rules: Write a line, read a line; and always have a timeout. Prob
> >need a UID of some sort for a cache of running process objects.
> >
> >--Matt
> >
> >On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
> >
> >
> >Inserting a script inline is ok for low throughput and prototyping but
> once
> >you get higher throughput (millions of events per second), it’s probably
> >going to be a bottleneck.
> >
> >
> >For Metron-571 you might want to consider a java based extension plugin
> >similar to Eclipse plugins.
> >
> >Thanks
> >Carolyn
> >
> >On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
> >
> >>Thanks Jon,
> >>
> >>I'll look over the tutorial and put something together for the SHELL_EXEC
> >>stellar function.
> >>I don't believe I have permissions to assign in Jira if you want to
> assign
> >>to me my username is devopsec.
> >>I'll post back details and we can review security issues
> >>
> >>Regards,
> >>
> >>Tyler Moore
> >>Software Engineer
> >>Phone: 248-909-2769 <(248)%20909-2769>
> >>Email: moore.tyler@goflyball.com
> >>
> >>
> >>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com>
> wrote:
> >>
> >>> Casey did a tutorial on how to add your own Stellar function here
> >>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an
> existing
> >>> function that does this (current functions are listed here
> >>> <https://github.com/apache/incubator-metron/tree/master/
> >>> metron-platform/metron-common#stellar-core-functions>).
> >>> I noticed that some of the Stellar function documentation was a bit
> dated
> >>> so I've opened a PR to update it here
> >>> <https://github.com/apache/incubator-metron/pull/407>.
> >>>
> >>> As this is something I need as well, I'd be happy to assist you where I
> >>> can. Perhaps you want to self-assign METRON-571
> >>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
> >>> security concerns with a SHELL_EXEC function because it could result in
> >RCE
> >>> - if that's the route you go I could probably help with a thorough
> secure
> >>> code review.
> >>>
> >>> Jon
> >>>
> >>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
> >wrote:
> >>>
> >>> Thank you everyone for your suggestions,
> >>>
> >>> I believe that kicking off the function via stellar would be the
> optimal
> >>> solution. If anyone has an example of calling external code via stellar
> >>> that would be very helpful. Thanks!
> >>>
> >>> Regards,
> >>>
> >>> Tyler Moore
> >>> IT Specialist
> >>> tyler.mathieu@yahoo.com
> >>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> >>>
> >>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com>
> wrote:
> >>> >
> >>> > They are all extension points.
> >>> >
> >>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (
> zeolla@gmail.com)
> >>> wrote:
> >>> >>
> >>> >> Right but unless I'm missing something, both of those options are
> more
> >>> >> rigid and the MaaS service would have an unnecessary delay as
> opposed
> >to
> >>> >> doing it entirely in Stellar. Unless there's a reason to do
> otherwise
> >>> that
> >>> >> I'm missing, I would think doing this in Stellar gives you a more
> >timely
> >>> >> and (re)configurable end result.
> >>> >>
> >>> >> Jon
> >>> >>
> >>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
> >>> wrote:
> >>> >>>
> >>> >>> I think there are a couple of things you can do here. There way to
> >get
> >>> >>> something else into the split is to have another adapter to split
> to,
> >>> which
> >>> >>> is what I think you mean. You can also integrate with MaaS and
> create
> >>> a
> >>> >>> service that you can call via STELLAR.
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
> >ottobackwards@gmail.com
> >>> )
> >>> >>> wrote:
> >>> >>>
> >>> >>> Or a Maas service?
> >>> >>>
> >>> >>>
> >>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (
> zeolla@gmail.com)
> >>> >>> wrote:
> >>> >>>
> >>> >>> Depending on the details it sounds like a much simpler solution
> would
> >>> be
> >>> >>> to
> >>> >>> handle this in a Stellar function.
> >>> >>>
> >>> >>> Jon
> >>> >>>
> >>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
> >wrote:
> >>> >>>>
> >>> >>>> Happy Holidays Metron Devs!
> >>> >>>>
> >>> >>>> Could anyone lend me some guidance on customizing the storm
> >topologies
> >>> >>> in
> >>> >>>> metron? What I am am trying to accomplish:
> >>> >>>>
> >>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
> >>> post
> >>> >>>> with the score of the threat to a remote rest api. This will
> >>> >>> conditionally
> >>> >>>> trigger notifications based on user settings in another database
> >(the
> >>> >>>> backend processing logic is on another platform).
> >>> >>>> The score should be available within the JSONObject but I am not
> an
> >>> >>> expert
> >>> >>>> with storm and I am not completely understanding what conditions
> >>> >>> constitute
> >>> >>>> when the threat feed is considered an "alert" in metron. Please
> >>> clarify.
> >>> >>>>
> >>> >>>> 2) How would I add an external dependency, my http rest java
> class,
> >to
> >>> >>> the
> >>> >>>> metron maven build process? More specifically, if I was adding a
> >>> custom
> >>> >>>> class that needed accessed by a bolt in storm, how would I add
> this
> >in
> >>> >>>> maven as a dependency. I have limited experience with maven but,
> my
> >>> >>>> understanding is that I would add it to the pom.xml ​and
> recompile.
> >>> >>>> Although, the metron quick dev platform is built on a vm, would I
> >need
> >>> >>> to
> >>> >>>> account for this? Please advise.
> >>> >>>>
> >>> >>>> ​Regards,​
> >>> >>>>
> >>> >>>> Tyler Moore
> >>> >>>> ​​
> >>> >>>>
> >>> >>>> Software Engineer
> >>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
> >>> >>>> Email: moore.tyler@goflyball.com
> >>> >>>>
> >>> >>> --
> >>> >>>
> >>> >>> Jon
> >>> >>>
> >>> >>> Sent from my mobile device
> >>> >>>
> >>> >>> --
> >>> >>
> >>> >> Jon
> >>> >>
> >>> >> Sent from my mobile device
> >>> >>
> >>>
> >>> --
> >>>
> >>> Jon
> >>>
> >>> Sent from my mobile device
> >>>
> >
> >
> >
> >
> >
> >--
> >
> >Jon
> >
> >Sent from my mobile device
>

Re: Custom Storm Topologies

Posted by Carolyn Duby <cd...@hortonworks.com>.
Also please consider the security of the scripts and script injection attacks.  For example, we should probably restrict file access.

Thanks
Carolyn



On 1/3/17, 3:25 PM, "Otto Fowler" <ot...@gmail.com> wrote:

>A script bolt would still allow them to write the script the way they want
>to, but would avoid having to write all the scaffolding.
>The matter then would be how to integrate that script bolt into the
>topologies.
>
>
>On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com) wrote:
>
>Right, that definitely is more efficient, but part of the point here is to
>lower the barrier of entry to using Metron.
>
>It makes Metron's triage abilities more flexible and allows a user to reuse
>existing code quickly and easily.  Having this available for PoC,
>prototyping, and low volume environments or situations (only when threat
>score is 100, for instance) is important, as it lowers the barrier to entry
>of migrating a company to a Metron environment.
>
>I see this as a tradeoff where I would prioritize ease of use over
>efficiency.  There's nothing wrong with making both options available, at
>some point, and making their different use cases clear.
>
>Jon
>
>On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:
>
>Well, yes :-)
>And clearly it should always be more efficient to write a custom bolt in
>Java than to invoke a script and manage it.
>
>--Matt
>
>From: Otto Fowler <ot...@gmail.com>
>Date: Tuesday, January 3, 2017 at 7:08 AM
>To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
>Matt Foley <ma...@apache.org>
>Subject: Re: Custom Storm Topologies
>
>Wouldn’t that be a bolt?
>
>
>On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
>Should we consider a script calling capability that can launch a streaming
>script and keep it alive and fed, long-term, rather than launching the
>script anew every time the Stellar function is invoked? I’m thinking two
>basic rules: Write a line, read a line; and always have a timeout. Prob
>need a UID of some sort for a cache of running process objects.
>
>--Matt
>
>On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:
>
>
>Inserting a script inline is ok for low throughput and prototyping but once
>you get higher throughput (millions of events per second), it’s probably
>going to be a bottleneck.
>
>
>For Metron-571 you might want to consider a java based extension plugin
>similar to Eclipse plugins.
>
>Thanks
>Carolyn
>
>On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
>
>>Thanks Jon,
>>
>>I'll look over the tutorial and put something together for the SHELL_EXEC
>>stellar function.
>>I don't believe I have permissions to assign in Jira if you want to assign
>>to me my username is devopsec.
>>I'll post back details and we can review security issues
>>
>>Regards,
>>
>>Tyler Moore
>>Software Engineer
>>Phone: 248-909-2769 <(248)%20909-2769>
>>Email: moore.tyler@goflyball.com
>>
>>
>>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>>
>>> Casey did a tutorial on how to add your own Stellar function here
>>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>>> function that does this (current functions are listed here
>>> <https://github.com/apache/incubator-metron/tree/master/
>>> metron-platform/metron-common#stellar-core-functions>).
>>> I noticed that some of the Stellar function documentation was a bit dated
>>> so I've opened a PR to update it here
>>> <https://github.com/apache/incubator-metron/pull/407>.
>>>
>>> As this is something I need as well, I'd be happy to assist you where I
>>> can. Perhaps you want to self-assign METRON-571
>>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>>> security concerns with a SHELL_EXEC function because it could result in
>RCE
>>> - if that's the route you go I could probably help with a thorough secure
>>> code review.
>>>
>>> Jon
>>>
>>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
>wrote:
>>>
>>> Thank you everyone for your suggestions,
>>>
>>> I believe that kicking off the function via stellar would be the optimal
>>> solution. If anyone has an example of calling external code via stellar
>>> that would be very helpful. Thanks!
>>>
>>> Regards,
>>>
>>> Tyler Moore
>>> IT Specialist
>>> tyler.mathieu@yahoo.com
>>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>>
>>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>>> >
>>> > They are all extension points.
>>> >
>>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>>> wrote:
>>> >>
>>> >> Right but unless I'm missing something, both of those options are more
>>> >> rigid and the MaaS service would have an unnecessary delay as opposed
>to
>>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>>> that
>>> >> I'm missing, I would think doing this in Stellar gives you a more
>timely
>>> >> and (re)configurable end result.
>>> >>
>>> >> Jon
>>> >>
>>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> I think there are a couple of things you can do here. There way to
>get
>>> >>> something else into the split is to have another adapter to split to,
>>> which
>>> >>> is what I think you mean. You can also integrate with MaaS and create
>>> a
>>> >>> service that you can call via STELLAR.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
>ottobackwards@gmail.com
>>> )
>>> >>> wrote:
>>> >>>
>>> >>> Or a Maas service?
>>> >>>
>>> >>>
>>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>>> >>> wrote:
>>> >>>
>>> >>> Depending on the details it sounds like a much simpler solution would
>>> be
>>> >>> to
>>> >>> handle this in a Stellar function.
>>> >>>
>>> >>> Jon
>>> >>>
>>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
>wrote:
>>> >>>>
>>> >>>> Happy Holidays Metron Devs!
>>> >>>>
>>> >>>> Could anyone lend me some guidance on customizing the storm
>topologies
>>> >>> in
>>> >>>> metron? What I am am trying to accomplish:
>>> >>>>
>>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>>> post
>>> >>>> with the score of the threat to a remote rest api. This will
>>> >>> conditionally
>>> >>>> trigger notifications based on user settings in another database
>(the
>>> >>>> backend processing logic is on another platform).
>>> >>>> The score should be available within the JSONObject but I am not an
>>> >>> expert
>>> >>>> with storm and I am not completely understanding what conditions
>>> >>> constitute
>>> >>>> when the threat feed is considered an "alert" in metron. Please
>>> clarify.
>>> >>>>
>>> >>>> 2) How would I add an external dependency, my http rest java class,
>to
>>> >>> the
>>> >>>> metron maven build process? More specifically, if I was adding a
>>> custom
>>> >>>> class that needed accessed by a bolt in storm, how would I add this
>in
>>> >>>> maven as a dependency. I have limited experience with maven but, my
>>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>>> >>>> Although, the metron quick dev platform is built on a vm, would I
>need
>>> >>> to
>>> >>>> account for this? Please advise.
>>> >>>>
>>> >>>> ​Regards,​
>>> >>>>
>>> >>>> Tyler Moore
>>> >>>> ​​
>>> >>>>
>>> >>>> Software Engineer
>>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>> >>>> Email: moore.tyler@goflyball.com
>>> >>>>
>>> >>> --
>>> >>>
>>> >>> Jon
>>> >>>
>>> >>> Sent from my mobile device
>>> >>>
>>> >>> --
>>> >>
>>> >> Jon
>>> >>
>>> >> Sent from my mobile device
>>> >>
>>>
>>> --
>>>
>>> Jon
>>>
>>> Sent from my mobile device
>>>
>
>
>
>
>
>--
>
>Jon
>
>Sent from my mobile device

Re: Custom Storm Topologies

Posted by Otto Fowler <ot...@gmail.com>.
A script bolt would still allow them to write the script the way they want
to, but would avoid having to write all the scaffolding.
The matter then would be how to integrate that script bolt into the
topologies.


On January 3, 2017 at 15:17:59, Zeolla@GMail.com (zeolla@gmail.com) wrote:

Right, that definitely is more efficient, but part of the point here is to
lower the barrier of entry to using Metron.

It makes Metron's triage abilities more flexible and allows a user to reuse
existing code quickly and easily.  Having this available for PoC,
prototyping, and low volume environments or situations (only when threat
score is 100, for instance) is important, as it lowers the barrier to entry
of migrating a company to a Metron environment.

I see this as a tradeoff where I would prioritize ease of use over
efficiency.  There's nothing wrong with making both options available, at
some point, and making their different use cases clear.

Jon

On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:

Well, yes :-)
And clearly it should always be more efficient to write a custom bolt in
Java than to invoke a script and manage it.

--Matt

From: Otto Fowler <ot...@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
Matt Foley <ma...@apache.org>
Subject: Re: Custom Storm Topologies

Wouldn’t that be a bolt?


On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.

--Matt

On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:


Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.


For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.

Thanks
Carolyn

On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:

>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769 <(248)%20909-2769>
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> ​Regards,​
>> >>>>
>> >>>> Tyler Moore
>> >>>> ​​
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>





--

Jon

Sent from my mobile device

Re: Custom Storm Topologies

Posted by "Zeolla@GMail.com" <ze...@gmail.com>.
Right, that definitely is more efficient, but part of the point here is to
lower the barrier of entry to using Metron.

It makes Metron's triage abilities more flexible and allows a user to reuse
existing code quickly and easily.  Having this available for PoC,
prototyping, and low volume environments or situations (only when threat
score is 100, for instance) is important, as it lowers the barrier to entry
of migrating a company to a Metron environment.

I see this as a tradeoff where I would prioritize ease of use over
efficiency.  There's nothing wrong with making both options available, at
some point, and making their different use cases clear.

Jon

On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote:

Well, yes :-)
And clearly it should always be more efficient to write a custom bolt in
Java than to invoke a script and manage it.

--Matt

From: Otto Fowler <ot...@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>,
Matt Foley <ma...@apache.org>
Subject: Re: Custom Storm Topologies

Wouldn’t that be a bolt?


On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.

--Matt

On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:


Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.


For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.

Thanks
Carolyn

On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:

>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769 <(248)%20909-2769>
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> ​Regards,​
>> >>>>
>> >>>> Tyler Moore
>> >>>> ​​
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>





-- 

Jon

Sent from my mobile device

Re: Custom Storm Topologies

Posted by Matt Foley <ma...@apache.org>.
Well, yes :-)  
And clearly it should always be more efficient to write a custom bolt in Java than to invoke a script and manage it.

--Matt

From: Otto Fowler <ot...@gmail.com>
Date: Tuesday, January 3, 2017 at 7:08 AM
To: "dev@metron.incubator.apache.org" <de...@metron.incubator.apache.org>, Matt Foley <ma...@apache.org>
Subject: Re: Custom Storm Topologies

Wouldn’t that be a bolt?


On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:
Should we consider a script calling capability that can launch a streaming script and keep it alive and fed, long-term, rather than launching the script anew every time the Stellar function is invoked? I’m thinking two basic rules: Write a line, read a line; and always have a timeout. Prob need a UID of some sort for a cache of running process objects. 

--Matt 

On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote: 


Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck. 


For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins. 

Thanks 
Carolyn 

On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote: 

>Thanks Jon, 
> 
>I'll look over the tutorial and put something together for the SHELL_EXEC 
>stellar function. 
>I don't believe I have permissions to assign in Jira if you want to assign 
>to me my username is devopsec. 
>I'll post back details and we can review security issues 
> 
>Regards, 
> 
>Tyler Moore 
>Software Engineer 
>Phone: 248-909-2769 
>Email: moore.tyler@goflyball.com 
> 
> 
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote: 
> 
>> Casey did a tutorial on how to add your own Stellar function here 
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing 
>> function that does this (current functions are listed here 
>> <https://github.com/apache/incubator-metron/tree/master/ 
>> metron-platform/metron-common#stellar-core-functions>). 
>> I noticed that some of the Stellar function documentation was a bit dated 
>> so I've opened a PR to update it here 
>> <https://github.com/apache/incubator-metron/pull/407>. 
>> 
>> As this is something I need as well, I'd be happy to assist you where I 
>> can. Perhaps you want to self-assign METRON-571 
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some 
>> security concerns with a SHELL_EXEC function because it could result in RCE 
>> - if that's the route you go I could probably help with a thorough secure 
>> code review. 
>> 
>> Jon 
>> 
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote: 
>> 
>> Thank you everyone for your suggestions, 
>> 
>> I believe that kicking off the function via stellar would be the optimal 
>> solution. If anyone has an example of calling external code via stellar 
>> that would be very helpful. Thanks! 
>> 
>> Regards, 
>> 
>> Tyler Moore 
>> IT Specialist 
>> tyler.mathieu@yahoo.com 
>> 248-909-2769 <(248)%20909-2769> 
>> 
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote: 
>> > 
>> > They are all extension points. 
>> > 
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com) 
>> wrote: 
>> >> 
>> >> Right but unless I'm missing something, both of those options are more 
>> >> rigid and the MaaS service would have an unnecessary delay as opposed to 
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise 
>> that 
>> >> I'm missing, I would think doing this in Stellar gives you a more timely 
>> >> and (re)configurable end result. 
>> >> 
>> >> Jon 
>> >> 
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com> 
>> wrote: 
>> >>> 
>> >>> I think there are a couple of things you can do here. There way to get 
>> >>> something else into the split is to have another adapter to split to, 
>> which 
>> >>> is what I think you mean. You can also integrate with MaaS and create 
>> a 
>> >>> service that you can call via STELLAR. 
>> >>> 
>> >>> 
>> >>> 
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com 
>> ) 
>> >>> wrote: 
>> >>> 
>> >>> Or a Maas service? 
>> >>> 
>> >>> 
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com) 
>> >>> wrote: 
>> >>> 
>> >>> Depending on the details it sounds like a much simpler solution would 
>> be 
>> >>> to 
>> >>> handle this in a Stellar function. 
>> >>> 
>> >>> Jon 
>> >>> 
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote: 
>> >>>> 
>> >>>> Happy Holidays Metron Devs! 
>> >>>> 
>> >>>> Could anyone lend me some guidance on customizing the storm topologies 
>> >>> in 
>> >>>> metron? What I am am trying to accomplish: 
>> >>>> 
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http 
>> post 
>> >>>> with the score of the threat to a remote rest api. This will 
>> >>> conditionally 
>> >>>> trigger notifications based on user settings in another database (the 
>> >>>> backend processing logic is on another platform). 
>> >>>> The score should be available within the JSONObject but I am not an 
>> >>> expert 
>> >>>> with storm and I am not completely understanding what conditions 
>> >>> constitute 
>> >>>> when the threat feed is considered an "alert" in metron. Please 
>> clarify. 
>> >>>> 
>> >>>> 2) How would I add an external dependency, my http rest java class, to 
>> >>> the 
>> >>>> metron maven build process? More specifically, if I was adding a 
>> custom 
>> >>>> class that needed accessed by a bolt in storm, how would I add this in 
>> >>>> maven as a dependency. I have limited experience with maven but, my 
>> >>>> understanding is that I would add it to the pom.xml ​and recompile. 
>> >>>> Although, the metron quick dev platform is built on a vm, would I need 
>> >>> to 
>> >>>> account for this? Please advise. 
>> >>>> 
>> >>>> ​Regards,​ 
>> >>>> 
>> >>>> Tyler Moore 
>> >>>> ​​ 
>> >>>> 
>> >>>> Software Engineer 
>> >>>> Phone: 248-909-2769 <(248)%20909-2769> 
>> >>>> Email: moore.tyler@goflyball.com 
>> >>>> 
>> >>> -- 
>> >>> 
>> >>> Jon 
>> >>> 
>> >>> Sent from my mobile device 
>> >>> 
>> >>> -- 
>> >> 
>> >> Jon 
>> >> 
>> >> Sent from my mobile device 
>> >> 
>> 
>> -- 
>> 
>> Jon 
>> 
>> Sent from my mobile device 
>> 






Re: Custom Storm Topologies

Posted by Otto Fowler <ot...@gmail.com>.
Wouldn’t that be a bolt?


On January 2, 2017 at 14:39:34, Matt Foley (mattf@apache.org) wrote:

Should we consider a script calling capability that can launch a streaming
script and keep it alive and fed, long-term, rather than launching the
script anew every time the Stellar function is invoked? I’m thinking two
basic rules: Write a line, read a line; and always have a timeout. Prob
need a UID of some sort for a cache of running process objects.

--Matt

On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:


Inserting a script inline is ok for low throughput and prototyping but once
you get higher throughput (millions of events per second), it’s probably
going to be a bottleneck.


For Metron-571 you might want to consider a java based extension plugin
similar to Eclipse plugins.

Thanks
Carolyn

On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:

>Thanks Jon,
>
>I'll look over the tutorial and put something together for the SHELL_EXEC
>stellar function.
>I don't believe I have permissions to assign in Jira if you want to assign
>to me my username is devopsec.
>I'll post back details and we can review security issues
>
>Regards,
>
>Tyler Moore
>Software Engineer
>Phone: 248-909-2769
>Email: moore.tyler@goflyball.com
>
>
>On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com>
wrote:
>
>> Casey did a tutorial on how to add your own Stellar function here
>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
>> function that does this (current functions are listed here
>> <https://github.com/apache/incubator-metron/tree/master/
>> metron-platform/metron-common#stellar-core-functions>).
>> I noticed that some of the Stellar function documentation was a bit
dated
>> so I've opened a PR to update it here
>> <https://github.com/apache/incubator-metron/pull/407>.
>>
>> As this is something I need as well, I'd be happy to assist you where I
>> can. Perhaps you want to self-assign METRON-571
>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some
>> security concerns with a SHELL_EXEC function because it could result in
RCE
>> - if that's the route you go I could probably help with a thorough
secure
>> code review.
>>
>> Jon
>>
>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com>
wrote:
>>
>> Thank you everyone for your suggestions,
>>
>> I believe that kicking off the function via stellar would be the optimal
>> solution. If anyone has an example of calling external code via stellar
>> that would be very helpful. Thanks!
>>
>> Regards,
>>
>> Tyler Moore
>> IT Specialist
>> tyler.mathieu@yahoo.com
>> 248-909-2769 <(248)%20909-2769>
>>
>> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com>
wrote:
>> >
>> > They are all extension points.
>> >
>> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
>> wrote:
>> >>
>> >> Right but unless I'm missing something, both of those options are
more
>> >> rigid and the MaaS service would have an unnecessary delay as opposed
to
>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise
>> that
>> >> I'm missing, I would think doing this in Stellar gives you a more
timely
>> >> and (re)configurable end result.
>> >>
>> >> Jon
>> >>
>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
>> wrote:
>> >>>
>> >>> I think there are a couple of things you can do here. There way to
get
>> >>> something else into the split is to have another adapter to split
to,
>> which
>> >>> is what I think you mean. You can also integrate with MaaS and
create
>> a
>> >>> service that you can call via STELLAR.
>> >>>
>> >>>
>> >>>
>> >>> On December 30, 2016 at 15:08:48, Otto Fowler (
ottobackwards@gmail.com
>> )
>> >>> wrote:
>> >>>
>> >>> Or a Maas service?
>> >>>
>> >>>
>> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)

>> >>> wrote:
>> >>>
>> >>> Depending on the details it sounds like a much simpler solution
would
>> be
>> >>> to
>> >>> handle this in a Stellar function.
>> >>>
>> >>> Jon
>> >>>
>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com>
wrote:
>> >>>>
>> >>>> Happy Holidays Metron Devs!
>> >>>>
>> >>>> Could anyone lend me some guidance on customizing the storm
topologies
>> >>> in
>> >>>> metron? What I am am trying to accomplish:
>> >>>>
>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
>> post
>> >>>> with the score of the threat to a remote rest api. This will
>> >>> conditionally
>> >>>> trigger notifications based on user settings in another database
(the
>> >>>> backend processing logic is on another platform).
>> >>>> The score should be available within the JSONObject but I am not an
>> >>> expert
>> >>>> with storm and I am not completely understanding what conditions
>> >>> constitute
>> >>>> when the threat feed is considered an "alert" in metron. Please
>> clarify.
>> >>>>
>> >>>> 2) How would I add an external dependency, my http rest java class,
to
>> >>> the
>> >>>> metron maven build process? More specifically, if I was adding a
>> custom
>> >>>> class that needed accessed by a bolt in storm, how would I add this
in
>> >>>> maven as a dependency. I have limited experience with maven but, my
>> >>>> understanding is that I would add it to the pom.xml ​and recompile.
>> >>>> Although, the metron quick dev platform is built on a vm, would I
need
>> >>> to
>> >>>> account for this? Please advise.
>> >>>>
>> >>>> ​Regards,​
>> >>>>
>> >>>> Tyler Moore
>> >>>> ​​
>> >>>>
>> >>>> Software Engineer
>> >>>> Phone: 248-909-2769 <(248)%20909-2769>
>> >>>> Email: moore.tyler@goflyball.com
>> >>>>
>> >>> --
>> >>>
>> >>> Jon
>> >>>
>> >>> Sent from my mobile device
>> >>>
>> >>> --
>> >>
>> >> Jon
>> >>
>> >> Sent from my mobile device
>> >>
>>
>> --
>>
>> Jon
>>
>> Sent from my mobile device
>>

Re: Custom Storm Topologies

Posted by Matt Foley <ma...@apache.org>.
Should we consider a script calling capability that can launch a streaming script and keep it alive and fed, long-term, rather than launching the script anew every time the Stellar function is invoked?  I’m thinking two basic rules:  Write a line, read a line; and always have a timeout.  Prob need a UID of some sort for a cache of running process objects.

--Matt

On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote:

    
    Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck.
    
    
    For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins.
    
    Thanks
    Carolyn
    
    On 12/31/16, 5:22 PM, "Tyler Moore" <tm...@goflyball.com> wrote:
    
    >Thanks Jon,
    >
    >I'll look over the tutorial and put something together for the SHELL_EXEC
    >stellar function.
    >I don't believe I have permissions to assign in Jira if you want to assign
    >to me my username is devopsec.
    >I'll post back details and we can review security issues
    >
    >Regards,
    >
    >Tyler Moore
    >Software Engineer
    >Phone: 248-909-2769
    >Email: moore.tyler@goflyball.com
    >
    >
    >On Sat, Dec 31, 2016 at 9:46 AM, Zeolla@GMail.com <ze...@gmail.com> wrote:
    >
    >> Casey did a tutorial on how to add your own Stellar function here
    >> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing
    >> function that does this (current functions are listed here
    >> <https://github.com/apache/incubator-metron/tree/master/
    >> metron-platform/metron-common#stellar-core-functions>).
    >> I noticed that some of the Stellar function documentation was a bit dated
    >> so I've opened a PR to update it here
    >> <https://github.com/apache/incubator-metron/pull/407>.
    >>
    >> As this is something I need as well, I'd be happy to assist you where I
    >> can.  Perhaps you want to self-assign METRON-571
    >> <https://issues.apache.org/jira/browse/METRON-571>?  I do have some
    >> security concerns with a SHELL_EXEC function because it could result in RCE
    >> - if that's the route you go I could probably help with a thorough secure
    >> code review.
    >>
    >> Jon
    >>
    >> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tm...@goflyball.com> wrote:
    >>
    >> Thank you everyone for your suggestions,
    >>
    >> I believe that kicking off the function via stellar would be the optimal
    >> solution. If anyone has an example of calling external code via stellar
    >> that would be very helpful. Thanks!
    >>
    >> Regards,
    >>
    >> Tyler Moore
    >> IT Specialist
    >> tyler.mathieu@yahoo.com
    >> 248-909-2769 <(248)%20909-2769>
    >>
    >> > On Dec 30, 2016, at 17:54, Otto Fowler <ot...@gmail.com> wrote:
    >> >
    >> > They are all extension points.
    >> >
    >> >> On December 30, 2016 at 16:34:58, Zeolla@GMail.com (zeolla@gmail.com)
    >> wrote:
    >> >>
    >> >> Right but unless I'm missing something, both of those options are more
    >> >> rigid and the MaaS service would have an unnecessary delay as opposed to
    >> >> doing it entirely in Stellar.  Unless there's a reason to do otherwise
    >> that
    >> >> I'm missing, I would think doing this in Stellar gives you a more timely
    >> >> and (re)configurable end result.
    >> >>
    >> >> Jon
    >> >>
    >> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ot...@gmail.com>
    >> wrote:
    >> >>>
    >> >>> I think there are a couple of things you can do here.  There way to get
    >> >>> something else into the split is to have another adapter to split to,
    >> which
    >> >>> is what I think you mean.  You can also integrate with MaaS and create
    >> a
    >> >>> service that you can call via STELLAR.
    >> >>>
    >> >>>
    >> >>>
    >> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwards@gmail.com
    >> )
    >> >>> wrote:
    >> >>>
    >> >>> Or a Maas service?
    >> >>>
    >> >>>
    >> >>> On December 30, 2016 at 13:52:06, Zeolla@GMail.com (zeolla@gmail.com)
    >> >>> wrote:
    >> >>>
    >> >>> Depending on the details it sounds like a much simpler solution would
    >> be
    >> >>> to
    >> >>> handle this in a Stellar function.
    >> >>>
    >> >>> Jon
    >> >>>
    >> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tm...@goflyball.com> wrote:
    >> >>>>
    >> >>>> Happy Holidays Metron Devs!
    >> >>>>
    >> >>>> Could anyone lend me some guidance on customizing the storm topologies
    >> >>> in
    >> >>>> metron? What I am am trying to accomplish:
    >> >>>>
    >> >>>> 1) Add a method to the threat intel joiner bolt that sends an http
    >> post
    >> >>>> with the score of the threat to a remote rest api. This will
    >> >>> conditionally
    >> >>>> trigger notifications based on user settings in another database (the
    >> >>>> backend processing logic is on another platform).
    >> >>>> The score should be available within the JSONObject but I am not an
    >> >>> expert
    >> >>>> with storm and I am not completely understanding what conditions
    >> >>> constitute
    >> >>>> when the threat feed is considered an "alert" in metron. Please
    >> clarify.
    >> >>>>
    >> >>>> 2) How would I add an external dependency, my http rest java class, to
    >> >>> the
    >> >>>> metron maven build process? More specifically, if I was adding a
    >> custom
    >> >>>> class that needed accessed by a bolt in storm, how would I add this in
    >> >>>> maven as a dependency. I have limited experience with maven but, my
    >> >>>> understanding is that I would add it to the pom.xml ​and recompile.
    >> >>>> Although, the metron quick dev platform is built on a vm, would I need
    >> >>> to
    >> >>>> account for this? Please advise.
    >> >>>>
    >> >>>> ​Regards,​
    >> >>>>
    >> >>>> Tyler Moore
    >> >>>> ​​
    >> >>>>
    >> >>>> Software Engineer
    >> >>>> Phone: 248-909-2769 <(248)%20909-2769>
    >> >>>> Email: moore.tyler@goflyball.com
    >> >>>>
    >> >>> --
    >> >>>
    >> >>> Jon
    >> >>>
    >> >>> Sent from my mobile device
    >> >>>
    >> >>> --
    >> >>
    >> >> Jon
    >> >>
    >> >> Sent from my mobile device
    >> >>
    >>
    >> --
    >>
    >> Jon
    >>
    >> Sent from my mobile device
    >>