You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andre <an...@fucs.org> on 2016/06/19 01:30:52 UTC
Suggestion of processors
Devs,
I am continuing to drive the migration of our logging pipeline to NiFi and
in the process identified some areas of log processing that could be
improved by the introduction of new processors.
I wonder Would anyone oppose the idea of introducing the following
processors:
1. ParseCEF (think of it like logstash-codec-cef)
Processor to parse CEF format - (
https://www.protect724.hpe.com/docs/DOC-1072);
CEF attributes would be converted into NiFi FlowFiles attributes;
2. ParseKV (think of it like Splunk's kv parser)
A processor to split strings by keys and values (delimiter based) would be
added to FlowFIle attributes;
Parser would support extracting multiple instances of the same key via
attributes like parse.kv.key_name.0 , parse.kv.key_name.1, etc)
3. QueryBulkWhoisAPI
This processor would read a batch of Flowfiles, extract the appropriate
field (e.g. ip address), make the batch whois query, parse results and then
append results to individual FlowFiles.
This processor would complement QueryDNS (PR#496). QueryDNS only makes
individual queries and depending on API access conditions it may lead to
blacklisting. Some providers will license access (e.g. Spamhaus RBLs),
while others (e.g. SHadowServer) suggest instead the use of bulk queries.
Keen to hear your opinion
Re: Suggestion of processors
Posted by Aldrin Piri <al...@gmail.com>.
Andre,
Sounds good. I suspect we can find some value there so if you're willing
to take up the task, would you be willing to perhaps open up a JIRA? We can
move this discussion to there and kind of flesh out what it might look
like. As mentioned, I think the case, and its ilk, is a common pattern so
perhaps we can draw from the community at large to see what makes sense and
do some iterations to get a processor that closes this gap in
functionality.
On Wed, Jun 22, 2016 at 8:31 PM, Andre <an...@fucs.org> wrote:
> Aldrin,
>
> ParseKV definitely seems to fulfill a certain need but am curious as to how
> > we might make it provide a bit more coverage of similar formats. With
> > configuration that allows specification of both the key value separators
> > (in the example you provided a space, or new line) and the delimiter of
> the
> > pairings, this would possibly provide support for types of files as well,
> > such as Properties/ini file formats. I do find myself uncertain of how
> > much it would apply to the latter cases though. I can see how the format
> > would map pretty nicely to columnar type stores.
> >
>
> This is exactly what I have in mind. Not only plain kv but the ability to:
>
> 1. run in "shlex like" mode parsing (escaping single quotes, double quotes,
> slashes, etc)
> 2. define the key value separator
> 3. define the kv pair separator
>
> It may make sense to allow user to define the EOL demarcation character as
> well .
>
>
> > Might you be able to expand on how this information would typically be
> > handled downstream?
> >
>
> Good question. Given attributes don't seem to be able to contain
> unserialized arrays, I suspect replacing content with JSON downstream would
> be a handy choice but I am open to different suggestions.
>
> Cheers.
>
>
>
> > On Wed, Jun 22, 2016 at 10:35 AM, Andre <an...@fucs.org> wrote:
> >
> > > Aldrin,
> > >
> > > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com>
> > > wrote:
> > >
> > > > Concerning the ParseKV, are you aware of the getDelimitedField[1]
> > > function
> > > > in Expression Language? I think this may take care of this case for
> > > > handling these items.
> > > >
> > >
> > > I am aware of getDelimitedField but I found a few cases where using
> > becomes
> > > a bit challenging:
> > >
> > > * Multiple instances of the same key and poorly defined format(note how
> > > just one field uses quotes):
> > >
> > > from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had
> > enough
> > > of this"
> > >
> > > * Variable set of keys (tag wasn't present, now it is):
> > >
> > > from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
> > > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of
> this
> > >
> > > If you think if reasonably doable I am happy to reconsider.
> > >
> > >
> > >
> > > For the security folks like me, QueryBulkWhois and QueryDNS are very
> > > different beasts:
> > >
> > > * QueryDNS does what a normal DNS resolver does, but because of the
> > parsing
> > > mechanism it can be used to handle responses in a smart way. As such
> one
> > > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
> > > SenderBase [1]), RBLs (Spamhaus, etc).
> > >
> > > * Enters QueryBulkWhois: batching optimises queries by allowing a large
> > > number of subjects to be submitted using a single request.
> > >
> > > Yes, it may BulkWhois may be offered by providers that may also provide
> > API
> > > but these are note restricted to overlapping offerings, however
> projects
> > > like "Prefix WhoIs Project" only offer Whois with no DNS API available
> at
> > > all.
> > >
> > >
> > > [1]
> > >
> > >
> >
> http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org
> > >
> > >
> > > > With the QueryBulkWhois API, does it make sense to roll this into the
> > > > QueryDNS as a configurable property to do batch? Performing a
> cursory
> > > > review of the PR, it looks like this would potentially be targeting
> > those
> > > > same servers. Are batch lookups to more web service oriented
> endpoints
> > > as
> > > > opposed to just querying DNS?
> > > >
> > > > --aldrin
> > > >
> > > >
> > > >
> > >
> >
>
Re: Suggestion of processors
Posted by Andre <an...@fucs.org>.
Aldrin,
ParseKV definitely seems to fulfill a certain need but am curious as to how
> we might make it provide a bit more coverage of similar formats. With
> configuration that allows specification of both the key value separators
> (in the example you provided a space, or new line) and the delimiter of the
> pairings, this would possibly provide support for types of files as well,
> such as Properties/ini file formats. I do find myself uncertain of how
> much it would apply to the latter cases though. I can see how the format
> would map pretty nicely to columnar type stores.
>
This is exactly what I have in mind. Not only plain kv but the ability to:
1. run in "shlex like" mode parsing (escaping single quotes, double quotes,
slashes, etc)
2. define the key value separator
3. define the kv pair separator
It may make sense to allow user to define the EOL demarcation character as
well .
> Might you be able to expand on how this information would typically be
> handled downstream?
>
Good question. Given attributes don't seem to be able to contain
unserialized arrays, I suspect replacing content with JSON downstream would
be a handy choice but I am open to different suggestions.
Cheers.
> On Wed, Jun 22, 2016 at 10:35 AM, Andre <an...@fucs.org> wrote:
>
> > Aldrin,
> >
> > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com>
> > wrote:
> >
> > > Concerning the ParseKV, are you aware of the getDelimitedField[1]
> > function
> > > in Expression Language? I think this may take care of this case for
> > > handling these items.
> > >
> >
> > I am aware of getDelimitedField but I found a few cases where using
> becomes
> > a bit challenging:
> >
> > * Multiple instances of the same key and poorly defined format(note how
> > just one field uses quotes):
> >
> > from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had
> enough
> > of this"
> >
> > * Variable set of keys (tag wasn't present, now it is):
> >
> > from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
> > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of this
> >
> > If you think if reasonably doable I am happy to reconsider.
> >
> >
> >
> > For the security folks like me, QueryBulkWhois and QueryDNS are very
> > different beasts:
> >
> > * QueryDNS does what a normal DNS resolver does, but because of the
> parsing
> > mechanism it can be used to handle responses in a smart way. As such one
> > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
> > SenderBase [1]), RBLs (Spamhaus, etc).
> >
> > * Enters QueryBulkWhois: batching optimises queries by allowing a large
> > number of subjects to be submitted using a single request.
> >
> > Yes, it may BulkWhois may be offered by providers that may also provide
> API
> > but these are note restricted to overlapping offerings, however projects
> > like "Prefix WhoIs Project" only offer Whois with no DNS API available at
> > all.
> >
> >
> > [1]
> >
> >
> http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org
> >
> >
> > > With the QueryBulkWhois API, does it make sense to roll this into the
> > > QueryDNS as a configurable property to do batch? Performing a cursory
> > > review of the PR, it looks like this would potentially be targeting
> those
> > > same servers. Are batch lookups to more web service oriented endpoints
> > as
> > > opposed to just querying DNS?
> > >
> > > --aldrin
> > >
> > >
> > >
> >
>
Re: Suggestion of processors
Posted by Andre <an...@fucs.org>.
All,
Apologies for the delay but ParseCEF is pending review on PR785 / NIFI-2341.
The processor is fairly simple but this hides the underlying library
(ParCEFone) that was written to perform the parsing [1].
While developing the processor I noticed the absence of Open Source CEF
parsing & validation routines. Given that Metron, Morphlines, Hive and a
number of Apache projects lack CEF support (or SerDes), it came to me it
could be a good idea to spin off the parser so it can be used by the above
projects.
Given the complexity of the CEF standard, additional real world testing of
the parsing logic (with or without the Processor) is welcome.
[1] - https://github.com/fluenda/ParCEFone
On Thu, Jun 23, 2016 at 8:30 AM, Andre <an...@fucs.org> wrote:
> Ryan,
>
> Grok is not a replacement for automatic kv and cef parsing.
>
> Instead grok targets to simplify the use of complex regexes (underneath
> the surface most of grok is regex).
>
> To be honest I considered creating a ParseGrok processor but as a light
> user of grok I left it for a second stage.
>
> Cheers
> On 23 Jun 2016 06:58, "Ryan Ward" <ry...@gmail.com> wrote:
>
>> Aldrin,
>>
>> Would adding support for grok be more advantageous then specific
>> processors
>> to convert a particular format. Grok is very popular for such use cases.
>>
>> Ryan
>>
>> On Wed, Jun 22, 2016 at 4:17 PM, Aldrin Piri <al...@gmail.com>
>> wrote:
>>
>> > Andre,
>> >
>> > Thanks for the insights and context, the information regarding DNS was
>> > certainly helpful and I think that seems like a pretty good breakdown of
>> > functionality and unique purpose. All seem like they would provide some
>> > neat applications for enrichment.
>> >
>> > ParseKV definitely seems to fulfill a certain need but am curious as to
>> how
>> > we might make it provide a bit more coverage of similar formats. With
>> > configuration that allows specification of both the key value separators
>> > (in the example you provided a space, or new line) and the delimiter of
>> the
>> > pairings, this would possibly provide support for types of files as
>> well,
>> > such as Properties/ini file formats. I do find myself uncertain of how
>> > much it would apply to the latter cases though. I can see how the
>> format
>> > would map pretty nicely to columnar type stores.
>> >
>> > Might you be able to expand on how this information would typically be
>> > handled downstream?
>> >
>> > On Wed, Jun 22, 2016 at 10:35 AM, Andre <an...@fucs.org> wrote:
>> >
>> > > Aldrin,
>> > >
>> > > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com>
>> > > wrote:
>> > >
>> > > > Concerning the ParseKV, are you aware of the getDelimitedField[1]
>> > > function
>> > > > in Expression Language? I think this may take care of this case for
>> > > > handling these items.
>> > > >
>> > >
>> > > I am aware of getDelimitedField but I found a few cases where using
>> > becomes
>> > > a bit challenging:
>> > >
>> > > * Multiple instances of the same key and poorly defined format(note
>> how
>> > > just one field uses quotes):
>> > >
>> > > from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had
>> > enough
>> > > of this"
>> > >
>> > > * Variable set of keys (tag wasn't present, now it is):
>> > >
>> > > from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
>> > > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of
>> this
>> > >
>> > > If you think if reasonably doable I am happy to reconsider.
>> > >
>> > >
>> > >
>> > > For the security folks like me, QueryBulkWhois and QueryDNS are very
>> > > different beasts:
>> > >
>> > > * QueryDNS does what a normal DNS resolver does, but because of the
>> > parsing
>> > > mechanism it can be used to handle responses in a smart way. As such
>> one
>> > > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
>> > > SenderBase [1]), RBLs (Spamhaus, etc).
>> > >
>> > > * Enters QueryBulkWhois: batching optimises queries by allowing a
>> large
>> > > number of subjects to be submitted using a single request.
>> > >
>> > > Yes, it may BulkWhois may be offered by providers that may also
>> provide
>> > API
>> > > but these are note restricted to overlapping offerings, however
>> projects
>> > > like "Prefix WhoIs Project" only offer Whois with no DNS API
>> available at
>> > > all.
>> > >
>> > >
>> > > [1]
>> > >
>> > >
>> > http://stackoverflow.com/questions/14145886/how-to-
>> programmatically-query-senderbase-org
>> > >
>> > >
>> > > > With the QueryBulkWhois API, does it make sense to roll this into
>> the
>> > > > QueryDNS as a configurable property to do batch? Performing a
>> cursory
>> > > > review of the PR, it looks like this would potentially be targeting
>> > those
>> > > > same servers. Are batch lookups to more web service oriented
>> endpoints
>> > > as
>> > > > opposed to just querying DNS?
>> > > >
>> > > > --aldrin
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
>
Re: Suggestion of processors
Posted by Andre <an...@fucs.org>.
Ryan,
Grok is not a replacement for automatic kv and cef parsing.
Instead grok targets to simplify the use of complex regexes (underneath the
surface most of grok is regex).
To be honest I considered creating a ParseGrok processor but as a light
user of grok I left it for a second stage.
Cheers
On 23 Jun 2016 06:58, "Ryan Ward" <ry...@gmail.com> wrote:
> Aldrin,
>
> Would adding support for grok be more advantageous then specific processors
> to convert a particular format. Grok is very popular for such use cases.
>
> Ryan
>
> On Wed, Jun 22, 2016 at 4:17 PM, Aldrin Piri <al...@gmail.com> wrote:
>
> > Andre,
> >
> > Thanks for the insights and context, the information regarding DNS was
> > certainly helpful and I think that seems like a pretty good breakdown of
> > functionality and unique purpose. All seem like they would provide some
> > neat applications for enrichment.
> >
> > ParseKV definitely seems to fulfill a certain need but am curious as to
> how
> > we might make it provide a bit more coverage of similar formats. With
> > configuration that allows specification of both the key value separators
> > (in the example you provided a space, or new line) and the delimiter of
> the
> > pairings, this would possibly provide support for types of files as well,
> > such as Properties/ini file formats. I do find myself uncertain of how
> > much it would apply to the latter cases though. I can see how the format
> > would map pretty nicely to columnar type stores.
> >
> > Might you be able to expand on how this information would typically be
> > handled downstream?
> >
> > On Wed, Jun 22, 2016 at 10:35 AM, Andre <an...@fucs.org> wrote:
> >
> > > Aldrin,
> > >
> > > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com>
> > > wrote:
> > >
> > > > Concerning the ParseKV, are you aware of the getDelimitedField[1]
> > > function
> > > > in Expression Language? I think this may take care of this case for
> > > > handling these items.
> > > >
> > >
> > > I am aware of getDelimitedField but I found a few cases where using
> > becomes
> > > a bit challenging:
> > >
> > > * Multiple instances of the same key and poorly defined format(note how
> > > just one field uses quotes):
> > >
> > > from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had
> > enough
> > > of this"
> > >
> > > * Variable set of keys (tag wasn't present, now it is):
> > >
> > > from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
> > > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of
> this
> > >
> > > If you think if reasonably doable I am happy to reconsider.
> > >
> > >
> > >
> > > For the security folks like me, QueryBulkWhois and QueryDNS are very
> > > different beasts:
> > >
> > > * QueryDNS does what a normal DNS resolver does, but because of the
> > parsing
> > > mechanism it can be used to handle responses in a smart way. As such
> one
> > > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
> > > SenderBase [1]), RBLs (Spamhaus, etc).
> > >
> > > * Enters QueryBulkWhois: batching optimises queries by allowing a large
> > > number of subjects to be submitted using a single request.
> > >
> > > Yes, it may BulkWhois may be offered by providers that may also provide
> > API
> > > but these are note restricted to overlapping offerings, however
> projects
> > > like "Prefix WhoIs Project" only offer Whois with no DNS API available
> at
> > > all.
> > >
> > >
> > > [1]
> > >
> > >
> >
> http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org
> > >
> > >
> > > > With the QueryBulkWhois API, does it make sense to roll this into the
> > > > QueryDNS as a configurable property to do batch? Performing a
> cursory
> > > > review of the PR, it looks like this would potentially be targeting
> > those
> > > > same servers. Are batch lookups to more web service oriented
> endpoints
> > > as
> > > > opposed to just querying DNS?
> > > >
> > > > --aldrin
> > > >
> > > >
> > > >
> > >
> >
>
Re: Suggestion of processors
Posted by Ryan Ward <ry...@gmail.com>.
Aldrin,
Would adding support for grok be more advantageous then specific processors
to convert a particular format. Grok is very popular for such use cases.
Ryan
On Wed, Jun 22, 2016 at 4:17 PM, Aldrin Piri <al...@gmail.com> wrote:
> Andre,
>
> Thanks for the insights and context, the information regarding DNS was
> certainly helpful and I think that seems like a pretty good breakdown of
> functionality and unique purpose. All seem like they would provide some
> neat applications for enrichment.
>
> ParseKV definitely seems to fulfill a certain need but am curious as to how
> we might make it provide a bit more coverage of similar formats. With
> configuration that allows specification of both the key value separators
> (in the example you provided a space, or new line) and the delimiter of the
> pairings, this would possibly provide support for types of files as well,
> such as Properties/ini file formats. I do find myself uncertain of how
> much it would apply to the latter cases though. I can see how the format
> would map pretty nicely to columnar type stores.
>
> Might you be able to expand on how this information would typically be
> handled downstream?
>
> On Wed, Jun 22, 2016 at 10:35 AM, Andre <an...@fucs.org> wrote:
>
> > Aldrin,
> >
> > On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com>
> > wrote:
> >
> > > Concerning the ParseKV, are you aware of the getDelimitedField[1]
> > function
> > > in Expression Language? I think this may take care of this case for
> > > handling these items.
> > >
> >
> > I am aware of getDelimitedField but I found a few cases where using
> becomes
> > a bit challenging:
> >
> > * Multiple instances of the same key and poorly defined format(note how
> > just one field uses quotes):
> >
> > from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had
> enough
> > of this"
> >
> > * Variable set of keys (tag wasn't present, now it is):
> >
> > from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
> > tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of this
> >
> > If you think if reasonably doable I am happy to reconsider.
> >
> >
> >
> > For the security folks like me, QueryBulkWhois and QueryDNS are very
> > different beasts:
> >
> > * QueryDNS does what a normal DNS resolver does, but because of the
> parsing
> > mechanism it can be used to handle responses in a smart way. As such one
> > can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
> > SenderBase [1]), RBLs (Spamhaus, etc).
> >
> > * Enters QueryBulkWhois: batching optimises queries by allowing a large
> > number of subjects to be submitted using a single request.
> >
> > Yes, it may BulkWhois may be offered by providers that may also provide
> API
> > but these are note restricted to overlapping offerings, however projects
> > like "Prefix WhoIs Project" only offer Whois with no DNS API available at
> > all.
> >
> >
> > [1]
> >
> >
> http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org
> >
> >
> > > With the QueryBulkWhois API, does it make sense to roll this into the
> > > QueryDNS as a configurable property to do batch? Performing a cursory
> > > review of the PR, it looks like this would potentially be targeting
> those
> > > same servers. Are batch lookups to more web service oriented endpoints
> > as
> > > opposed to just querying DNS?
> > >
> > > --aldrin
> > >
> > >
> > >
> >
>
Re: Suggestion of processors
Posted by Aldrin Piri <al...@gmail.com>.
Andre,
Thanks for the insights and context, the information regarding DNS was
certainly helpful and I think that seems like a pretty good breakdown of
functionality and unique purpose. All seem like they would provide some
neat applications for enrichment.
ParseKV definitely seems to fulfill a certain need but am curious as to how
we might make it provide a bit more coverage of similar formats. With
configuration that allows specification of both the key value separators
(in the example you provided a space, or new line) and the delimiter of the
pairings, this would possibly provide support for types of files as well,
such as Properties/ini file formats. I do find myself uncertain of how
much it would apply to the latter cases though. I can see how the format
would map pretty nicely to columnar type stores.
Might you be able to expand on how this information would typically be
handled downstream?
On Wed, Jun 22, 2016 at 10:35 AM, Andre <an...@fucs.org> wrote:
> Aldrin,
>
> On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com>
> wrote:
>
> > Concerning the ParseKV, are you aware of the getDelimitedField[1]
> function
> > in Expression Language? I think this may take care of this case for
> > handling these items.
> >
>
> I am aware of getDelimitedField but I found a few cases where using becomes
> a bit challenging:
>
> * Multiple instances of the same key and poorly defined format(note how
> just one field uses quotes):
>
> from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had enough
> of this"
>
> * Variable set of keys (tag wasn't present, now it is):
>
> from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
> tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of this
>
> If you think if reasonably doable I am happy to reconsider.
>
>
>
> For the security folks like me, QueryBulkWhois and QueryDNS are very
> different beasts:
>
> * QueryDNS does what a normal DNS resolver does, but because of the parsing
> mechanism it can be used to handle responses in a smart way. As such one
> can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
> SenderBase [1]), RBLs (Spamhaus, etc).
>
> * Enters QueryBulkWhois: batching optimises queries by allowing a large
> number of subjects to be submitted using a single request.
>
> Yes, it may BulkWhois may be offered by providers that may also provide API
> but these are note restricted to overlapping offerings, however projects
> like "Prefix WhoIs Project" only offer Whois with no DNS API available at
> all.
>
>
> [1]
>
> http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org
>
>
> > With the QueryBulkWhois API, does it make sense to roll this into the
> > QueryDNS as a configurable property to do batch? Performing a cursory
> > review of the PR, it looks like this would potentially be targeting those
> > same servers. Are batch lookups to more web service oriented endpoints
> as
> > opposed to just querying DNS?
> >
> > --aldrin
> >
> >
> >
>
Re: Suggestion of processors
Posted by Andre <an...@fucs.org>.
Aldrin,
On Wed, Jun 22, 2016 at 12:24 AM, Aldrin Piri <al...@gmail.com> wrote:
> Concerning the ParseKV, are you aware of the getDelimitedField[1] function
> in Expression Language? I think this may take care of this case for
> handling these items.
>
I am aware of getDelimitedField but I found a few cases where using becomes
a bit challenging:
* Multiple instances of the same key and poorly defined format(note how
just one field uses quotes):
from=bob@acme.com to=alice@acme.com to=eve@acme.com subject="I had enough
of this"
* Variable set of keys (tag wasn't present, now it is):
from=bob@acme.com to=alice@acme.com to=eve@acme.com to=jay@acme.com
tag=important tag=vip tag=tag1 ... tag=tag55 subject=I had enough of this
If you think if reasonably doable I am happy to reconsider.
For the security folks like me, QueryBulkWhois and QueryDNS are very
different beasts:
* QueryDNS does what a normal DNS resolver does, but because of the parsing
mechanism it can be used to handle responses in a smart way. As such one
can use QueryDNS to use DNS based API (ShadowServer, Cymru, Cisco
SenderBase [1]), RBLs (Spamhaus, etc).
* Enters QueryBulkWhois: batching optimises queries by allowing a large
number of subjects to be submitted using a single request.
Yes, it may BulkWhois may be offered by providers that may also provide API
but these are note restricted to overlapping offerings, however projects
like "Prefix WhoIs Project" only offer Whois with no DNS API available at
all.
[1]
http://stackoverflow.com/questions/14145886/how-to-programmatically-query-senderbase-org
> With the QueryBulkWhois API, does it make sense to roll this into the
> QueryDNS as a configurable property to do batch? Performing a cursory
> review of the PR, it looks like this would potentially be targeting those
> same servers. Are batch lookups to more web service oriented endpoints as
> opposed to just querying DNS?
>
> --aldrin
>
>
>
Re: Suggestion of processors
Posted by Aldrin Piri <al...@gmail.com>.
Andre,
The proposed processors all sound like nice functionality. I did have a
couple of questions.
Concerning the ParseKV, are you aware of the getDelimitedField[1] function
in Expression Language? I think this may take care of this case for
handling these items.
With the QueryBulkWhois API, does it make sense to roll this into the
QueryDNS as a configurable property to do batch? Performing a cursory
review of the PR, it looks like this would potentially be targeting those
same servers. Are batch lookups to more web service oriented endpoints as
opposed to just querying DNS?
--aldrin
[1]
https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#getdelimitedfield
On Sat, Jun 18, 2016 at 11:14 PM, Andre <an...@fucs.org> wrote:
> Johny,
>
> I haven't used graylog heavily so would you mind clarifying what do you
> mean by
>
> "Would those work like graylog also"
>
>
>
> Cheers
>
> On Sun, Jun 19, 2016 at 12:48 PM, johny casanova <
> computertech2124@gmail.com
> > wrote:
>
> > Great idea! Would those work like graylog also?
> > On Jun 18, 2016 9:30 PM, "Andre" <an...@fucs.org> wrote:
> >
> > > Devs,
> > >
> > > I am continuing to drive the migration of our logging pipeline to NiFi
> > and
> > > in the process identified some areas of log processing that could be
> > > improved by the introduction of new processors.
> > >
> > > I wonder Would anyone oppose the idea of introducing the following
> > > processors:
> > >
> > >
> > > 1. ParseCEF (think of it like logstash-codec-cef)
> > > Processor to parse CEF format - (
> > > https://www.protect724.hpe.com/docs/DOC-1072);
> > > CEF attributes would be converted into NiFi FlowFiles attributes;
> > >
> > >
> > > 2. ParseKV (think of it like Splunk's kv parser)
> > > A processor to split strings by keys and values (delimiter based) would
> > be
> > > added to FlowFIle attributes;
> > > Parser would support extracting multiple instances of the same key via
> > > attributes like parse.kv.key_name.0 , parse.kv.key_name.1, etc)
> > >
> > >
> > > 3. QueryBulkWhoisAPI
> > > This processor would read a batch of Flowfiles, extract the appropriate
> > > field (e.g. ip address), make the batch whois query, parse results and
> > then
> > > append results to individual FlowFiles.
> > >
> > > This processor would complement QueryDNS (PR#496). QueryDNS only makes
> > > individual queries and depending on API access conditions it may lead
> to
> > > blacklisting. Some providers will license access (e.g. Spamhaus RBLs),
> > > while others (e.g. SHadowServer) suggest instead the use of bulk
> queries.
> > >
> > >
> > > Keen to hear your opinion
> > >
> >
>
Re: Suggestion of processors
Posted by Andre <an...@fucs.org>.
Johny,
I haven't used graylog heavily so would you mind clarifying what do you
mean by
"Would those work like graylog also"
Cheers
On Sun, Jun 19, 2016 at 12:48 PM, johny casanova <computertech2124@gmail.com
> wrote:
> Great idea! Would those work like graylog also?
> On Jun 18, 2016 9:30 PM, "Andre" <an...@fucs.org> wrote:
>
> > Devs,
> >
> > I am continuing to drive the migration of our logging pipeline to NiFi
> and
> > in the process identified some areas of log processing that could be
> > improved by the introduction of new processors.
> >
> > I wonder Would anyone oppose the idea of introducing the following
> > processors:
> >
> >
> > 1. ParseCEF (think of it like logstash-codec-cef)
> > Processor to parse CEF format - (
> > https://www.protect724.hpe.com/docs/DOC-1072);
> > CEF attributes would be converted into NiFi FlowFiles attributes;
> >
> >
> > 2. ParseKV (think of it like Splunk's kv parser)
> > A processor to split strings by keys and values (delimiter based) would
> be
> > added to FlowFIle attributes;
> > Parser would support extracting multiple instances of the same key via
> > attributes like parse.kv.key_name.0 , parse.kv.key_name.1, etc)
> >
> >
> > 3. QueryBulkWhoisAPI
> > This processor would read a batch of Flowfiles, extract the appropriate
> > field (e.g. ip address), make the batch whois query, parse results and
> then
> > append results to individual FlowFiles.
> >
> > This processor would complement QueryDNS (PR#496). QueryDNS only makes
> > individual queries and depending on API access conditions it may lead to
> > blacklisting. Some providers will license access (e.g. Spamhaus RBLs),
> > while others (e.g. SHadowServer) suggest instead the use of bulk queries.
> >
> >
> > Keen to hear your opinion
> >
>
Re: Suggestion of processors
Posted by johny casanova <co...@gmail.com>.
Great idea! Would those work like graylog also?
On Jun 18, 2016 9:30 PM, "Andre" <an...@fucs.org> wrote:
> Devs,
>
> I am continuing to drive the migration of our logging pipeline to NiFi and
> in the process identified some areas of log processing that could be
> improved by the introduction of new processors.
>
> I wonder Would anyone oppose the idea of introducing the following
> processors:
>
>
> 1. ParseCEF (think of it like logstash-codec-cef)
> Processor to parse CEF format - (
> https://www.protect724.hpe.com/docs/DOC-1072);
> CEF attributes would be converted into NiFi FlowFiles attributes;
>
>
> 2. ParseKV (think of it like Splunk's kv parser)
> A processor to split strings by keys and values (delimiter based) would be
> added to FlowFIle attributes;
> Parser would support extracting multiple instances of the same key via
> attributes like parse.kv.key_name.0 , parse.kv.key_name.1, etc)
>
>
> 3. QueryBulkWhoisAPI
> This processor would read a batch of Flowfiles, extract the appropriate
> field (e.g. ip address), make the batch whois query, parse results and then
> append results to individual FlowFiles.
>
> This processor would complement QueryDNS (PR#496). QueryDNS only makes
> individual queries and depending on API access conditions it may lead to
> blacklisting. Some providers will license access (e.g. Spamhaus RBLs),
> while others (e.g. SHadowServer) suggest instead the use of bulk queries.
>
>
> Keen to hear your opinion
>