How can I ExtractGrok from end-of-file?

[Eric Chaves <er...@uolet.com>]

Hi folks,

I'm learning how to use grok with nifi starting  with the ExtractGrok
processor and I'd like to use it to extract data from file headers and
trailers however since the GrokExtract processor only apply the grok
expression on the defined buffer size (and each of my file differs on size)
I can't evaluate trailers on every file.

Any ideas on how could I apply the grok expression from the end of file
instead of from the beginning, or any alternative processor?

Cheers,

Re: How can I ExtractGrok from end-of-file?

[Eric Chaves <er...@uolet.com>]

Hi Koji, thanks for the tip. I've ended writing a processor script for this
task. Since I need the footer to validate the entire file, It was easier
than splitting the files for the sole purpose of finding out the last line.

Regards,

Em qua, 20 de mar de 2019 às 00:59, Koji Kawamura <ij...@gmail.com>
escreveu:

> Hello Eric,
>
> Have you found any solution for this?
> If your trailers (footer?) starts with certain byte sequence, then
> SplitContent may be helpful to split the content into Header+Payload,
> and the Trailers.
> If that works, then the subsequent flow can do something creative
> probably using RouteOnAttribute, GrokExtract, MergeContent (with
> defragment merge strategy) ... etc.
>
> Thanks,
> Koji
>
> On Fri, Mar 15, 2019 at 11:34 PM Eric Chaves <er...@uolet.com> wrote:
> >
> > Hi folks,
> >
> > I'm learning how to use grok with nifi starting  with the ExtractGrok
> processor and I'd like to use it to extract data from file headers and
> trailers however since the GrokExtract processor only apply the grok
> expression on the defined buffer size (and each of my file differs on size)
> I can't evaluate trailers on every file.
> >
> > Any ideas on how could I apply the grok expression from the end of file
> instead of from the beginning, or any alternative processor?
> >
> > Cheers,
> >
>

Re: How can I ExtractGrok from end-of-file?

[Koji Kawamura <ij...@gmail.com>]

Hello Eric,

Have you found any solution for this?
If your trailers (footer?) starts with certain byte sequence, then
SplitContent may be helpful to split the content into Header+Payload,
and the Trailers.
If that works, then the subsequent flow can do something creative
probably using RouteOnAttribute, GrokExtract, MergeContent (with
defragment merge strategy) ... etc.

Thanks,
Koji

On Fri, Mar 15, 2019 at 11:34 PM Eric Chaves <er...@uolet.com> wrote:
>
> Hi folks,
>
> I'm learning how to use grok with nifi starting  with the ExtractGrok processor and I'd like to use it to extract data from file headers and trailers however since the GrokExtract processor only apply the grok expression on the defined buffer size (and each of my file differs on size) I can't evaluate trailers on every file.
>
> Any ideas on how could I apply the grok expression from the end of file instead of from the beginning, or any alternative processor?
>
> Cheers,
>