Get the failure reason from ValidateResult processor

[Martin Mucha <al...@gmail.com>]

Hi,

I'm having difficulties to find a way how to get the failure reason from
ValidateResult processor. I know, it emits the route provenance event, but
you cannot integrate that in your flow as far as I know. I also saw
suggestion to write custom reporting task, but a) that one does not have
access to original FlowFile and generally it does not seem to be sound
solution.

On the otherhand ConvertToAvro validator creates new `Details` attribute to
describe failure. (well at least it used to do so? I cannot find that
Details property in documentation any more).

am I overlooking something? What I'd like to do, is to work with failure
reason further in flow, where I get along `invalid` relationship.

Thanks for advice.

Martin.

Re: Get the failure reason from ValidateResult processor

[Koji Kawamura <ij...@gmail.com>]

Hi Martin,

I assume you wanted to ask about ValidateRecord.

As you know, ValidateRecord processor emits ROUTE provenance events
with 'Details' that explains validation error.
E.g. "Records in this FlowFile were invalid for the following reasons:
; The following 1 fields were present in the Record but not in the
schema: [/extra]"
Those provenance events can be integrated by your flow if you use
SiteToSiteProvenanceReportingTask.
Provenance events can be ingested in JSON representation into your
flow via the reporting task through an Input Port via SiteToSite
protocol.

However, that approach may be challenging as the JSON provenance
events are not tied to the original FlowFiles caused the validation
errors.

Possible improvement would be adding ValidateRecord an option to embed
validation errors into the outgoing Records to the 'invalid'
relationship.
That requires user to specify RecordWriter with a schema having one
extra field to embed the validation detail.
Then you can process the 'invalid' records further since each invalid
record has validation error detail in it.

Does it sound useful in your use case? If so, please file a JIRA for
improvement.
If it suites your needs and you can contribute the improvement, that's
even better.
https://issues.apache.org/jira/projects/NIFI

Hope this helps.

Thanks,
Koji

On Fri, Jan 5, 2018 at 12:12 AM, Martin Mucha <al...@gmail.com> wrote:
> Hi,
>
> I'm having difficulties to find a way how to get the failure reason from
> ValidateResult processor. I know, it emits the route provenance event, but
> you cannot integrate that in your flow as far as I know. I also saw
> suggestion to write custom reporting task, but a) that one does not have
> access to original FlowFile and generally it does not seem to be sound
> solution.
>
> On the otherhand ConvertToAvro validator creates new `Details` attribute to
> describe failure. (well at least it used to do so? I cannot find that
> Details property in documentation any more).
>
> am I overlooking something? What I'd like to do, is to work with failure
> reason further in flow, where I get along `invalid` relationship.
>
> Thanks for advice.
>
> Martin.