Penalization on retry, on failure and session commit clarificatoion

[Michal Klempa <mi...@gmail.com>]

Hi,
I came across an issue, which looks similar to
https://stackoverflow.com/questions/51891413/nifi-processor-task-count-is-very-high-what-must-be-the-reason
Many tasks/time on downstream processor.

In my case, we are using PutElasticsearchHttpRecord and the issue is,
when RecordReader cannot parse input records (e.g. invalid JSON).
We fall into catch block:
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L455

Where the FlowFile is penalized and transferred to FAILURE.

Now the questions:
1. is penalization meant for retry-able errors? According to
https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
it seems like the answer is Yes, but the example given is also with
FAILURE relationship (not RETRY relationship)
2. I can't find any session.commit in processor code. Is it OK? May it
be, that the problem happens when session is not committed? Does NiFi
any auto-commit then?
3. is there any design document for new processor regarding Penalize +
RETRY/FAILURE relationship?

Thanks.
Michal

Re: Penalization on retry, on failure and session commit clarificatoion

[Bryan Bende <bb...@gmail.com>]

Hi Michal,

Regarding the session.commit... most processors extend from
AbstractProcessor which takes care of the commit [1]. If you extend
from AbstractSessionFactoryProcessor, or directly implement the
Processor interface with no parent class, then it is up to you to call
commit.

For the specific elastic search situation, I'm not sure if penalizing
makes sense in all those cases. The catch block for
IdentifierNotFoundException has a penalize, but this doesn't seem like
something that would resolve itself if tried again, similar for
MalformedRecordException in the catch block below.

[1] https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/processor/AbstractProcessor.java#L28

On Tue, May 7, 2019 at 6:10 AM Michal Klempa <mi...@gmail.com> wrote:
>
> Hi,
> I came across an issue, which looks similar to
> https://stackoverflow.com/questions/51891413/nifi-processor-task-count-is-very-high-what-must-be-the-reason
> Many tasks/time on downstream processor.
>
> In my case, we are using PutElasticsearchHttpRecord and the issue is,
> when RecordReader cannot parse input records (e.g. invalid JSON).
> We fall into catch block:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-processors/src/main/java/org/apache/nifi/processors/elasticsearch/PutElasticsearchHttpRecord.java#L455
>
> Where the FlowFile is penalized and transferred to FAILURE.
>
> Now the questions:
> 1. is penalization meant for retry-able errors? According to
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#penalization-vs-yielding
> it seems like the answer is Yes, but the example given is also with
> FAILURE relationship (not RETRY relationship)
> 2. I can't find any session.commit in processor code. Is it OK? May it
> be, that the problem happens when session is not committed? Does NiFi
> any auto-commit then?
> 3. is there any design document for new processor regarding Penalize +
> RETRY/FAILURE relationship?
>
> Thanks.
> Michal