[jira] [Comment Edited] (NIFI-7380) NiFi Stateless does not validate CS correctly

["Mark Weghorst (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-7380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106641#comment-17106641 ] 

Mark Weghorst edited comment on NIFI-7380 at 5/13/20, 8:26 PM:
---------------------------------------------------------------

Thank you for the patch Matthieu Cauffiez, sorry it's taken me a couple of days to try it out and report back. 

I had the chance to run this against a somewhat complicated real-world flow that does enrichment and data format transformation between 2 Kafka topics. 

To get it to work, i had to use JSON instead of Avro (Confluent Schema Registry issues), and eliminate my Jolt transforms (Jolt processor class path issues)

So far I would say that this is a very substantial improvement over PR4220. 

In my flow, I have the following controller services:
 * JsonRecordSetWriter
 * JsonTreeReader
 * SimpleKeyValueLookupService (3 different instances)
 * StandardSSLContextService
 * ConfluentSchemaRegistry

All of these controller services are working except for the ConfluentSchemaRegistry, which is failing with a ClassCastException. 

Additionally, the Jolt processor will no longer initialize, Pierre Villard can  you please let me know if we should split this Jolt issue out to a separate issue or work it from here?   Jolt works fine in the NiFi server, so this is a stateless specific issue. 

From my first glance, it looks like a packaging problem that is likely unrelated to this controller service fix.  What do you think?

I'm attaching 2 new files to the issue, containing the Confluent and Jolt exceptions. 

 


was (Author: mark.weghorst):
Thank you for the patch Matthieu Cauffiez, sorry it's taken me a couple of days to try it out and report back. 

I had the chance to run this against a somewhat complicated real-world flow that does enrichment and data format transformation between 2 Kafka topics. 

To get it to work, i had to use JSON instead of Avro (Confluent Schema Registry issues), and eliminate my Jolt transforms (Jolt processor class path issues)

So far I would say that this is a very substantial improvement over PR4220. 

In my flow, I have the following controller services:
 * JsonRecordSetWriter
 * JsonTreeReader
 * SimpleKeyValueLookupService (3 different instances)
 * StandardSSLContextService
 * ConfluentSchemaRegistry

All of these controller services are working except for the ConfluentSchemaRegistry, which is failing with a ClassCastException. 

Additionally, the Jolt processor will no longer initialize, Pierre Villard can  you please let me know if we should split this Jolt issue out to a separate issue or work it from here?  From my first glance, it looks like a packaging problem that is likely unrelated to this controller service fix.  What do you think?

I'm attaching 2 new files to the issue, containing the Confluent and Jolt exceptions. 

 

> NiFi Stateless does not validate CS correctly
> ---------------------------------------------
>
>                 Key: NIFI-7380
>                 URL: https://issues.apache.org/jira/browse/NIFI-7380
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: NiFi Stateless
>            Reporter: Pierre Villard
>            Assignee: Pierre Villard
>            Priority: Critical
>              Labels: nifi-stateless, stateless
>         Attachments: nifi-7380-confluent-schema-registry-exception.txt, nifi-7380-exception.txt, nifi-7380-flow-config.json, nifi-7380-jolt-exception.txt, stateless.json
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When the flow executed with the NiFi Stateless running mode contains a Controller Service with required properties, it'll fail as it does not take into account the configuration when performing the validation of the component.
> In *StatelessControllerServiceLookup*, the method
> {code:java}
> public void enableControllerServices(final VariableRegistry variableRegistry) 
> {code}
> first validates the configured controller services and calls
> {code:java}
> public Collection<ValidationResult> validate(...){code}
> This will create a *StatelessProcessContext* object and a *StatelessValidationContext* object. Then the method *validate* is called on the controller service and pass the validation context as argument. It will go through the properties of the controller service and will retrieve the configured value of the properties as set in the *StatelessProcessContext* object. The problem is that the *properties* map in the Stateless Process Context supposed to contain the configured values is never set. As such, any required property in a Controller Service is considered as configured with a null value if there is no default value. This will cause the component validation to fail and the flow won't be executed.
> I opened a PR with a solution that does solve this issue. However I'm not sure this issue does not affect other scenarios and a better approach could be necessary (more in line with what is done in NiFi core).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)