[jira] [Comment Edited] (FLINK-26537) Allow disabling StatefulFunctionsConfigValidator validation for classloader.parent-first-patterns.additional

["Fil Karnicki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17503753#comment-17503753 ] 

Fil Karnicki edited comment on FLINK-26537 at 3/9/22, 5:30 PM:
---------------------------------------------------------------

hi [~igal], I think that depending on which classes get put on the classpath by the existing cluster, you may or may not need to have this validation disabled. I'm not sure there's a one-size-fits-all solution here, where for all embedded runs we shouldn't check for these classloader settings. At the same time, for people who give running the job on an existing cluster a go and face classloading problems, we could let them disable this validation, provided that they know that they need to include the correct version of statefun, kafka and protobuf dependencies in the uber jar (and not for example protobuf 2.5.0 which statefun 3.3 won't play nicely with).


was (Author: JIRAUSER284249):
hi [~igal], I think that depending on which classes get put on the classpath by the existing cluster, you may or may not need to have this validation disabled. I'm not sure there's a one-size-fits-all solution here, where for all embedded runs we shouldn't check for these classloader settings. At the same time, for people who give running the job on an existing cluster a go and face classloading problems, we could let them disable this validation, provided that they know that they need to include the correct version of statefun, kafka and protobuf dependencies (and not for example protobuf 2.5.0 which statefun 3.3 won't play nicely with).

> Allow disabling StatefulFunctionsConfigValidator validation for classloader.parent-first-patterns.additional
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-26537
>                 URL: https://issues.apache.org/jira/browse/FLINK-26537
>             Project: Flink
>          Issue Type: Improvement
>          Components: Stateful Functions
>            Reporter: Fil Karnicki
>            Priority: Major
>
> For some deployments of stateful functions onto existing, shared clusters, it is impossible to tailor which classes exist on the classpath. An example would be a Cloudera Flink cluster, which adds protobuf-java classes that clash with statefun ones.
> Stateful functions require the following flink-config.yaml setting:
> {{classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;{+}*com.google.protobuf*{+}}} 
> In the case of the cloudera flink cluster, this will cause old, 2.5.0 protobuf classes to be loaded by statefun, which causes MethodNotFound exceptions. 
> The idea is to allow disabling of the validation below, if some config setting is present in the global flink configuration, for example: statefun.validation.parent-first-classloader.disable=true
>  
> {code:java}
> private static void validateParentFirstClassloaderPatterns(Configuration configuration) {
>   final Set<String> parentFirstClassloaderPatterns =
>       parentFirstClassloaderPatterns(configuration);
>   if (!parentFirstClassloaderPatterns.containsAll(PARENT_FIRST_CLASSLOADER_PATTERNS)) {
>     throw new StatefulFunctionsInvalidConfigException(
>         CoreOptions.ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL,
>         "Must contain all of " + String.join(", ", PARENT_FIRST_CLASSLOADER_PATTERNS));
>   }
> } {code}
>  
> Then, we wouldn't need to contain com.google.protobuf in {{classloader.parent-first-patterns.additional:}} and it would be up to the statefun user to use protobuf classes which are compatible with their version of statefun.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)