[jira] [Closed] (FLINK-31140) Load additional dependencies in operator classpath

["Gyula Fora (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-31140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gyula Fora closed FLINK-31140.
------------------------------
    Fix Version/s: kubernetes-operator-1.5.0
       Resolution: Fixed

merged to main ff5c00c695cdf844aca0c68242279da3afad124c

> Load additional dependencies in operator classpath 
> ---------------------------------------------------
>
>                 Key: FLINK-31140
>                 URL: https://issues.apache.org/jira/browse/FLINK-31140
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kubernetes Operator
>            Reporter: Tamir Sagi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kubernetes-operator-1.5.0
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> To date is is not possible to add additional jars to operator classpath. 
> In our case, We have a Kafka appender with custom layout that works with IAM authentication and SSL along with AWS MSK.
> log4j.properties file
>  
> {code:java}
> appender.kafka.type = Kafka
> appender.kafka.name = Kafka
> appender.kafka.bootstrap.servers = <brokers>
> appender.kafka.topic = <topic>
> appender.kafka.security.protocol = SASL_SSL
> appender.kafka.sasl.mechanism = AWS_MSK_IAM
> appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;
> appender.kafka.sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler
> appender.kafka.layout.type = CustomJsonLayout
> appender.kafka.layout.class = our.package.layouts.CustomJsonLayout 
> appender.kafka.layout.service_name = <service-name>{code}
> if CustomLayout is not present in classpath the logger fails to start.
> *main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.KafkaAppender for element Kafka: java.lang.NullPointerException java.lang.NullPointerException*
> furthermore, all the essential AWS dependencies must be added to path.
>  
> To support additional jars I needed to add them and then replace the following lines
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]
> [https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]
> using 'sed' command.
> LOGGER_JAR in that case is an uber jar contains all the necessary jars.
> {code:java}
> ENV OPERATOR_LIB=/opt/flink/operator-lib
> RUN mkdir -p $OPERATOR_LIB
> COPY target/$LOGGER_JAR $OPERATOR_LIB
> USER root
> RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/' /docker-entrypoint.sh
> USER flink{code}
> It works great but not ideal and IMO should not be handled that way.
>  
> The idea of that ticket is to allow users to add additional jars into specific location which is added to classpath while Kubernetes operator starts.
> I'd like to add the 'OPERATOR_LIB' ENV and create that folder (/opt/flink/operator-lib) as well in root Dockerfile.
> along with a minor modification in 'docker-entrypoint.sh' (Add  $OPERATOR_LIB to 'cp' command)
>  
> Once it's supported , the only thing users have to do is copy all dependencies to that folder ($OPERATOR_LIB) while building custom image
> I tested it locally and it seems to be working.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)