[jira] [Created] (FLINK-31140) Load additional dependencies in Operator classpath

["Tamir Sagi (Jira)" <ji...@apache.org>]

Tamir Sagi created FLINK-31140:
----------------------------------

             Summary: Load additional dependencies in Operator classpath 
                 Key: FLINK-31140
                 URL: https://issues.apache.org/jira/browse/FLINK-31140
             Project: Flink
          Issue Type: Improvement
          Components: Kubernetes Operator
            Reporter: Tamir Sagi


To date is is not possible to add additional jars to operator classpath. 

In our case, We have a custom layout which need to work with IAM authentication and SSL against AWS MSK.


log4j.properties file

 
{code:java}
appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.bootstrap.servers = <brokers>
appender.kafka.topic = <topic>
appender.kafka.security.protocol = SASL_SSL
appender.kafka.sasl.mechanism = AWS_MSK_IAM
appender.kafka.sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;
appender.kafka.sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler
appender.kafka.layout.type = CustomJsonLayout
appender.kafka.layout.class = our.package.layouts.CustomJsonLayout appender.kafka.layout.service_name = <service-name>{code}
if CustomLayout is not present in classpath the logger fails to start.

main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.KafkaAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException

furthermore, we need to add all the essential AWS dependencies.

 

To support additional jar we need to add them and then replace the following lines

[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L32]

[https://github.com/apache/flink-kubernetes-operator/blob/main/docker-entrypoint.sh#L37]

using 'sed' command

 

 
{code:java}
ENV USER_DEPENDENCIES_DIR=/opt/flink/dependencies
COPY target/$LOGGER_JAR $USER_DEPENDENCIES_DIR

USER root

RUN sed -i 's/java -cp /java -cp $USER_DEPENDENCIES_DIR/*:/' /docker-entrypoint.sh

USER flink{code}
 

It works great  but it should not be handled that way.

 

The idea of that ticket is to allow users to add additional jars into specific location which is added to classpath while launching Kubernetes operator.

I'd like to add the 'USER_DEPENDENCIES_DIR' ENV and create that folder as well in root Dockerfile.

along with minor modification in docker-entrypoint.sh (Add  $USER_DEPENDENCIES_DIR to 'cp' command)

 


I tested it locally and it seems to be working.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)