Using KafkaAvroSerializer.class with a Kafka Log4j Appender class.

[Meadowlark Bradsher <mb...@guidewire.com>]

Hello,

It seems to me that there are two different log4jappender classes:


1.       Apache Kafka: https://github.com/apache/kafka/blob/6eacc0de303e4d29e083b89c1f53615c1dfa291e/log4j-appender/src/main/java/org/apache/kafka/log4jappender/KafkaLog4jAppender.java

2.       Log4J2: https://github.com/apache/logging-log4j2/blob/13aadaae9dc3abe680b0ad362b4b7e98f7a167fb/log4j-core/src/main/java/org/apache/logging/log4j/core/appender/mom/kafka/KafkaAppender.java

I am using Confluent 3.0.0 Kafka Connect JDBC Connector and attempting to use the 3.0.0 version of io.confluent.kafka.serializers.KafkaAvroSerializer.class for my key.serializer class as build by schema-registry v3.0.0.

Currently I have been using the log4j kafka appender (#2 above) since the example usage was more well documented but it is unable to find the KafkaAvroSerializer.class that is available within the Connector UberJar. Log4J must be using its own classloader with some classpath that I haven't figured out yet.

I am using the following log4j2.xml as my properties file.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.apache.kafka.log4jappender">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
    </Console>
      <File name="MyFile" fileName="/home/ubuntu/logs/app.log">
          <PatternLayout>
              <Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
          </PatternLayout>
      </File>
   <Kafka name="Kafka" topic="log_test10" ignoreExceptions="false" >
        <PatternLayout pattern="%message"/>
        <Property name="bootstrap.servers">localhost:9092</Property>
        <Property name = "key.serializer">io.confluent.kafka.serializers.KafkaAvroSerializer.class</Property>
        <Property name = "value.serializer">io.confluent.kafka.serializers.KafkaAvroSerializer.class</Property>
        <Property name = "key.converter.schema.registry.url">localhost:8081</Property>
        <Property name = "value.converter.schema.registry.url">localhost:8081</Property>
     </Kafka>
    <!--
    <Async name="Async">
      <AppenderRef ref="Kafka"/>
    </Async>
    -->
  </Appenders>
  <Loggers>
    <Root level="INFO">
     <AppenderRef ref="MyFile"/>
    </Root>
    <logger name="MyKafkaLogger" level="DEBUG">
      <appender-ref ref="Kafka" />
    </logger>
    <!--<Logger name="org.apache.kafka" level="INFO" />-->
        <!-- avoid recursive logging/ haven't tried OFF yet -->
  </Loggers>
</Configuration>

With the kafka version of the log4j KafkaAppender will it be easier to include the Avro serializer class? If so is there an example somewhere, or can one be provided? I'm running on guesswork and scope creep at the moment.

_Meadowlark