Kafka Appender for Log4j Core 2.x/3.x

["Piotr P. Karwasz" <pi...@mailing.copernik.eu>]

Hi all,

As some of you are probably aware the Log4j 1.x Kafka Appender was 
removed in KAFKA-17860.

The Apache Log4j project currently maintains a Kafka Appender for Log4j 
Core 2.x[2], but following a poll on `log4j-user@logging`[3] last year 
and a long discussion in the PMC, the appender has been deprecated and 
has been removed from the 3.x branch. The reasons behind this decision 
are mostly:

* There are longstanding bugs opened against the Kafka Appender ([4] and 
[5]) that need to be addressed,

* There are no Apache Log4j devs currently working with Kafka. A 
personal interpretation of this fact is that the features implemented in 
the Kafka Appender are based on the Kafka documentation and not on what 
people actually use.

* There is a mismatch between the lifecycle of the Kafka Appender and 
Log4j Core: during the lifetime of Log4j Core 2.x, several major version 
of Kafka were released. Our appender is basically using only the 
features from Kafka 1.x.

* Last but not least, we are not sure if the Kafka Appender is used at 
all. In 2.x the Kafka Appender is included in `log4j-core`, so it is 
hard to get usage statistics from downloads.

This is why I would like to ask the Kafka community in general and the 
dev team in particular, whether:

* Is there a need for a Kafka Appender at all? Instinctively I would say 
yes, because Apache Flume is dormant, so out of the four appenders with 
guaranteed delivery[6] only Jakarta JMS and JeroMQ will remain.

* Are there some Kafka devs interested in developing one? The old 
appender can be taken as prototype, but I suspect that a lot of work is 
required to make it industrial-grade.

* Where such an appender should be hosted? This is just a minor 
technical problem. If there is a Kafka dev willing to provide long term 
help on Kafka-related issues, we can host it at Apache Logging Services. 
If the Apache Kafka project wants to host the appender, I can provide 
long term support on Log4j Core-related issues.

Piotr

[1] https://issues.apache.org/jira/browse/KAFKA-17860

[2] 
https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender

[3] https://lists.apache.org/thread/76f260b933ol7og757123lyl1ckjvm8y

[4] 
https://issues.apache.org/jira/browse/LOG4J2-1650?jql=project%20%3D%20LOG4J2%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22kafka%22%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC

[5] 
https://github.com/apache/logging-log4j2/issues?q=is%3Aissue+is%3Aopen+kafka

[6] https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html