You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/02/16 03:22:09 UTC

[GitHub] [kafka] vvcephei opened a new pull request #11767: KAFKA-13598: set log4j appender to default acks

vvcephei opened a new pull request #11767:
URL: https://github.com/apache/kafka/pull/11767


   After https://github.com/apache/kafka/pull/11691, the
   KafkaLog4JAppender causes the process to crash due
   to the mismatch between the new produce default
   idempotent mode and the appender's overridden
   `acks=1` property.
   
   * Removes the `acks=1` override, so the appender falls
   back on the producer's default acks config.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] vvcephei commented on pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #11767:
URL: https://github.com/apache/kafka/pull/11767#issuecomment-1041059391


   This issue surfaced in Confluent's ksqlDB. Once ksqlDB started to build against Kafka post-#11691, it crashes with:
   
   ```
   Exception in thread "main" java.lang.ExceptionInInitializerError
   	at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
   	at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
   	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
   	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
   	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
   	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
   	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
   	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
   	at io.confluent.ksql.rest.server.KsqlServerMain.<clinit>(KsqlServerMain.java:39)
   Caused by: org.apache.kafka.common.config.ConfigException: Must set acks to all in order to use the idempotent producer. Otherwise we cannot guarantee idempotence.
   	at org.apache.kafka.clients.producer.ProducerConfig.postProcessAndValidateIdempotenceConfigs(ProducerConfig.java:475)
   	at org.apache.kafka.clients.producer.ProducerConfig.postProcessParsedConfig(ProducerConfig.java:443)
   	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:114)
   	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:133)
   	at org.apache.kafka.clients.producer.ProducerConfig.<init>(ProducerConfig.java:511)
   	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:289)
   	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:316)
   	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:301)
   	at org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:341)
   	at org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:335)
   	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
   	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
   	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
   	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
   	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
   	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
   	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
   	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
   	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
   	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
   ```
   
   I would expect any other project using the `KafkaLog4JAppender` to experience the same behavior.
   
   I'm not sure why the KafkaLog4JAppender originally overrode the acks to 1, but it's certainly incompatible with the new idempotence default. An alternative would be to leave acks=1 and disable idempotence, but I figured fewer overrides make the appender easier to reason about. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #11767:
URL: https://github.com/apache/kafka/pull/11767#issuecomment-1042497943


   > Yes, I don't know the getters serve any purpose, either. But I'm still worried that there's someone relying on it. 
   I'd like to hear what @hachikuji 's thought since he did the change to make `requiredNumAcks` to 1 in this PR: https://github.com/apache/kafka/pull/5425#discussion_r205861230
   
   Unfair to mention a PR from 4 years ago 😉 . Regardless whether anyone actually uses this class, it does seem weird to have getters that we know are broken. It does seem better to initialize the defaults explicitly. Perhaps we may as well fix the other broken getters while we're at it. At a quick glance, it looks like `maxBlockMs` is the only other one?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] vvcephei commented on pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #11767:
URL: https://github.com/apache/kafka/pull/11767#issuecomment-1041078717


   Thanks, @showuon ! I noticed that also, but there are also a whole bunch of other nullable object references with primitive-typed getters in that class, so I figured it was just some funky log4j thing.
   
   I figured out that the setter is used by log4 to apply the configuration (for example, `log4j.appender.kafka_appender.RequiredNumAcks=-1` calls `setRequredNumAcks(-1)`), but I don't know if the getters serve any purpose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] vvcephei commented on pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
vvcephei commented on pull request #11767:
URL: https://github.com/apache/kafka/pull/11767#issuecomment-1043135161


   Closing in favor of https://issues.apache.org/jira/browse/KAFKA-13673 / https://github.com/apache/kafka/pull/11788


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] vvcephei closed pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
vvcephei closed pull request #11767:
URL: https://github.com/apache/kafka/pull/11767


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji edited a comment on pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
hachikuji edited a comment on pull request #11767:
URL: https://github.com/apache/kafka/pull/11767#issuecomment-1042497943


   > Yes, I don't know the getters serve any purpose, either. But I'm still worried that there's someone relying on it. 
   I'd like to hear what @hachikuji 's thought since he did the change to make `requiredNumAcks` to 1 in this PR: https://github.com/apache/kafka/pull/5425#discussion_r205861230
   
   Unfair to mention a PR from 4 years ago 😉 . Regardless whether anyone actually uses this class, it does seem weird to have getters that we know are broken. I agree it seems better to initialize the defaults explicitly (and to use types consistently). Perhaps we may as well fix the other broken getters while we're at it. At a quick glance, it looks like `maxBlockMs` is the only other one?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] showuon commented on pull request #11767: KAFKA-13598: set log4j appender to default acks

Posted by GitBox <gi...@apache.org>.
showuon commented on pull request #11767:
URL: https://github.com/apache/kafka/pull/11767#issuecomment-1041150174


   Yes, I don't know the getters serve any purpose, either. But I'm still worried that there's someone relying on it. 
   I'd like to hear what @hachikuji 's thought since he did the change to make `requiredNumAcks` to 1 in this PR: https://github.com/apache/kafka/pull/5425#discussion_r205861230


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org