You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "AHeise (via GitHub)" <gi...@apache.org> on 2024/09/16 08:36:22 UTC
[PR] [FLINK-36278] Reduce log size by avoiding container logs by default [flink-connector-kafka]
AHeise opened a new pull request, #119:
URL: https://github.com/apache/flink-connector-kafka/pull/119
Currently, container logs appear under an o.a.f logger and thus are visible on CI. This results in compressed log size >40MB for a run and often leads to download errors.
This PR reroutes container logs to a special container logger. It also uses a custom format to significantly reduce the size of each log line. The logs for containers are disabled by default.
Here is the original log of an ITCase (308 KB)
[old.log](https://github.com/user-attachments/files/17010325/old.log)
```
2794 [docker-java-stream--2048098242] INFO org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] - [Kafka-0] STDOUT: [2024-09-16 08:26:53,236] INFO Server environment:java.library.path=/usr/java/packages/lib:/lib:/usr/lib:/usr/lib64:/lib64 (org.apache.zookeeper.server.ZooKeeperServer)
2794 [docker-java-stream--2048098242] INFO org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] - [Kafka-0] STDOUT: [2024-09-16 08:26:53,236] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
2794 [docker-java-stream--2048098242] INFO org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] - [Kafka-0] STDOUT: [2024-09-16 08:26:53,236] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
```
New formatter, same level (199 KB)
[with container.log](https://github.com/user-attachments/files/17010326/with.container.log)
```
[Kafka-0] [2024-09-16 08:18:25,496] INFO Server environment:java.library.path=/usr/java/packages/lib:/lib:/usr/lib:/usr/lib64:/lib64 (org.apache.zookeeper.server.ZooKeeperServer)
[Kafka-0] [2024-09-16 08:18:25,496] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[Kafka-0] [2024-09-16 08:18:25,496] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
```
Disabled container log (113 KB)
[no container.log](https://github.com/user-attachments/files/17010324/no.container.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36278] Reduce log size [flink-connector-kafka]
Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on code in PR #119:
URL: https://github.com/apache/flink-connector-kafka/pull/119#discussion_r1767405829
##########
flink-connector-kafka/src/test/resources/log4j2-test.properties:
##########
@@ -27,15 +27,41 @@ appender.testlogger.target = SYSTEM_ERR
appender.testlogger.layout.type = PatternLayout
appender.testlogger.layout.pattern = %-4r [%t] %-5p %c %x - %m%n
-logger.kafka.name = kafka
+# Overwrite the level for all Flink related loggers
+logger.flink.name = org.apache.flink
+logger.flink.level = OFF # WARN for starting debugging
+logger.flinkconnector.name = org.apache.flink.connector
+logger.flinkconnector.level = OFF # INFO/DEBUG for starting debugging
+
+# Kafka producer and consumer level
+logger.kafka.name = org.apache.kafka
logger.kafka.level = OFF
-logger.kafka2.name = state.change
+
+# For old school kafka test that spawn the test server in the same JVM
+logger.kafka2.name = kafka
Review Comment:
Errr it seems none. Good catch.
We still have test dependencies on kafka server and zookeeper that I can probably remove and with it these log confs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36278] Reduce log size [flink-connector-kafka]
Posted by "fapaul (via GitHub)" <gi...@apache.org>.
fapaul commented on code in PR #119:
URL: https://github.com/apache/flink-connector-kafka/pull/119#discussion_r1767018924
##########
flink-connector-kafka/src/test/resources/log4j2-test.properties:
##########
@@ -27,15 +27,41 @@ appender.testlogger.target = SYSTEM_ERR
appender.testlogger.layout.type = PatternLayout
appender.testlogger.layout.pattern = %-4r [%t] %-5p %c %x - %m%n
-logger.kafka.name = kafka
+# Overwrite the level for all Flink related loggers
+logger.flink.name = org.apache.flink
+logger.flink.level = OFF # WARN for starting debugging
+logger.flinkconnector.name = org.apache.flink.connector
+logger.flinkconnector.level = OFF # INFO/DEBUG for starting debugging
+
+# Kafka producer and consumer level
+logger.kafka.name = org.apache.kafka
logger.kafka.level = OFF
-logger.kafka2.name = state.change
+
+# For old school kafka test that spawn the test server in the same JVM
+logger.kafka2.name = kafka
Review Comment:
Which tests still spawn kafka in the same JVM?
##########
flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/testutils/KafkaUtil.java:
##########
@@ -81,24 +89,19 @@ public static KafkaContainer createKafkaContainer(
} else {
logLevel = "OFF";
}
+ return logLevel;
+ }
- Slf4jLogConsumer logConsumer = new Slf4jLogConsumer(logger);
- if (!StringUtils.isNullOrWhitespaceOnly(loggerPrefix)) {
- logConsumer.withPrefix(loggerPrefix);
- }
- return new KafkaContainer(DockerImageName.parse(dockerImageVersion))
- .withEnv("KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR", "1")
- .withEnv("KAFKA_TRANSACTION_STATE_LOG_MIN_ISR", "1")
- .withEnv("KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR", "1")
- .withEnv("KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE", "false")
- .withEnv("KAFKA_LOG4J_ROOT_LOGLEVEL", logLevel)
- .withEnv("KAFKA_LOG4J_LOGGERS", "state.change.logger=" + logLevel)
- .withEnv("KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE", "false")
- .withEnv(
- "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
- String.valueOf(Duration.ofHours(2).toMillis()))
- .withEnv("KAFKA_LOG4J_TOOLS_ROOT_LOGLEVEL", logLevel)
- .withLogConsumer(logConsumer);
+ public static Logger getLogger(String containerName) {
+ return LoggerFactory.getLogger("container." + containerName);
+ }
+
+ public static Logger getLogger(String type, Class<?> testClass) {
Review Comment:
Afaict this method is always called with `"flink"` as a first parameter. WDY about removing the type parameter and always set it to `flink` when called with a testClass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36278] Reduce log size by avoiding container logs by default [flink-connector-kafka]
Posted by "fapaul (via GitHub)" <gi...@apache.org>.
fapaul commented on PR #119:
URL: https://github.com/apache/flink-connector-kafka/pull/119#issuecomment-2353092492
Looking at the current test failures of this PR the log size (https://github.com/apache/flink-connector-kafka/actions/runs/10881066111/job/30189211552?pr=119) is still 50MB+. I wonder if this PR fixes the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36278] Reduce log size [flink-connector-kafka]
Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise merged PR #119:
URL: https://github.com/apache/flink-connector-kafka/pull/119
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36278] Reduce log size [flink-connector-kafka]
Posted by "AHeise (via GitHub)" <gi...@apache.org>.
AHeise commented on code in PR #119:
URL: https://github.com/apache/flink-connector-kafka/pull/119#discussion_r1767400568
##########
flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/testutils/KafkaUtil.java:
##########
@@ -81,24 +89,19 @@ public static KafkaContainer createKafkaContainer(
} else {
logLevel = "OFF";
}
+ return logLevel;
+ }
- Slf4jLogConsumer logConsumer = new Slf4jLogConsumer(logger);
- if (!StringUtils.isNullOrWhitespaceOnly(loggerPrefix)) {
- logConsumer.withPrefix(loggerPrefix);
- }
- return new KafkaContainer(DockerImageName.parse(dockerImageVersion))
- .withEnv("KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR", "1")
- .withEnv("KAFKA_TRANSACTION_STATE_LOG_MIN_ISR", "1")
- .withEnv("KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR", "1")
- .withEnv("KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE", "false")
- .withEnv("KAFKA_LOG4J_ROOT_LOGLEVEL", logLevel)
- .withEnv("KAFKA_LOG4J_LOGGERS", "state.change.logger=" + logLevel)
- .withEnv("KAFKA_CONFLUENT_SUPPORT_METRICS_ENABLE", "false")
- .withEnv(
- "KAFKA_TRANSACTION_MAX_TIMEOUT_MS",
- String.valueOf(Duration.ofHours(2).toMillis()))
- .withEnv("KAFKA_LOG4J_TOOLS_ROOT_LOGLEVEL", logLevel)
- .withLogConsumer(logConsumer);
+ public static Logger getLogger(String containerName) {
+ return LoggerFactory.getLogger("container." + containerName);
+ }
+
+ public static Logger getLogger(String type, Class<?> testClass) {
Review Comment:
Good idea. I'll rename to getFlinkLogger then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org