You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sreejasahithi (via GitHub)" <gi...@apache.org> on 2025/04/29 05:41:46 UTC
[PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
sreejasahithi opened a new pull request, #8357:
URL: https://github.com/apache/ozone/pull/8357
## What changes were proposed in this pull request?
Configured Log4j properties to enable GZIP compression for rolled-over service log files using the Daily Rolling File Appender (DRFA). This configuration is applied to all three components: Ozone Manager (OM), Storage Container Manager (SCM), and Datanode (DN), with support for maximum file size and backup limits.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12920
## How was this patch tested?
Below is the link to the successfully passed workflow:
https://github.com/sreejasahithi/ozone/actions/runs/14723274281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #8357:
URL: https://github.com/apache/ozone/pull/8357#discussion_r2065591218
##########
hadoop-ozone/dist/src/shell/conf/log4j.properties:
##########
@@ -59,16 +59,32 @@ log4j.appender.ASYNCRFA.conversionPattern=%d{ISO8601} [%t] %p %c: %m%n
# Daily Rolling File Appender
#
-log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppender
+log4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
+log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
-# Rollover at midnight
-log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
+log4j.appender.DRFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-%i.log.gz
+#MaxFileSize param accepts a long value
+log4j.appender.DRFA.triggeringPolicy.MaxFileSize=268435456
+log4j.appender.DRFA.rollingPolicy.maxIndex=10
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
# Pattern format: Date LogLevel LoggerName LogMessage
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} [%t] %p %c: %m%n
+
+
Review Comment:
NIT: We can provide a small one-line description of what the older commented-out properties do, if someone wants to re-enable them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8357:
URL: https://github.com/apache/ozone/pull/8357#discussion_r2068474891
##########
hadoop-ozone/dist/src/shell/conf/log4j.properties:
##########
@@ -31,21 +31,40 @@ log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender
#
# Rolling File Appender - cap space usage at 5gb.
#
-hadoop.log.maxfilesize=256MB
-hadoop.log.maxbackupindex=20
-log4j.appender.RFA=org.apache.log4j.RollingFileAppender
-log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
+#hadoop.log.maxfilesize=256MB
+#hadoop.log.maxbackupindex=20
+#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
+#log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
-log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}
-log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}
+#log4j.appender.RFA.MaxFileSize=${hadoop.log.maxfilesize}
+#log4j.appender.RFA.MaxBackupIndex=${hadoop.log.maxbackupindex}
-log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
+#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
# Pattern format: Date LogLevel LoggerName LogMessage
-log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %p %c: %m%n
+#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %p %c: %m%n
# Debugging Pattern format
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} (%F:%M(%L)) - %m%n
+#
+# Rolling File Appender that gzip rolled over log files
+#
+
+log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
+log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
+log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
+
+log4j.appender.RFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.RFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-%i.log.gz
Review Comment:
This will result in archive files named like `hadoop.log-1.log.gz`, so `.log` is duplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8357:
URL: https://github.com/apache/ozone/pull/8357#issuecomment-2838997397
> The current RFA properties don't gzip the rolled-over log files.
Got it.
> I also tried to achieve this by using `org.apache.log4j.rolling.RollingFileAppender `with `TimeBasedRollingPolicy` in RFA, but it didn't give the expected results as it caused frequent rollovers and failed to delete older ones when max backup limit was reached, leading to excessive log buildup. Hence, DRFA is being used instead.
`RFA` and `DRFA` are just names. When using `TimeBasedRollingPolicy` with `RFA`, the problem is not the name, but the policy.
So this should work:
```properties
#
# Rolling File Appender that gzip rolled over log files
#
log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
log4j.appender.RFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file}
log4j.appender.RFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-%i.log.gz
#MaxFileSize param accepts a long value
log4j.appender.RFA.triggeringPolicy.MaxFileSize=268435456
log4j.appender.RFA.rollingPolicy.maxIndex=10
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
# Pattern format: Date LogLevel LoggerName LogMessage
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %p %c: %m%n
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #8357:
URL: https://github.com/apache/ozone/pull/8357#discussion_r2065680870
##########
hadoop-ozone/dist/src/shell/conf/log4j.properties:
##########
@@ -56,19 +56,38 @@ log4j.appender.ASYNCRFA.maxBackupIndex=${hadoop.log.maxbackupindex}
log4j.appender.ASYNCRFA.conversionPattern=%d{ISO8601} [%t] %p %c: %m%n
#
-# Daily Rolling File Appender
+# Daily Rolling File Appender that gzip rolled over log files
#
-log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppender
+log4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
+log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
-# Rollover at midnight
-log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
+log4j.appender.DRFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-%i.log.gz
+#MaxFileSize param accepts a long value
+log4j.appender.DRFA.triggeringPolicy.MaxFileSize=268435456
Review Comment:
nit: can this be defined with unit, i.e. `256MB`?
##########
hadoop-ozone/dist/pom.xml:
##########
@@ -36,6 +36,10 @@
</properties>
<dependencies>
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>apache-log4j-extras</artifactId>
+ </dependency>
Review Comment:
Why is this needed for `ozone-dist`?
##########
hadoop-hdds/container-service/pom.xml:
##########
@@ -102,6 +102,10 @@
<groupId>jakarta.xml.bind</groupId>
<artifactId>jakarta.xml.bind-api</artifactId>
</dependency>
+ <dependency>
+ <groupId>log4j</groupId>
+ <artifactId>apache-log4j-extras</artifactId>
+ </dependency>
Review Comment:
Since no Ozone code uses this, I think this is required only at runtime. Thus, please add `<scope>runtime</scope>` (for each service).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8357:
URL: https://github.com/apache/ozone/pull/8357#issuecomment-2838595738
Also, what about Recon and S3 Gateway?
Maybe we should add the dependency in `hdds-server-framework` to let all services get it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8357:
URL: https://github.com/apache/ozone/pull/8357#issuecomment-2839029294
> Have also tested these properties locally in a docker cluster
Can you please provide details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai merged PR #8357:
URL: https://github.com/apache/ozone/pull/8357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on PR #8357:
URL: https://github.com/apache/ozone/pull/8357#issuecomment-2838903789
> Sorry @sreejasahithi just two more questions:
>
> 1. Doesn't existing RFA do what we need?
> 2. If not, shouldn't we replace RFA instead of DRFA, since rolling is no longer day-based?
>
> https://github.com/apache/ozone/blob/9f3dd011119a33b6e5f458d826442bbda49810fb/hadoop-ozone/dist/src/shell/conf/log4j.properties#L31-L40
The current RFA properties don't gzip the rolled-over log files. I also tried to achieve this by using `org.apache.log4j.rolling.RollingFileAppender `with `TimeBasedRollingPolicy` in RFA, but it didn't give the expected results as it caused frequent rollovers and failed to delete older ones when max backup limit was reached, leading to excessive log buildup. Hence, DRFA is being used instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on PR #8357:
URL: https://github.com/apache/ozone/pull/8357#issuecomment-2840822716
> > The current RFA properties don't gzip the rolled-over log files.
>
> Got it.
>
> > I also tried to achieve this by using `org.apache.log4j.rolling.RollingFileAppender `with `TimeBasedRollingPolicy` in RFA, but it didn't give the expected results as it caused frequent rollovers and failed to delete older ones when max backup limit was reached, leading to excessive log buildup. Hence, DRFA is being used instead.
>
> `RFA` and `DRFA` are just names. When using `TimeBasedRollingPolicy` with `RFA`, the problem is not the name, but the policy.
>
> So this should work:
>
> ```ini
> #
> # Rolling File Appender that gzip rolled over log files
> #
>
> log4j.appender.RFA=org.apache.log4j.rolling.RollingFileAppender
> log4j.appender.RFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
> log4j.appender.RFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
>
> log4j.appender.RFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file}
> log4j.appender.RFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-%i.log.gz
> #MaxFileSize param accepts a long value
> log4j.appender.RFA.triggeringPolicy.MaxFileSize=268435456
> log4j.appender.RFA.rollingPolicy.maxIndex=10
>
> log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
>
> # Pattern format: Date LogLevel LoggerName LogMessage
> log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} [%t] %p %c: %m%n
> ```
okay I got it , thank you for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on code in PR #8357:
URL: https://github.com/apache/ozone/pull/8357#discussion_r2065954526
##########
hadoop-ozone/dist/src/shell/conf/log4j.properties:
##########
@@ -56,19 +56,38 @@ log4j.appender.ASYNCRFA.maxBackupIndex=${hadoop.log.maxbackupindex}
log4j.appender.ASYNCRFA.conversionPattern=%d{ISO8601} [%t] %p %c: %m%n
#
-# Daily Rolling File Appender
+# Daily Rolling File Appender that gzip rolled over log files
#
-log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
-log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.DRFA=org.apache.log4j.rolling.RollingFileAppender
+log4j.appender.DRFA.rollingPolicy=org.apache.log4j.rolling.FixedWindowRollingPolicy
+log4j.appender.DRFA.triggeringPolicy=org.apache.log4j.rolling.SizeBasedTriggeringPolicy
-# Rollover at midnight
-log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
+log4j.appender.DRFA.rollingPolicy.ActiveFileName=${hadoop.log.dir}/${hadoop.log.file}
+log4j.appender.DRFA.rollingPolicy.FileNamePattern=${hadoop.log.dir}/${hadoop.log.file}-%i.log.gz
+#MaxFileSize param accepts a long value
+log4j.appender.DRFA.triggeringPolicy.MaxFileSize=268435456
Review Comment:
MaxFileSize only accepts long value , so it would give something like
`log4j:WARN Failed to set property [maxFileSize] to value "256MB". `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12920. Configure log4j to gzip rolled over service log files [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #8357:
URL: https://github.com/apache/ozone/pull/8357#issuecomment-2846707505
Thanks @sreejasahithi for iterating on this, @aryangupta1998, @ashishkumar50, @jojochuang, @tanvipenumudy for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org