You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nuttx.apache.org by GitBox <gi...@apache.org> on 2021/06/03 14:04:59 UTC
[GitHub] [incubator-nuttx] fjpanag opened a new pull request #3842: Added automatic log rotation, when log file is opened.
fjpanag opened a new pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842
## Summary
Automatic log rotation mechanism.
When a log file is opened, its size is checked. If it is found larger than the specified limit, it is rotated automatically.
This feature is very useful in file loggers so a) the log file does not get excessively large, b) the physical memory does not get exhausted due to log files.
This implementation tries to be as lightweight as possible:
- The log file is only checked during opening, so there is no additional overhead during normal logging.
- The existing log file is kept as a backup, and a new empty file is created for logging. This approach is very fast and simple compared to actually manipulating the contents of the log file (in order to shrink it).
Limitations / Assumptions
- Since the file is only checked during opening, the log file can (and will !) exceed the limit. The file system must have enough space available to allow for this "overflow", till next check.
- It is assumed that re-opening of the log file will happen occasionally. Either by any system reset, or by (un)mounting of the file system.
Regardless of the above limitations, I have tested the this exact mechanism in other applications, and I found it to work nicely. In real-life applications I wouldn't expect the above limitations to be met.
## Impact
None on existing configurations.
## Testing
Pending. I will comment bellow when ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645036824
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is not rotating the log, but only creating a backup of the last log file.
I proper rotating log should define how many log files the system will support and it should always move all the previous log files to log.NNN + 1. It could happen until the max log files supported is reached. When it happen the oldest log file is deleted and replaced with oldest - 1.
If you don't want to implement it, I suggest rename the function from rotate log to backup log, to avoid confusion :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#issuecomment-854063840
Just tested it on a custom STM32F4 target.
It works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645049787
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is still a rotation, it just happens that there is only one rotation available. :)
Renaming this as "backup" would imply that this is a backup functionality. Which is not.
A backup would also keep the original file in place, while here it is deleted.
This can be very easily be extended to keep multiple rotations, instead of only one.
I have done this in the past, and I can easily add it to this implementation.
But I intentionally left it out. Do we really need so many log files?
If you think so, I will add it tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645036824
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is not rotating the log, but only creating a backup of the last log file.
A proper rotating log should define how many log files the system will support and it should always move all the previous log files to log.NNN + 1. It could happen until the max log files supported is reached. When it happen the oldest log file is deleted and replaced with oldest - 1.
If you don't want to implement it, I suggest rename the function from rotate log to backup log, to avoid confusion :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645425612
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -108,6 +168,12 @@ FAR struct syslog_channel_s *syslog_file_channel(FAR const char *devpath)
syslog_dev_uninitialize(g_syslog_file_channel);
}
+ /* Rotate the log file, if needed. */
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+ log_rotate(devpath);
Review comment:
Theoretically you are right. This can happen.
However, in practice this is very rare, and I prefer to keep the simple solution.
What you describe is a good idea, but I am afraid that it won't work easily/correctly.
There is no easy way to manipulate the log after it is opened without side-effects:
* Either the system will block during the rotation, possibly harming high priority threads.
* Or there will be lost logs.
Unfortunately, it is not possible to predict when the rotation will take place. This will have an impact on the real-time behavior of the system. Keeping the rotations during opening only keeps the system predictable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645062352
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
You can still increase the log file limit if you want to keep all logs for X amount of time.
Nevertheless, I will improve this tomorrow, as an option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645049787
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is still a rotation, it just happens that there is only one rotation available. :)
Renaming this as "backup" would imply that this is a backup functionality. Which is not.
A backup would also keep the original file in place, while here it is deleted.
This can be very easily be extended to keep multiple rotations, instead of only one.
I have done this in the past, and I can easily add it to this implementation.
But I intentionally left it out. Do we really need so many log files?
If you think so, I will add it tomorrow.
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
Review comment:
Fixed!
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
You can still increase the log file limit if you want to keep all logs for X amount of time.
Nevertheless, I will improve this tomorrow, as an option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] xiaoxiang781216 commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
xiaoxiang781216 commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645261641
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -108,6 +168,12 @@ FAR struct syslog_channel_s *syslog_file_channel(FAR const char *devpath)
syslog_dev_uninitialize(g_syslog_file_channel);
}
+ /* Rotate the log file, if needed. */
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+ log_rotate(devpath);
Review comment:
how to handle the long live device? some device may run many day and then the log size will become very very big. It's better to detect the file size reach to the limitation in the write/putc/force_putc callback. We can save the initial file size after opening the file, and increase this value in these callback and rename the file once the limitation reach.
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char *backup_file;
Review comment:
add FAR
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char *backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
Review comment:
it's prefer to call nx_xxx or file_xxx in kernel space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645052192
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
> It is still a rotation, it just happens that there is only one rotation available. :)
>
> Renaming this as "backup" would imply that this is a backup functionality. Which is not.
> A backup would also keep the original file in place, while here it is deleted.
>
> This can be very easily be extended to keep multiple rotations, instead of only one.
> I have done this in the past, and I can easily add it to this implementation.
>
> But I intentionally left it out. Do we really need so many log files?
> If you think so, I will add it tomorrow.
I think it should be useful for some system. Imagine a system were we keep it in the field and after some amount to days (i.e. 30 days) you can go there and get the complete log history.
I can merge this one and wait for your improvement over it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis merged pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis merged pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645036824
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is not rotating the log, but only creating a backup of the last log file.
A proper rotating log should define how many log files the system will support and it should always move all the previous log files to log.NNN + 1. It could happen until the max log files supported is reached. When it happen the oldest log file is deleted and replaced with oldest - 1.
If you don't want to implement it, I suggest renaming the function and the Kconfig from rotate log to backup log, to avoid confusion :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis merged pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis merged pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] xiaoxiang781216 commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
xiaoxiang781216 commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645261641
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -108,6 +168,12 @@ FAR struct syslog_channel_s *syslog_file_channel(FAR const char *devpath)
syslog_dev_uninitialize(g_syslog_file_channel);
}
+ /* Rotate the log file, if needed. */
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+ log_rotate(devpath);
Review comment:
how to handle the long live device? some device may run many day and then the log size will become very very big. It's better to detect the file size reach to the limitation in the write/putc/force_putc callback. We can save the initial file size after opening the file, and increase this value in these callback and rename the file once the limitation reach.
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char *backup_file;
Review comment:
add FAR
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char *backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
Review comment:
it's prefer to call nx_xxx or file_xxx in kernel space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645051154
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
Review comment:
Fixed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645026965
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
Review comment:
char *backup_file;
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is not rotating the log, but only creating a backup of the last log file.
I proper rotating log should define how many log files the system will support and it should always move all the previous log files to log.NNN + 1. It could happen until the max log files supported is reached. When it happen the oldest log file is deleted and replaced with oldest - 1.
If you don't want to implement it, I suggest rename the function from rotate log to backup log, to avoid confusion :-)
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is not rotating the log, but only creating a backup of the last log file.
A proper rotating log should define how many log files the system will support and it should always move all the previous log files to log.NNN + 1. It could happen until the max log files supported is reached. When it happen the oldest log file is deleted and replaced with oldest - 1.
If you don't want to implement it, I suggest rename the function from rotate log to backup log, to avoid confusion :-)
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
It is not rotating the log, but only creating a backup of the last log file.
A proper rotating log should define how many log files the system will support and it should always move all the previous log files to log.NNN + 1. It could happen until the max log files supported is reached. When it happen the oldest log file is deleted and replaced with oldest - 1.
If you don't want to implement it, I suggest renaming the function and the Kconfig from rotate log to backup log, to avoid confusion :-)
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
+
+ /* Get the size of the current log file. */
+
+ fd = open(log_file, O_RDONLY);
+ if (fd < 0)
+ {
+ return;
+ }
+
+ fstat(fd, &f_stat);
+ size = f_stat.st_size;
+ close(fd);
+
+ /* If it does not exceed the limit we are OK. */
+
+ if (size < CONFIG_SYSLOG_FILE_SIZE_LIMIT)
+ {
+ return;
+ }
+
+ /* Construct the backup file name. */
+
+ backup_file = malloc(strlen(log_file) + 3);
+ if (backup_file == NULL)
+ {
+ return;
+ }
+
+ sprintf(backup_file, "%s.0", log_file);
+
+ /* Delete any old backup files. */
+
+ if (access(backup_file, F_OK) == 0)
+ {
+ remove(backup_file);
+ }
+
+ /* Rotate the log. */
+
+ rename(log_file, backup_file);
+
+ free(backup_file);
Review comment:
> It is still a rotation, it just happens that there is only one rotation available. :)
>
> Renaming this as "backup" would imply that this is a backup functionality. Which is not.
> A backup would also keep the original file in place, while here it is deleted.
>
> This can be very easily be extended to keep multiple rotations, instead of only one.
> I have done this in the past, and I can easily add it to this implementation.
>
> But I intentionally left it out. Do we really need so many log files?
> If you think so, I will add it tomorrow.
I think it should be useful for some system. Imagine a system were we keep it in the field and after some amount to days (i.e. 30 days) you can go there and get the complete log history.
I can merge this one and wait for your improvement over it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645476406
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -108,6 +168,12 @@ FAR struct syslog_channel_s *syslog_file_channel(FAR const char *devpath)
syslog_dev_uninitialize(g_syslog_file_channel);
}
+ /* Rotate the log file, if needed. */
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+ log_rotate(devpath);
Review comment:
> It isn't rare, many IoT product will run a long time.
I mean, it is rare to have a system run for so long AND produce so much *system* logs at the same time that fill up today's huge memories. Most IoT devices are low-power, and do not have very verbose logs...
To be honest, I don't ever expect a several GB SD to fill-up. My primary concern is to somehow constrain the log size, in case I need to upload it to an external server. So it doesn't get into gigabyte scale after several months/years.
This is why I don't emphasize on log size accuracy, rather than performance.
> The rename process for all filesystem implementaiton should be fast and not relate to the file size.
I am not that much concerned about how fast the operation is. Rather that it is an **extra** step to take, that occurs unpredictably.
This is what affects real-time performance. You calculate/measure the timings of the application, and then randomly they may be violated.
Another thing is that `syslog_device` knows nothing about files. It may log to a file, or maybe to a character device.
The only place that we know for sure that the log is on a file, and that it needs rotation is in `syslog_file_channel`.
So how can `syslog_device` perform the rotation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] fjpanag commented on pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
fjpanag commented on pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#issuecomment-854063840
Just tested it on a custom STM32F4 target.
It works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] acassis commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
acassis commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645026965
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -50,6 +54,62 @@
FAR static struct syslog_channel_s *g_syslog_file_channel;
+/****************************************************************************
+ * Private Functions
+ ****************************************************************************/
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+static void log_rotate(FAR const char *log_file)
+{
+ int fd;
+ off_t size;
+ struct stat f_stat;
+ char * backup_file;
Review comment:
char *backup_file;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-nuttx] xiaoxiang781216 commented on a change in pull request #3842: Added automatic log rotation, when log file is opened.
Posted by GitBox <gi...@apache.org>.
xiaoxiang781216 commented on a change in pull request #3842:
URL: https://github.com/apache/incubator-nuttx/pull/3842#discussion_r645458932
##########
File path: drivers/syslog/syslog_filechannel.c
##########
@@ -108,6 +168,12 @@ FAR struct syslog_channel_s *syslog_file_channel(FAR const char *devpath)
syslog_dev_uninitialize(g_syslog_file_channel);
}
+ /* Rotate the log file, if needed. */
+
+#ifdef CONFIG_SYSLOG_FILE_ROTATE
+ log_rotate(devpath);
Review comment:
> Theoretically you are right. This can happen.
> However, in practice this is very rare, and I prefer to keep the simple solution.
>
It isn't rare, many IoT product will run a long time.
> What you describe is a good idea, but I am afraid that it won't work easily/correctly.
> There is no easy way to manipulate the log after it is opened without side-effects:
The action is simple:
1. Check the total size after writing log
2. close/rename/open file if the limitation reach
>
> * Either the system will block during the rotation, possibly harming high priority threads.
> * Or there will be lost logs.
>
The rename process for all filesystem implementaiton should be fast and not relate to the file size.
> Unfortunately, it is not possible to predict when the rotation will take place. This will have an impact on the real-time behavior of the system. Keeping the rotations during opening only keeps the system predictable.
Normally, write data to file isn't simpler than rename a file. So if renaming impact the real-time behaviour very much, write data should have the same impact.
If the real-time is a concern, the right approach is:
1. Send the log to ramlog
2. log service read ramlog and write to file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org