You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "ZDandsomSP (via GitHub)" <gi...@apache.org> on 2023/09/20 10:50:26 UTC
[GitHub] [iotdb] ZDandsomSP opened a new pull request, #11184: [mlnode] Logging module, shut down function and the fixing of a bug about multi process training
ZDandsomSP opened a new pull request, #11184:
URL: https://github.com/apache/iotdb/pull/11184
### Description
Logging module, shut down function and the fixing of a bug about multi process training
We have developed new functions for the mlnode module of IoTDB, which mainly includes the following three parts:
### Content
#### Logging module
Developed a logging module for mlnode, supporting standard function calls and localized log information for multi-process security
#### Shut down function
Completed the addition of the mlnode shutdown function, and implemented the direct shutdown of the mlnode main process based on the port number on the Linux system
#### Bug fixing
Fixed the issue involved in related work, which is that the original process pool does not support creating nums_ Issues with worker subprocesses
### Usage
#### Logging module
We provide 5 levels of functions including debug, info, warn, error, and the module can localize logs according to the log level. In subsequent development, we only need to use a form similar to calling the Python standard logging module to use it.
#### Stop function
Since our start function is designed in the form of pip package, our stop function is developed in a similar form. If user start the mlnode in a terminal, we can stop the node in another one with the command "mlnode stop".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] ZDandsomSP commented on a diff in pull request #11184: [mlnode] Logging module, shut down function and the fixing of a bug about multi process training
Posted by "ZDandsomSP (via GitHub)" <gi...@apache.org>.
ZDandsomSP commented on code in PR #11184:
URL: https://github.com/apache/iotdb/pull/11184#discussion_r1331491860
##########
iotdb-core/mlnode/iotdb/mlnode/log.py:
##########
@@ -17,18 +17,106 @@
#
import logging
import os
-from logging.config import fileConfig
+import random
+import sys
+import multiprocessing
+import inspect
-from iotdb.mlnode.constant import (MLNODE_CONF_DIRECTORY_NAME,
- MLNODE_LOG_CONF_FILE_NAME)
+# log directory
+LOG_DIR = "./logs"
+# level fot stdout
+STD_LEVEL = logging.INFO
-log_conf_file = os.path.join(os.getcwd(), MLNODE_CONF_DIRECTORY_NAME, MLNODE_LOG_CONF_FILE_NAME)
-if os.path.exists(log_conf_file):
- fileConfig(log_conf_file)
-else:
- logging.basicConfig(
- level=logging.DEBUG,
- format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
- )
-logger = logging.getLogger()
+class LoggerFilter(logging.Filter):
+ def filter(self, record):
+ record.msg = f"{self.custom_log_info()} Message: {record.msg}"
+ return True
+
+ @staticmethod
+ def custom_log_info():
+ frame = inspect.currentframe()
+ stack_trace = inspect.getouterframes(frame)
+
+ pid = os.getpid()
+ process_name = multiprocessing.current_process().name
+
+ # 格式化函数调用栈信息
+ stack_info = ""
+ frame_info = stack_trace[7] # 忽略前两个栈帧
Review Comment:
We are already aware of these issues, and these fixes are already in our plan. We will correct these obvious deficiencies in subsequent commits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a diff in pull request #11184: [mlnode] Logging module, shut down function and the fixing of a bug about multi process training
Posted by "HTHou (via GitHub)" <gi...@apache.org>.
HTHou commented on code in PR #11184:
URL: https://github.com/apache/iotdb/pull/11184#discussion_r1331489456
##########
iotdb-core/mlnode/iotdb/mlnode/log.py:
##########
@@ -17,18 +17,106 @@
#
import logging
import os
-from logging.config import fileConfig
+import random
+import sys
+import multiprocessing
+import inspect
-from iotdb.mlnode.constant import (MLNODE_CONF_DIRECTORY_NAME,
- MLNODE_LOG_CONF_FILE_NAME)
+# log directory
+LOG_DIR = "./logs"
+# level fot stdout
+STD_LEVEL = logging.INFO
-log_conf_file = os.path.join(os.getcwd(), MLNODE_CONF_DIRECTORY_NAME, MLNODE_LOG_CONF_FILE_NAME)
-if os.path.exists(log_conf_file):
- fileConfig(log_conf_file)
-else:
- logging.basicConfig(
- level=logging.DEBUG,
- format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
- )
-logger = logging.getLogger()
+class LoggerFilter(logging.Filter):
+ def filter(self, record):
+ record.msg = f"{self.custom_log_info()} Message: {record.msg}"
+ return True
+
+ @staticmethod
+ def custom_log_info():
+ frame = inspect.currentframe()
+ stack_trace = inspect.getouterframes(frame)
+
+ pid = os.getpid()
+ process_name = multiprocessing.current_process().name
+
+ # 格式化函数调用栈信息
+ stack_info = ""
+ frame_info = stack_trace[7] # 忽略前两个栈帧
Review Comment:
Please change the comments to English.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org