You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by ma...@apache.org on 2022/12/16 08:02:05 UTC
[iotdb] branch master updated: [IOTDB-4595] Add monitoring for compaction temporal files (#8414)
This is an automated email from the ASF dual-hosted git repository.
marklau99 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iotdb.git
The following commit(s) were added to refs/heads/master by this push:
new 752d7cf9a5 [IOTDB-4595] Add monitoring for compaction temporal files (#8414)
752d7cf9a5 is described below
commit 752d7cf9a5ac055dce790188234893e48be0a55a
Author: Liu Xuxin <37...@users.noreply.github.com>
AuthorDate: Fri Dec 16 16:02:00 2022 +0800
[IOTDB-4595] Add monitoring for compaction temporal files (#8414)
---
docs/UserGuide/Monitor-Alert/Metric-Tool.md | 250 ++++++++++++---------
docs/zh/UserGuide/Monitor-Alert/Metric-Tool.md | 179 ++++++++-------
.../iotdb/db/engine/TsFileMetricManager.java | 54 ++++-
.../utils/AlignedSeriesCompactionExecutor.java | 6 +
.../utils/SingleSeriesCompactionExecutor.java | 1 +
.../performer/impl/FastCompactionPerformer.java | 16 +-
.../impl/ReadChunkCompactionPerformer.java | 11 +
.../impl/ReadPointCompactionPerformer.java | 19 ++
.../writer/AbstractCompactionWriter.java | 2 +
.../writer/AbstractCrossCompactionWriter.java | 9 +
.../writer/AbstractInnerCompactionWriter.java | 5 +
.../iotdb/db/engine/storagegroup/DataRegion.java | 2 +
.../db/engine/storagegroup/TsFileProcessor.java | 2 -
.../iotdb/db/service/metrics/FileMetrics.java | 97 ++++++++
14 files changed, 453 insertions(+), 200 deletions(-)
diff --git a/docs/UserGuide/Monitor-Alert/Metric-Tool.md b/docs/UserGuide/Monitor-Alert/Metric-Tool.md
index 0c3b75ade0..802e322c14 100644
--- a/docs/UserGuide/Monitor-Alert/Metric-Tool.md
+++ b/docs/UserGuide/Monitor-Alert/Metric-Tool.md
@@ -19,7 +19,9 @@
-->
-Along with IoTDB running, we hope to observe the status of IoTDB, so as to troubleshoot system problems or discover potential system risks in time. A series of metrics that can **reflect the operating status of the system** are system monitoring metrics.
+Along with IoTDB running, we hope to observe the status of IoTDB, so as to troubleshoot system problems or discover
+potential system risks in time. A series of metrics that can **reflect the operating status of the system** are system
+monitoring metrics.
## 1. When to use metric framework?
@@ -27,22 +29,26 @@ Belows are some typical application scenarios
1. System is running slowly
- When system is running slowly, we always hope to have information about system's running status as detail as possible, such as:
+ When system is running slowly, we always hope to have information about system's running status as detail as
+ possible, such as:
- - JVM：Is there FGC? How long does it cost? How much does the memory usage decreased after GC? Are there lots of threads?
- - System：Is the CPU usage too hi？Are there many disk IOs？
- - Connections：How many connections are there in the current time？
- - Interface：What is the TPS and latency of every interface？
- - Thread Pool：Are there many pending tasks？
- - Cache Hit Ratio
+ - JVM：Is there FGC? How long does it cost? How much does the memory usage decreased after GC? Are there lots of
+ threads?
+ - System：Is the CPU usage too hi？Are there many disk IOs？
+ - Connections：How many connections are there in the current time？
+ - Interface：What is the TPS and latency of every interface？
+ - Thread Pool：Are there many pending tasks？
+ - Cache Hit Ratio
2. No space left on device
- When meet a "no space left on device" error, we really want to know which kind of data file had a rapid rise in the past hours.
+ When meet a "no space left on device" error, we really want to know which kind of data file had a rapid rise in the
+ past hours.
3. Is the system running in abnormal status
- We could use the count of error logs、the alive status of nodes in cluster, etc, to determine whether the system is running abnormally.
+ We could use the count of error logs、the alive status of nodes in cluster, etc, to determine whether the system is
+ running abnormally.
## 2. Who will use metric framework?
Any person cares about the system's status, including but not limited to RD, QA, SRE, DBA, can use the metrics to work more efficiently.
@@ -68,15 +74,17 @@ In IoTDB's metric module, each metrics is uniquely identified by `Metric Name` a
### 3.2. External data format for metrics
- IoTDB provides metrics in JMX, Prometheus and IoTDB formats:
- - For JMX, metrics can be obtained through ```org.apache.iotdb.metrics```.
- - For Prometheus, the value of the metrics can be obtained through the externally exposed port
- - External exposure in IoTDB mode: metrics can be obtained by executing IoTDB queries
+ - For JMX, metrics can be obtained through ```org.apache.iotdb.metrics```.
+ - For Prometheus, the value of the metrics can be obtained through the externally exposed port
+ - External exposure in IoTDB mode: metrics can be obtained by executing IoTDB queries
## 4. The detail of metrics
-Currently, IoTDB provides metrics for some main modules externally, and with the development of new functions and system optimization or refactoring, metrics will be added and updated synchronously.
+Currently, IoTDB provides metrics for some main modules externally, and with the development of new functions and system
+optimization or refactoring, metrics will be added and updated synchronously.
-If you want to add your own metrics data in IoTDB, please see the [IoTDB Metric Framework] (https://github.com/apache/iotdb/tree/master/metrics) document.
+If you want to add your own metrics data in IoTDB, please see
+the [IoTDB Metric Framework] (https://github.com/apache/iotdb/tree/master/metrics) document.
### 4.1. Core level metrics
Core-level metrics are enabled by default during system operation. The addition of each Core-level metrics needs to be carefully evaluated. The current Core-level metrics are as follows:
@@ -187,7 +195,6 @@ Core-level metrics are enabled by default during system operation. The addition
| queue | name="flush",status="running/waiting" | AutoGauge | The number of flush tasks |
| queue | name="Sub_RawQuery",status="running/waiting" | AutoGauge | The number of Sub_RawQuery |
-
#### 4.2.8. Compaction
| Metric | Tags | Type | Description |
| --------------------- | --------------------------------------------------- | ------- | -------------------------------------- |
@@ -198,16 +205,24 @@ Core-level metrics are enabled by default during system operation. The addition
| compaction_task_count | name = "cross_compaction", type="cross" | Counter | The number of corss compction |
#### 4.2.9. File
-| Metric | Tags | Type | Description |
-| ---------- | ------------ | --------- | ----------------------------------------- |
-| file_size | name="wal" | AutoGauge | The size of WAL file, Unit: byte |
-| file_size | name="seq" | AutoGauge | The size of sequence TsFile, Unit: byte |
-| file_size | name="unseq" | AutoGauge | The size of unsequence TsFile, Unit: byte |
-| file_count | name="wal" | AutoGauge | The count of WAL file |
-| file_count | name="seq" | AutoGauge | The count of sequence TsFile |
-| file_count | name="unseq" | AutoGauge | The count of unsequence TsFile |
+
+| Metric | Tags | Type | Description |
+| ---------- | ------------ | --------- |--------------------------------------------------------------|
+| file_size | name="wal" | AutoGauge | The size of WAL file, Unit: byte |
+| file_size | name="seq" | AutoGauge | The size of sequence TsFile, Unit: byte |
+| file_size | name="unseq" | AutoGauge | The size of unsequence TsFile, Unit: byte |
+| file_count | name="wal" | AutoGauge | The count of WAL file |
+| file_count | name="seq" | AutoGauge | The count of sequence TsFile |
+| file_count | name="unseq" | AutoGauge | The count of unsequence TsFile |
+| file_count| name="inner-seq-temp-num" | AutoGauge | The count of inner sequence space compaction temporal file |
+| file_count | name="inner-unseq-temp-num"|AutoGauge| The count of inner unsequence space compaction temporal file |
+| file_count|name="cross-temp-num"|AutoGauge| The count of cross space compaction temporal file |
+| file_size|name="inner-seq-temp-size" | AutoGauge| The size of inner sequence space compaction temporal file |
+| file_size|name="inner-unseq-temp-size"|AutoGauge| The size of inner unsequence space compaction temporal file |
+|file_size|name="cross-temp-size"|AutoGauge|The size of cross space compaction temoporal file|
#### 4.2.10. IoTDB Process
+
| Metric | Tags | Type | Description |
| --------------------- | -------------- | --------- | ------------------------------------------- |
| process_used_mem | name="memory" | AutoGauge | The used memory of IoTDB process |
@@ -275,7 +290,8 @@ Currently there is no All level metrics, and it will continue to be added in the
## 5. How to get these metrics？
-The relevant configuration of the metric module is in `conf/iotdb-{datanode/confignode}.properties`, and all configuration items support hot loading through the `load configuration` command.
+The relevant configuration of the metric module is in `conf/iotdb-{datanode/confignode}.properties`, and all
+configuration items support hot loading through the `load configuration` command.
### 5.1. JMX
For metrics exposed externally using JMX, you can view them through Jconsole. After entering the Jconsole monitoring page, you will first see an overview of various running conditions of IoTDB. Here you can see heap memory information, thread information, class information, and the server's CPU usage.
@@ -290,13 +306,14 @@ After connecting to JMX, you can find the "MBean" named "org.apache.iotdb.servic
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/46039728/149951720-707f1ee8-32ee-4fde-9252-048caebd232e.png"> <br>
-In order to improve query performance, IOTDB caches ChunkMetaData and TsFileMetaData. Users can use MXBean and expand the sidebar `org.apache.iotdb.db.service` to view the cache hit ratio:
+In order to improve query performance, IOTDB caches ChunkMetaData and TsFileMetaData. Users can use MXBean and expand
+the sidebar `org.apache.iotdb.db.service` to view the cache hit ratio:
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://user-images.githubusercontent.com/19167280/112426760-73e3da80-8d73-11eb-9a8f-9232d1f2033b.png">
### 5.2. Prometheus
-#### 5.2.1. The mapping from metric type to prometheus forma
+#### 5.2.1. The mapping from metric type to prometheus format
> For metrics whose Metric Name is name and Tags are K1=V1, ..., Kn=Vn, the mapping is as follows, where value is a specific value
| Metric Type | Mapping |
@@ -319,8 +336,8 @@ dn_metric_prometheus_reporter_port=9091
Then you can get metrics data as follows
2) Start IoTDB DataNodes
-3) Open a browser or use ```curl``` to visit ```http://servier_ip:9091/metrics```, you can get the following metric data:
-
+3) Open a browser or use ```curl``` to visit ```http://servier_ip:9091/metrics```, you can get the following metric
+ data:
```
...
@@ -334,7 +351,8 @@ file_count{name="seq",} 2.0
#### 5.2.3. Prometheus + Grafana
-As shown above, IoTDB exposes monitoring metrics data in the standard Prometheus format to the outside world. Prometheus can be used to collect and store monitoring indicators, and Grafana can be used to visualize monitoring indicators.
+As shown above, IoTDB exposes monitoring metrics data in the standard Prometheus format to the outside world. Prometheus
+can be used to collect and store monitoring indicators, and Grafana can be used to visualize monitoring indicators.
The following picture describes the relationships among IoTDB, Prometheus and Grafana
@@ -343,7 +361,8 @@ The following picture describes the relationships among IoTDB, Prometheus and Gr
1. Along with running, IoTDB will collect its metrics continuously.
2. Prometheus scrapes metrics from IoTDB at a constant interval (can be configured).
3. Prometheus saves these metrics to its inner TSDB.
-4. Grafana queries metrics from Prometheus at a constant interval (can be configured) and then presents them on the graph.
+4. Grafana queries metrics from Prometheus at a constant interval (can be configured) and then presents them on the
+ graph.
So, we need to do some additional works to configure and deploy Prometheus and Grafana.
@@ -359,8 +378,8 @@ metrics_path: /metrics
scheme: http
follow_redirects: true
static_configs:
-- targets:
- - localhost:9091
+ - targets:
+ - localhost:9091
```
The following documents may help you have a good journey with Prometheus and Grafana.
@@ -380,95 +399,112 @@ We provide the Apache IoTDB Dashboard, and the rendering shown in Grafana is as
##### 5.2.4.1. How to get Apache IoTDB Dashboard
-1. You can obtain the json files of Dashboards corresponding to different iotdb versions in the grafana-metrics-example folder.
-2. You can visit [Grafana Dashboard official website](https://grafana.com/grafana/dashboards/), search for `Apache IoTDB Dashboard` and use
+1. You can obtain the json files of Dashboards corresponding to different iotdb versions in the grafana-metrics-example
+ folder.
+2. You can visit [Grafana Dashboard official website](https://grafana.com/grafana/dashboards/), search
+ for `Apache IoTDB Dashboard` and use
-When creating Grafana, you can select the json file you just downloaded to `Import` and select the corresponding target data source for Apache IoTDB Dashboard.
+When creating Grafana, you can select the json file you just downloaded to `Import` and select the corresponding target
+data source for Apache IoTDB Dashboard.
##### 5.2.4.2. Apache IoTDB ConfigNode Dashboard Instructions
> Except for the metrics specified specially, the following metrics are guaranteed to be available in the monitoring framework at the Important levels.
- `Overview`: system overview
- - `Registered Node`: The number of registered ConfigNode/DataNode
- - `DataNode`: The status of the cluster DataNode, including Online and Unknown.
- - `ConfigNode`: The status of the cluster ConfigNode, including Online and Unknown.
- - `The Status Of Node`: The status of specific nodes in the cluster, including Online and Unknown.
+ - `Registered Node`: The number of registered ConfigNode/DataNode
+ - `DataNode`: The status of the cluster DataNode, including Online and Unknown.
+ - `ConfigNode`: The status of the cluster ConfigNode, including Online and Unknown.
+ - `The Status Of Node`: The status of specific nodes in the cluster, including Online and Unknown.
- `Region`: Region overview
- - `Region Number`: the number of Regions, including the total number, the number of DataRegions and the number of SchemaRegions.
- - `Leadership distribution`: Cluster leader distribution, which refers to the number of Leaders corresponding to the Region on each node.
- - `Total Region in Node`: The total number of Regions of different Nodes.
- - `Region in Node`: the number of SchemaRegions/DataRegions of different Nodes.
- - `Region in Database` (Normal level): the number of Regions in different Databases, including SchemaRegion and DataRegion.
- - `Slot in Database` (Normal level): The number of Slots in different Databases, including the number of DataSlots and SchemaSlots.
+ - `Region Number`: the number of Regions, including the total number, the number of DataRegions and the number of
+ SchemaRegions.
+ - `Leadership distribution`: Cluster leader distribution, which refers to the number of Leaders corresponding to the
+ Region on each node.
+ - `Total Region in Node`: The total number of Regions of different Nodes.
+ - `Region in Node`: the number of SchemaRegions/DataRegions of different Nodes.
+ - `Region in Database` (Normal level): the number of Regions in different Databases, including SchemaRegion and
+ DataRegion.
+ - `Slot in Database` (Normal level): The number of Slots in different Databases, including the number of DataSlots
+ and SchemaSlots.
- `System`: system
- - `CPU Core`: the number of CPU cores in the system.
- - `CPU Load`: system CPU load, progress CPU load.
- - `CPU Time Per Minute`: The process takes up the system CPU time per minute on average. Note: multi-core will cause this value to exceed 1 minute.
- - `System Memory`: the physical memory size of the system, the physical memory size used by the system, and the memory size submitted by the virtual machine.
- - `System Swap Size`: the total size of the system swap area, the size used by the system swap area.
- - `Process Memory`: the maximum total memory size of the IoTDB process, the total memory size of the IoTDB process, and the memory size used by the IoTDB process.
- - `The Number of GC Per Minute`: The average number of GC per minute.
- - `The Time Consumed Of GC Per Minute`: Average GC time spent per minute.
- - `The Number Of Java Thread`: The number of threads in different states of the IoTDB process.
- - `Heap Memory`: the heap memory of the IoTDB process
- - `Off Heap Memory`: the off-heap memory of the IoTDB process
- - `Log Number Per Minute`: the average number of logs per minute of the IoTDB process
- - `The Time Consumed of Compliation Per Minute`: average compilation time per minute
- - `The Number Of Class`: The number of classes loaded and unloaded by the JVM
-
+ - `CPU Core`: the number of CPU cores in the system.
+ - `CPU Load`: system CPU load, progress CPU load.
+ - `CPU Time Per Minute`: The process takes up the system CPU time per minute on average. Note: multi-core will cause
+ this value to exceed 1 minute.
+ - `System Memory`: the physical memory size of the system, the physical memory size used by the system, and the
+ memory size submitted by the virtual machine.
+ - `System Swap Size`: the total size of the system swap area, the size used by the system swap area.
+ - `Process Memory`: the maximum total memory size of the IoTDB process, the total memory size of the IoTDB process,
+ and the memory size used by the IoTDB process.
+ - `The Number of GC Per Minute`: The average number of GC per minute.
+ - `The Time Consumed Of GC Per Minute`: Average GC time spent per minute.
+ - `The Number Of Java Thread`: The number of threads in different states of the IoTDB process.
+ - `Heap Memory`: the heap memory of the IoTDB process
+ - `Off Heap Memory`: the off-heap memory of the IoTDB process
+ - `Log Number Per Minute`: the average number of logs per minute of the IoTDB process
+ - `The Time Consumed of Compliation Per Minute`: average compilation time per minute
+ - `The Number Of Class`: The number of classes loaded and unloaded by the JVM
+
##### 5.2.4.3. Apache IoTDB DataNode Dashboard Instructions
> Except for the metrics specified specially, the following metrics are guaranteed to be available in the monitoring framework at the Important levels.
- `Overview`: system overview
- - `The Number Of Entity`: the number of entities, including time series, etc.
- - `Write Point Per Minute`: the average number of system write points per minute
- - `Database Used Memory`: the memory size used by each Database
+ - `The Number Of Entity`: the number of entities, including time series, etc.
+ - `Write Point Per Minute`: the average number of system write points per minute
+ - `Database Used Memory`: the memory size used by each Database
- `Interface`: interface
- - `The Time Consumed Of Operation(50%)`: Median time spent by different client operations
- - `The Time Consumed Of Operation(75%)`: The upper quartile of the time consumed by different client operations
- - `The Time Consumed Of Operation(100%)`: The maximum time spent by different client operations
- - `The QPS of Interface`: system interface visits per second
- - `The Time Consumed Of Interface`: the average time consumed by the system interface
- - `Cache Hit Rate`: cache hit rate
- - `Thrift Connection`: the number of Thrift connections established
- - `Thrift Active Thread`: The number of active Thrift connections established
-- `Engine`:
- - `Task Number`: the number of tasks in different states in the system
- - `The Time Consumed Of Tasking`: Time consumption of tasks in different states in the system
- - `Compaction Read And Write Per Minute`: the average amount of combined read and write data per minute
- - `Compaction R/W Ratio Per Minute`: The average ratio of combined read and write data per minute
- - `Compaction Number Per Minute`: the average number of different types of consolidation tasks per minute
+ - `The Time Consumed Of Operation(50%)`: Median time spent by different client operations
+ - `The Time Consumed Of Operation(75%)`: The upper quartile of the time consumed by different client operations
+ - `The Time Consumed Of Operation(100%)`: The maximum time spent by different client operations
+ - `The QPS of Interface`: system interface visits per second
+ - `The Time Consumed Of Interface`: the average time consumed by the system interface
+ - `Cache Hit Rate`: cache hit rate
+ - `Thrift Connection`: the number of Thrift connections established
+ - `Thrift Active Thread`: The number of active Thrift connections established
+- `Engine`:
+ - `Task Number`: the number of tasks in different states in the system
+ - `The Time Consumed Of Tasking`: Time consumption of tasks in different states in the system
+ - `Compaction Read And Write Per Minute`: the average amount of combined read and write data per minute
+ - `Compaction R/W Ratio Per Minute`: The average ratio of combined read and write data per minute
+ - `Compaction Number Per Minute`: the average number of different types of consolidation tasks per minute
- `IoTConsensus`：
- - `IoTConsensus Used Memory`：The size of the memory used by IoTConsensus consensus
- - `IoTConsensus Sync Index`：the searchIndex and safeIndex of region
- - `IoTConsensus Overview`：The total sync lag and total size of buffered requests of node
- - `The time consumed of different stages(50%)`：The median of the time consumed of different stages
- - `The time consumed of different stages(75%)`：The upper quartile of the time consumed of different stages
- - `The time consumed of different stages(100%)`：The max of the time consumed of different stages
- - `IoTConsensus Search Index Rate`：The increasing rate of searchIndex of region
- - `IoTConsensus Safe Index Rate`：The increasing rate of safeIndex of region
- - `IoTConsensus LogDispatcher Request Size`：The number of requests buffered in logDispatcher
- - `Sync Lag`：The sync lag of region
- - `Min Peer Sync Lag`：The sync lag between the searchIndex of IoTConsensusServerImpl and the max currentSyncIndex of LogDispatcher
- - `Sync speed diff of Peers`：The sync lag between the max currentSyncIndex of LogDispatcher and the min currentSyncIndex of LogDispatcher
+ - `IoTConsensus Used Memory`：The size of the memory used by IoTConsensus consensus
+ - `IoTConsensus Sync Index`：the searchIndex and safeIndex of region
+ - `IoTConsensus Overview`：The total sync lag and total size of buffered requests of node
+ - `The time consumed of different stages(50%)`：The median of the time consumed of different stages
+ - `The time consumed of different stages(75%)`：The upper quartile of the time consumed of different stages
+ - `The time consumed of different stages(100%)`：The max of the time consumed of different stages
+ - `IoTConsensus Search Index Rate`：The increasing rate of searchIndex of region
+ - `IoTConsensus Safe Index Rate`：The increasing rate of safeIndex of region
+ - `IoTConsensus LogDispatcher Request Size`：The number of requests buffered in logDispatcher
+ - `Sync Lag`：The sync lag of region
+ - `Min Peer Sync Lag`：The sync lag between the searchIndex of IoTConsensusServerImpl and the max currentSyncIndex of
+ LogDispatcher
+ - `Sync speed diff of Peers`：The sync lag between the max currentSyncIndex of LogDispatcher and the min
+ currentSyncIndex of LogDispatcher
- `System`: system
- - `CPU Core`: the number of CPU cores in the system.
- - `CPU Load`: system CPU load, progress CPU load.
- - `CPU Time Per Minute`: The process takes up the system CPU time per minute on average. Note: multi-core will cause this value to exceed 1 minute.
- - `System Memory`: the physical memory size of the system, the physical memory size used by the system, and the memory size submitted by the virtual machine.
- - `System Swap Size`: the total size of the system swap area, the size used by the system swap area.
- - `Process Memory`: the maximum total memory size of the IoTDB process, the total memory size of the IoTDB process, and the memory size used by the IoTDB process.
- - `The Size Of File`: IoTDB system-related file size, including the total file size under wal, the total size of tsfile files under seq, and the total size of tsfile files under unseq
- - `The Number Of File`: the number of files related to the IoTDB system, including the number of files under wal, the number of tsfile files under seq, and the number of tsfile files under unseq
- - `The Space Of Disk`: the total size and remaining size of the disk mounted in the current data directory
- - `The Number of GC Per Minute`: The average number of GC per minute.
- - `The Time Consumed Of GC Per Minute`: Average GC time spent per minute.
- - `The Number Of Java Thread`: The number of threads in different states of the IoTDB process.
- - `Heap Memory`: the heap memory of the IoTDB process
- - `Off Heap Memory`: the off-heap memory of the IoTDB process
- - `Log Number Per Minute`: the average number of logs per minute of the IoTDB process
- - `The Time Consumed of Compliation Per Minute`: average compilation time per minute
- - `The Number Of Class`: The number of classes loaded and unloaded by the JVM
+ - `CPU Core`: the number of CPU cores in the system.
+ - `CPU Load`: system CPU load, progress CPU load.
+ - `CPU Time Per Minute`: The process takes up the system CPU time per minute on average. Note: multi-core will cause
+ this value to exceed 1 minute.
+ - `System Memory`: the physical memory size of the system, the physical memory size used by the system, and the
+ memory size submitted by the virtual machine.
+ - `System Swap Size`: the total size of the system swap area, the size used by the system swap area.
+ - `Process Memory`: the maximum total memory size of the IoTDB process, the total memory size of the IoTDB process,
+ and the memory size used by the IoTDB process.
+ - `The Size Of File`: IoTDB system-related file size, including the total file size under wal, the total size of
+ tsfile files under seq, and the total size of tsfile files under unseq
+ - `The Number Of File`: the number of files related to the IoTDB system, including the number of files under wal,
+ the number of tsfile files under seq, and the number of tsfile files under unseq
+ - `The Space Of Disk`: the total size and remaining size of the disk mounted in the current data directory
+ - `The Number of GC Per Minute`: The average number of GC per minute.
+ - `The Time Consumed Of GC Per Minute`: Average GC time spent per minute.
+ - `The Number Of Java Thread`: The number of threads in different states of the IoTDB process.
+ - `Heap Memory`: the heap memory of the IoTDB process
+ - `Off Heap Memory`: the off-heap memory of the IoTDB process
+ - `Log Number Per Minute`: the average number of logs per minute of the IoTDB process
+ - `The Time Consumed of Compliation Per Minute`: average compilation time per minute
+ - `The Number Of Class`: The number of classes loaded and unloaded by the JVM
### 5.3. IoTDB
diff --git a/docs/zh/UserGuide/Monitor-Alert/Metric-Tool.md b/docs/zh/UserGuide/Monitor-Alert/Metric-Tool.md
index 045b13a8d6..5f931de572 100644
--- a/docs/zh/UserGuide/Monitor-Alert/Metric-Tool.md
+++ b/docs/zh/UserGuide/Monitor-Alert/Metric-Tool.md
@@ -66,16 +66,17 @@
- `Important`：模块的重要指标，供**运维和测试人员**使用，直接关乎**每个模块的运行状态**，比如合并文件个数、执行情况等。
- `Normal`：模块的一般指标，供**开发人员**使用，方便在出现问题时**定位模块**，比如合并中的特定关键操作情况。
- `All`：模块的全部指标，供**模块开发人员**使用，往往在复现问题的时候使用，从而快速解决问题。
-
+
### 3.2. 监控指标对外获取数据格式
- IoTDB 对外提供 JMX、 Prometheus 和 IoTDB 格式的监控指标：
- - 对于 JMX ，可以通过```org.apache.iotdb.metrics```获取系统监控指标指标。
- - 对于 Prometheus ，可以通过对外暴露的端口获取监控指标的值
- - 对于 IoTDB 方式对外暴露：可以通过执行 IoTDB 的查询来获取监控指标
+ - 对于 JMX ，可以通过```org.apache.iotdb.metrics```获取系统监控指标指标。
+ - 对于 Prometheus ，可以通过对外暴露的端口获取监控指标的值
+ - 对于 IoTDB 方式对外暴露：可以通过执行 IoTDB 的查询来获取监控指标
## 4. 监控指标有哪些？
-目前，IoTDB 对外提供一些主要模块的监控指标，并且随着新功能的开发以及系统优化或者重构，监控指标也会同步添加和更新。如果想自己在 IoTDB 中添加更多系统监控指标埋点，可以参考[IoTDB Metrics Framework](https://github.com/apache/iotdb/tree/master/metrics)使用说明。
+目前，IoTDB 对外提供一些主要模块的监控指标，并且随着新功能的开发以及系统优化或者重构，监控指标也会同步添加和更新。如果想自己在 IoTDB
+中添加更多系统监控指标埋点，可以参考[IoTDB Metrics Framework](https://github.com/apache/iotdb/tree/master/metrics)使用说明。
### 4.1. Core 级别监控指标
@@ -200,14 +201,21 @@ Core 级别的监控指标在系统运行中默认开启，每一个 Core 级别
| compaction_task_count | name = "cross_compaction", type="cross" | Counter | 跨空间合并次数 |
#### 4.2.9. 文件统计信息
-| Metric | Tags | Type | Description |
-| ---------- | ------------ | --------- | ---------------------------- |
-| file_size | name="wal" | AutoGauge | 写前日志总大小，单位为byte |
-| file_size | name="seq" | AutoGauge | 顺序TsFile总大小，单位为byte |
-| file_size | name="unseq" | AutoGauge | 乱序TsFile总大小，单位为byte |
-| file_count | name="wal" | AutoGauge | 写前日志文件个数 |
-| file_count | name="seq" | AutoGauge | 顺序TsFile文件个数 |
-| file_count | name="unseq" | AutoGauge | 乱序TsFile文件个数 |
+
+| Metric | Tags | Type | Description |
+| ---------- |------------------------------| --------- |----------------------|
+| file_size | name="wal" | AutoGauge | 写前日志总大小，单位为byte |
+| file_size | name="seq" | AutoGauge | 顺序TsFile总大小，单位为byte |
+| file_size | name="unseq" | AutoGauge | 乱序TsFile总大小，单位为byte |
+| file_count | name="wal" | AutoGauge | 写前日志文件个数 |
+| file_count | name="seq" | AutoGauge | 顺序TsFile文件个数 |
+| file_count | name="unseq" | AutoGauge | 乱序TsFile文件个数 |
+| file_count| name="inner-seq-file-num" |AutoGauge| 顺序空间内合并临时文件个数 |
+| file_count| name="inner-unseq-file-num" |AutoGauge| 乱序空间内合并临时文件个数 |
+| file_count| name="cross-file-num" |AutoGauge| 跨空间合并临时文件个数 |
+| file_count| name="inner-seq-file-size" |AutoGauge| 顺序空间内合并临时文件大小，单位为byte |
+| file_count| name="inner-unseq-file-size" |AutoGauge| 乱序空间内合并临时文件大小，单位为byte |
+| file_count| name="cross-file-size" |AutoGauge| 跨空间合并临时文件大小，单位为byte |
#### 4.2.10. IoTDB 进程统计
| Metric | Tags | Type | Description |
@@ -264,7 +272,6 @@ Core 级别的监控指标在系统运行中默认开启，每一个 Core 级别
| ----------------------- | --------------------------------------------- | --------- | ------------------ |
| jvm_compilation_time_ms | {compiler="HotSpot 64-Bit Tiered Compilers",} | AutoGauge | 耗费在编译上的时间 |
-
### 4.3. All 级别监控指标
目前还没有All级别的监控指标，后续会持续添加。
@@ -382,86 +389,86 @@ static_configs:
> 除特殊说明的监控项以外，以下监控项均保证在Important级别的监控框架中可用。
- `Overview`：系统概述
- - `Registered Node`：注册的ConfigNode/DataNode个数
- - `DataNode`：集群DataNode的存活状态，包括Online和Unknown两种。
- - `ConfigNode`：集群ConfigNode的存活状态，包括Online和Unknown两种。
- - `The Status Of Node`：集群具体节点运行状态，包括Online和Unkown两种。
+ - `Registered Node`：注册的ConfigNode/DataNode个数
+ - `DataNode`：集群DataNode的存活状态，包括Online和Unknown两种。
+ - `ConfigNode`：集群ConfigNode的存活状态，包括Online和Unknown两种。
+ - `The Status Of Node`：集群具体节点运行状态，包括Online和Unkown两种。
- `Region`：Region概述
- - `Region Number`：Region个数，包括总个数，DataRegion 个数和 SchemaRegion 个数。
- - `Leadership distribution`：集群 Leader 分布情况，指每个节点上对应 Region 的 Leader 的个数。
- - `Total Region in Node`：不同 Node 的 Region 总数量。
- - `Region in Node`：不同 Node 的 SchemaRegion/DataRegion 数量。
- - `Region in Database`（Normal级别）：不同 Database 的 Region 数量，包括 SchemaRegion、DataRegion。
- - `Slot in Database`(Normal级别)：不同 Database 的Slot数量，包括 DataSlot 数量和 SchemaSlot 数量。
-- `System`：系统
- - `CPU Core`：系统 CPU 核数情况。
- - `CPU Load`：系统 CPU 负载情况、进度 CPU 负载情况。
- - `CPU Time Per Minute`：进程平均每分钟占用系统 CPU 时间，注意：多核会导致该值超过1分钟。
- - `System Memory`：系统物理内存大小、系统使用物理内存大小、虚拟机提交的内存大小。
- - `System Swap Size`：系统交换区总大小、系统交换区使用大小。
- - `Process Memory`：IoTDB 进程最大内存总大小、IoTDB 进程总内存大小、IoTDB 进程使用内存大小。
- - `The Number of GC Per Minute`：平均每分钟 GC 次数。
- - `The Time Consumed Of GC Per Minute`：平均每分钟 GC 耗时。
- - `The Number Of Java Thread`：IoTDB 进程的不同状态的线程数。
- - `Heap Memory`：IoTDB 进程的堆内存
- - `Off Heap Memory`：IoTDB 进程的堆外内存
- - `Log Number Per Minute`：IoTDB 进程平均每分钟日志数
- - `The Time Consumed of Compliation Per Minute`：平均每分钟编译耗时
- - `The Number Of Class`：JVM 加载和卸载的类数量
-
+ - `Region Number`：Region个数，包括总个数，DataRegion 个数和 SchemaRegion 个数。
+ - `Leadership distribution`：集群 Leader 分布情况，指每个节点上对应 Region 的 Leader 的个数。
+ - `Total Region in Node`：不同 Node 的 Region 总数量。
+ - `Region in Node`：不同 Node 的 SchemaRegion/DataRegion 数量。
+ - `Region in Database`（Normal级别）：不同 Database 的 Region 数量，包括 SchemaRegion、DataRegion。
+ - `Slot in Database`(Normal级别)：不同 Database 的Slot数量，包括 DataSlot 数量和 SchemaSlot 数量。
+- `System`：系统
+ - `CPU Core`：系统 CPU 核数情况。
+ - `CPU Load`：系统 CPU 负载情况、进度 CPU 负载情况。
+ - `CPU Time Per Minute`：进程平均每分钟占用系统 CPU 时间，注意：多核会导致该值超过1分钟。
+ - `System Memory`：系统物理内存大小、系统使用物理内存大小、虚拟机提交的内存大小。
+ - `System Swap Size`：系统交换区总大小、系统交换区使用大小。
+ - `Process Memory`：IoTDB 进程最大内存总大小、IoTDB 进程总内存大小、IoTDB 进程使用内存大小。
+ - `The Number of GC Per Minute`：平均每分钟 GC 次数。
+ - `The Time Consumed Of GC Per Minute`：平均每分钟 GC 耗时。
+ - `The Number Of Java Thread`：IoTDB 进程的不同状态的线程数。
+ - `Heap Memory`：IoTDB 进程的堆内存
+ - `Off Heap Memory`：IoTDB 进程的堆外内存
+ - `Log Number Per Minute`：IoTDB 进程平均每分钟日志数
+ - `The Time Consumed of Compliation Per Minute`：平均每分钟编译耗时
+ - `The Number Of Class`：JVM 加载和卸载的类数量
+
##### 5.2.4.3. Apache IoTDB DataNode Dashboard 说明
> 除特殊说明的监控项以外，以下监控项均保证在Important级别的监控框架中可用。
- `Overview`：系统概述
- - `The Number Of Entity`：实体数量，包含时间序列等
- - `Write Point Per Minute`：每分钟系统平均写入点数
- - `Database Used Memory`：每个 Database 使用的内存大小
+ - `The Number Of Entity`：实体数量，包含时间序列等
+ - `Write Point Per Minute`：每分钟系统平均写入点数
+ - `Database Used Memory`：每个 Database 使用的内存大小
- `Interface`：接口
- - `The Time Consumed Of Operation(50%)`：不同客户端操作耗时的中位数
- - `The Time Consumed Of Operation(75%)`：不同客户端操作耗时的上四分位数
- - `The Time Consumed Of Operation(100%)`：不同客户端操作耗时的最大值
- - `The QPS of Interface`：系统接口每秒钟访问次数
- - `The Time Consumed Of Interface`：系统接口的平均耗时
- - `Cache Hit Rate`：缓存命中率
- - `Thrift Connection`：建立的 Thrift 连接个数
- - `Thrift Active Thread`：建立的活跃的 Thrift 连接的个数
+ - `The Time Consumed Of Operation(50%)`：不同客户端操作耗时的中位数
+ - `The Time Consumed Of Operation(75%)`：不同客户端操作耗时的上四分位数
+ - `The Time Consumed Of Operation(100%)`：不同客户端操作耗时的最大值
+ - `The QPS of Interface`：系统接口每秒钟访问次数
+ - `The Time Consumed Of Interface`：系统接口的平均耗时
+ - `Cache Hit Rate`：缓存命中率
+ - `Thrift Connection`：建立的 Thrift 连接个数
+ - `Thrift Active Thread`：建立的活跃的 Thrift 连接的个数
- `Engine`：引擎
- - `Task Number`：系统中不同状态的任务个数
- - `The Time Consumed Of Tasking`：系统中不同状态的任务的耗时
- - `Compaction Read And Write Per Minute`：平均每分钟合并读取和写入数据量
- - `Compaction R/W Ratio Per Minute`：平均每分钟合并读取和写入数据比
- - `Compaction Number Per Minute`：平均每分钟不同类型的合并任务数量
+ - `Task Number`：系统中不同状态的任务个数
+ - `The Time Consumed Of Tasking`：系统中不同状态的任务的耗时
+ - `Compaction Read And Write Per Minute`：平均每分钟合并读取和写入数据量
+ - `Compaction R/W Ratio Per Minute`：平均每分钟合并读取和写入数据比
+ - `Compaction Number Per Minute`：平均每分钟不同类型的合并任务数量
- `IoTConsensus`：IoT共识协议
- - `IoTConsensus Used Memory`：IoT共识层使用的内存大小
- - `IoTConsensus Sync Index`：不同的Region的写入Index和同步Index
- - `IoTConsensus Overview`：不同节点的同步总差距、总缓存的请求个数
- - `The time consumed of different stages(50%)`：不同阶段耗时的中位数
- - `The time consumed of different stages(75%)`：不同阶段耗时的上四分位数
- - `The time consumed of different stages(100%)`：不同阶段耗时的最大值
- - `IoTConsensus Search Index Rate`：不同region的写入Index的增长速度
- - `IoTConsensus Safe Index Rate`：不同region的同步Index的增长速度
- - `IoTConsensus LogDispatcher Request Size`：不同的LogDispatcherThread缓存的请求个数
- - `Sync Lag`：每个region的同步index差距
- - `Min Peer Sync Lag`：每个region的写入index和同步最快的LogDispatcherThread的同步index之间的差距
- - `Sync speed diff of Peers`：每个region中同步最快的LogDispatcherThread与同步最慢的LogDispatcherThread之间的同步index差距
-- `System`：系统
- - `CPU Core`：系统 CPU 核数情况。
- - `CPU Load`：系统 CPU 负载情况、进度 CPU 负载情况。
- - `CPU Time Per Minute`：进程平均每分钟占用系统 CPU 时间，注意：多核会导致该值超过1分钟。
- - `System Memory`：系统物理内存大小、系统使用物理内存大小、虚拟机提交的内存大小。
- - `System Swap Size`：系统交换区总大小、系统交换区使用大小。
- - `Process Memory`：IoTDB 进程最大内存总大小、IoTDB 进程总内存大小、IoTDB 进程使用内存大小。
- - `The Size Of File`：IoTDB系统相关的文件大小，包括wal下的文件总大小、seq下的tsfile文件总大小、unseq下的tsfile文件总大小
- - `The Number Of File`：IoTDB系统相关的文件个数，包括wal下的文件个数、seq下的tsfile文件个数、unseq下的tsfile文件个数
- - `The Space Of Disk`：当前data目录所挂载的磁盘总大小和剩余大小
- - `The Number of GC Per Minute`：平均每分钟 GC 次数。
- - `The Time Consumed Of GC Per Minute`：平均每分钟 GC 耗时。
- - `The Number Of Java Thread`：IoTDB 进程的不同状态的线程数。
- - `Heap Memory`：IoTDB 进程的堆内存
- - `Off Heap Memory`：IoTDB 进程的堆外内存
- - `Log Number Per Minute`：IoTDB 进程平均每分钟日志数
- - `The Time Consumed of Compliation Per Minute`：平均每分钟编译耗时
- - `The Number Of Class`：JVM 加载和卸载的类数量
+ - `IoTConsensus Used Memory`：IoT共识层使用的内存大小
+ - `IoTConsensus Sync Index`：不同的Region的写入Index和同步Index
+ - `IoTConsensus Overview`：不同节点的同步总差距、总缓存的请求个数
+ - `The time consumed of different stages(50%)`：不同阶段耗时的中位数
+ - `The time consumed of different stages(75%)`：不同阶段耗时的上四分位数
+ - `The time consumed of different stages(100%)`：不同阶段耗时的最大值
+ - `IoTConsensus Search Index Rate`：不同region的写入Index的增长速度
+ - `IoTConsensus Safe Index Rate`：不同region的同步Index的增长速度
+ - `IoTConsensus LogDispatcher Request Size`：不同的LogDispatcherThread缓存的请求个数
+ - `Sync Lag`：每个region的同步index差距
+ - `Min Peer Sync Lag`：每个region的写入index和同步最快的LogDispatcherThread的同步index之间的差距
+ - `Sync speed diff of Peers`：每个region中同步最快的LogDispatcherThread与同步最慢的LogDispatcherThread之间的同步index差距
+- `System`：系统
+ - `CPU Core`：系统 CPU 核数情况。
+ - `CPU Load`：系统 CPU 负载情况、进度 CPU 负载情况。
+ - `CPU Time Per Minute`：进程平均每分钟占用系统 CPU 时间，注意：多核会导致该值超过1分钟。
+ - `System Memory`：系统物理内存大小、系统使用物理内存大小、虚拟机提交的内存大小。
+ - `System Swap Size`：系统交换区总大小、系统交换区使用大小。
+ - `Process Memory`：IoTDB 进程最大内存总大小、IoTDB 进程总内存大小、IoTDB 进程使用内存大小。
+ - `The Size Of File`：IoTDB系统相关的文件大小，包括wal下的文件总大小、seq下的tsfile文件总大小、unseq下的tsfile文件总大小
+ - `The Number Of File`：IoTDB系统相关的文件个数，包括wal下的文件个数、seq下的tsfile文件个数、unseq下的tsfile文件个数
+ - `The Space Of Disk`：当前data目录所挂载的磁盘总大小和剩余大小
+ - `The Number of GC Per Minute`：平均每分钟 GC 次数。
+ - `The Time Consumed Of GC Per Minute`：平均每分钟 GC 耗时。
+ - `The Number Of Java Thread`：IoTDB 进程的不同状态的线程数。
+ - `Heap Memory`：IoTDB 进程的堆内存
+ - `Off Heap Memory`：IoTDB 进程的堆外内存
+ - `Log Number Per Minute`：IoTDB 进程平均每分钟日志数
+ - `The Time Consumed of Compliation Per Minute`：平均每分钟编译耗时
+ - `The Number Of Class`：JVM 加载和卸载的类数量
### 5.3. 使用 IoTDB 方式
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/TsFileMetricManager.java b/server/src/main/java/org/apache/iotdb/db/engine/TsFileMetricManager.java
index afc6018e43..e08a063e0b 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/TsFileMetricManager.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/TsFileMetricManager.java
@@ -27,10 +27,18 @@ import java.util.concurrent.atomic.AtomicLong;
/** This class collect the number and size of tsfile, and send it to the {@link FileMetrics} */
public class TsFileMetricManager {
private static final TsFileMetricManager INSTANCE = new TsFileMetricManager();
- private AtomicLong seqFileSize = new AtomicLong(0);
- private AtomicLong unseqFileSize = new AtomicLong(0);
- private AtomicInteger seqFileNum = new AtomicInteger(0);
- private AtomicInteger unseqFileNum = new AtomicInteger(0);
+ private final AtomicLong seqFileSize = new AtomicLong(0);
+ private final AtomicLong unseqFileSize = new AtomicLong(0);
+ private final AtomicInteger seqFileNum = new AtomicInteger(0);
+ private final AtomicInteger unseqFileNum = new AtomicInteger(0);
+
+ // compaction temporal files
+ private final AtomicLong innerSeqCompactionTempFileSize = new AtomicLong(0);
+ private final AtomicLong innerUnseqCompactionTempFileSize = new AtomicLong(0);
+ private final AtomicLong crossCompactionTempFileSize = new AtomicLong(0);
+ private final AtomicInteger innerSeqCompactionTempFileNum = new AtomicInteger(0);
+ private final AtomicInteger innerUnseqCompactionTempFileNum = new AtomicInteger(0);
+ private final AtomicInteger crossCompactionTempFileNum = new AtomicInteger(0);
private TsFileMetricManager() {}
@@ -65,4 +73,42 @@ public class TsFileMetricManager {
public long getFileNum(boolean seq) {
return seq ? seqFileNum.get() : unseqFileNum.get();
}
+
+ public void addCompactionTempFileSize(boolean innerSpace, boolean seq, long delta) {
+ if (innerSpace) {
+ long unused =
+ seq
+ ? innerSeqCompactionTempFileSize.addAndGet(delta)
+ : innerUnseqCompactionTempFileSize.addAndGet(delta);
+ } else {
+ crossCompactionTempFileSize.addAndGet(delta);
+ }
+ }
+
+ public void addCompactionTempFileNum(boolean innerSpace, boolean seq, int delta) {
+ if (innerSpace) {
+ long unused =
+ seq
+ ? innerSeqCompactionTempFileNum.addAndGet(delta)
+ : innerUnseqCompactionTempFileNum.addAndGet(delta);
+ } else {
+ crossCompactionTempFileNum.addAndGet(delta);
+ }
+ }
+
+ public long getCompactionTempFileSize(boolean innerSpace, boolean seq) {
+ if (innerSpace) {
+ return seq ? innerSeqCompactionTempFileSize.get() : innerUnseqCompactionTempFileSize.get();
+ } else {
+ return crossCompactionTempFileSize.get();
+ }
+ }
+
+ public int getCompactionTempFileNum(boolean innerSpace, boolean seq) {
+ if (innerSpace) {
+ return seq ? innerSeqCompactionTempFileNum.get() : innerUnseqCompactionTempFileNum.get();
+ } else {
+ return crossCompactionTempFileNum.get();
+ }
+ }
}
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/AlignedSeriesCompactionExecutor.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/AlignedSeriesCompactionExecutor.java
index 8d1a94a093..3ecc19d11e 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/AlignedSeriesCompactionExecutor.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/AlignedSeriesCompactionExecutor.java
@@ -19,6 +19,7 @@
package org.apache.iotdb.db.engine.compaction.inner.utils;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.TsFileMetricManager;
import org.apache.iotdb.db.engine.cache.ChunkCache;
import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
import org.apache.iotdb.db.engine.compaction.constant.CompactionType;
@@ -127,6 +128,7 @@ public class AlignedSeriesCompactionExecutor {
}
public void execute() throws IOException {
+ long originTempFileSize = writer.getPos();
while (readerAndChunkMetadataList.size() > 0) {
Pair<TsFileSequenceReader, List<AlignedChunkMetadata>> readerListPair =
readerAndChunkMetadataList.removeFirst();
@@ -152,6 +154,10 @@ public class AlignedSeriesCompactionExecutor {
chunkWriter.writeToFileWriter(writer);
}
writer.checkMetadataSizeAndMayFlush();
+
+ // update temporal file metrics
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(true, true, writer.getPos() - originTempFileSize);
}
private void compactOneAlignedChunk(AlignedChunkReader chunkReader) throws IOException {
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/SingleSeriesCompactionExecutor.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/SingleSeriesCompactionExecutor.java
index 740656bcaa..e8f7a1186a 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/SingleSeriesCompactionExecutor.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/SingleSeriesCompactionExecutor.java
@@ -111,6 +111,7 @@ public class SingleSeriesCompactionExecutor {
* series compaction may contain more than one chunk.
*/
public void execute() throws IOException {
+ long originTempFileSize = fileWriter.getPos();
while (readerAndChunkMetadataList.size() > 0) {
Pair<TsFileSequenceReader, List<ChunkMetadata>> readerListPair =
readerAndChunkMetadataList.removeFirst();
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/FastCompactionPerformer.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/FastCompactionPerformer.java
index 60650e6b0c..84f77b2a82 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/FastCompactionPerformer.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/FastCompactionPerformer.java
@@ -22,6 +22,7 @@ import org.apache.iotdb.commons.conf.IoTDBConstant;
import org.apache.iotdb.commons.exception.IllegalPathException;
import org.apache.iotdb.commons.exception.MetadataException;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.TsFileMetricManager;
import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
import org.apache.iotdb.db.engine.compaction.CompactionUtils;
import org.apache.iotdb.db.engine.compaction.cross.rewrite.task.FastCompactionPerformerSubTask;
@@ -84,6 +85,8 @@ public class FastCompactionPerformer
private boolean isCrossCompaction;
+ private long tempFileSize = 0L;
+
public FastCompactionPerformer(
List<TsFileResource> seqFiles,
List<TsFileResource> unseqFiles,
@@ -106,6 +109,8 @@ public class FastCompactionPerformer
@Override
public void perform()
throws IOException, MetadataException, StorageEngineException, InterruptedException {
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileNum(!isCrossCompaction, seqFiles.size() > 0, targetFiles.size());
try (MultiTsFileDeviceIterator deviceIterator =
new MultiTsFileDeviceIterator(seqFiles, unseqFiles, readerCacheMap);
AbstractCompactionWriter compactionWriter =
@@ -117,7 +122,6 @@ public class FastCompactionPerformer
Pair<String, Boolean> deviceInfo = deviceIterator.nextDevice();
String device = deviceInfo.left;
boolean isAligned = deviceInfo.right;
-
// sort the resources by the start time of current device from old to new, and remove
// resource that does not contain the current device. Notice: when the level of time index
// is file, there will be a false positive judgment problem, that is, the device does not
@@ -138,6 +142,12 @@ public class FastCompactionPerformer
compactionWriter.endChunkGroup();
// check whether to flush chunk metadata or not
compactionWriter.checkAndMayFlushChunkMetadata();
+ // Add temp file metrics
+ long currentTempFileSize = compactionWriter.getWriterSize();
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(
+ !isCrossCompaction, seqFiles.size() > 0, currentTempFileSize - tempFileSize);
+ tempFileSize = currentTempFileSize;
sortedSourceFiles.clear();
}
compactionWriter.endFile();
@@ -150,6 +160,10 @@ public class FastCompactionPerformer
sortedSourceFiles = null;
readerCacheMap = null;
modificationCache = null;
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileNum(!isCrossCompaction, seqFiles.size() > 0, -targetFiles.size());
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(!isCrossCompaction, seqFiles.size() > 0, -tempFileSize);
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadChunkCompactionPerformer.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadChunkCompactionPerformer.java
index 389cb7f87c..443db403d4 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadChunkCompactionPerformer.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadChunkCompactionPerformer.java
@@ -22,6 +22,7 @@ import org.apache.iotdb.commons.conf.IoTDBConstant;
import org.apache.iotdb.commons.exception.MetadataException;
import org.apache.iotdb.commons.path.PartialPath;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.TsFileMetricManager;
import org.apache.iotdb.db.engine.compaction.inner.utils.AlignedSeriesCompactionExecutor;
import org.apache.iotdb.db.engine.compaction.inner.utils.MultiTsFileDeviceIterator;
import org.apache.iotdb.db.engine.compaction.inner.utils.SingleSeriesCompactionExecutor;
@@ -49,6 +50,7 @@ public class ReadChunkCompactionPerformer implements ISeqCompactionPerformer {
private TsFileResource targetResource;
private List<TsFileResource> seqFiles;
private CompactionTaskSummary summary;
+ private long tempFileSize = 0L;
public ReadChunkCompactionPerformer(List<TsFileResource> sourceFiles, TsFileResource targetFile) {
this.seqFiles = sourceFiles;
@@ -70,6 +72,7 @@ public class ReadChunkCompactionPerformer implements ISeqCompactionPerformer {
(SystemInfo.getInstance().getMemorySizeForCompaction()
/ IoTDBDescriptor.getInstance().getConfig().getCompactionThreadCount()
* IoTDBDescriptor.getInstance().getConfig().getChunkMetadataSizeProportion());
+ TsFileMetricManager.getInstance().addCompactionTempFileNum(true, true, 1);
try (MultiTsFileDeviceIterator deviceIterator = new MultiTsFileDeviceIterator(seqFiles);
TsFileIOWriter writer =
new TsFileIOWriter(targetResource.getTsFile(), true, sizeForFileWriter)) {
@@ -83,6 +86,11 @@ public class ReadChunkCompactionPerformer implements ISeqCompactionPerformer {
} else {
compactNotAlignedSeries(device, targetResource, writer, deviceIterator);
}
+ // update temporal file metrics
+ long newTempFileSize = writer.getPos();
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(true, true, newTempFileSize - tempFileSize);
+ tempFileSize = newTempFileSize;
}
for (TsFileResource tsFileResource : seqFiles) {
@@ -90,6 +98,9 @@ public class ReadChunkCompactionPerformer implements ISeqCompactionPerformer {
}
writer.endFile();
targetResource.close();
+ } finally {
+ TsFileMetricManager.getInstance().addCompactionTempFileSize(true, true, -tempFileSize);
+ TsFileMetricManager.getInstance().addCompactionTempFileNum(true, true, -1);
}
}
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadPointCompactionPerformer.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadPointCompactionPerformer.java
index 34259131d1..3e7e2e9516 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadPointCompactionPerformer.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/performer/impl/ReadPointCompactionPerformer.java
@@ -25,6 +25,7 @@ import org.apache.iotdb.commons.path.AlignedPath;
import org.apache.iotdb.commons.path.MeasurementPath;
import org.apache.iotdb.commons.path.PartialPath;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.TsFileMetricManager;
import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
import org.apache.iotdb.db.engine.compaction.CompactionUtils;
import org.apache.iotdb.db.engine.compaction.cross.rewrite.task.ReadPointPerformerSubTask;
@@ -75,6 +76,7 @@ public class ReadPointCompactionPerformer
private CompactionTaskSummary summary;
private List<TsFileResource> targetFiles = Collections.emptyList();
+ private long tempFileSize = 0L;
public ReadPointCompactionPerformer(
List<TsFileResource> seqFiles,
@@ -102,6 +104,8 @@ public class ReadPointCompactionPerformer
QueryResourceManager.getInstance()
.getQueryFileManager()
.addUsedFilesForQuery(queryId, queryDataSource);
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileNum(seqFiles.size() == 0, false, targetFiles.size());
try (AbstractCompactionWriter compactionWriter =
getCompactionWriter(seqFiles, unseqFiles, targetFiles)) {
// Do not close device iterator, because tsfile reader is managed by FileReaderManager.
@@ -128,6 +132,10 @@ public class ReadPointCompactionPerformer
} finally {
QueryResourceManager.getInstance().endQuery(queryId);
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileNum(seqFiles.size() == 0, false, -targetFiles.size());
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(seqFiles.size() == 0, false, tempFileSize);
}
}
@@ -177,6 +185,11 @@ public class ReadPointCompactionPerformer
// check whether to flush chunk metadata or not
compactionWriter.checkAndMayFlushChunkMetadata();
}
+ // add temp file metrics
+ long currentWriterSize = compactionWriter.getWriterSize();
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(seqFiles.size() == 0, false, currentWriterSize - tempFileSize);
+ tempFileSize = currentWriterSize;
}
private void compactNonAlignedSeries(
@@ -224,6 +237,12 @@ public class ReadPointCompactionPerformer
// check whether to flush chunk metadata or not
compactionWriter.checkAndMayFlushChunkMetadata();
}
+
+ // add temp file metrics
+ long currentWriterSize = compactionWriter.getWriterSize();
+ TsFileMetricManager.getInstance()
+ .addCompactionTempFileSize(seqFiles.size() == 0, false, currentWriterSize - tempFileSize);
+ tempFileSize = currentWriterSize;
}
/**
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java
index c733409a8c..e95a386b72 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java
@@ -123,6 +123,8 @@ public abstract class AbstractCompactionWriter implements AutoCloseable {
public abstract void endFile() throws IOException;
+ public abstract long getWriterSize() throws IOException;
+
/**
* Update startTime and endTime of the current device in each target resources, and check whether
* to flush chunk metadatas or not.
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCrossCompactionWriter.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCrossCompactionWriter.java
index c00ca5d231..f1ef4433e1 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCrossCompactionWriter.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCrossCompactionWriter.java
@@ -230,4 +230,13 @@ public abstract class AbstractCrossCompactionWriter extends AbstractCompactionWr
fileIndex++;
}
}
+
+ @Override
+ public long getWriterSize() throws IOException {
+ long totalSize = 0;
+ for (TsFileIOWriter writer : targetFileWriters) {
+ totalSize += writer.getPos();
+ }
+ return totalSize;
+ }
}
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractInnerCompactionWriter.java b/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractInnerCompactionWriter.java
index 1d77d35ad8..5762c8f845 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractInnerCompactionWriter.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractInnerCompactionWriter.java
@@ -105,4 +105,9 @@ public abstract class AbstractInnerCompactionWriter extends AbstractCompactionWr
public void checkAndMayFlushChunkMetadata() throws IOException {
fileWriter.checkMetadataSizeAndMayFlush();
}
+
+ @Override
+ public long getWriterSize() throws IOException {
+ return fileWriter.getPos();
+ }
}
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java b/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java
index 964c423f07..1f39ed6ccb 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/DataRegion.java
@@ -2125,6 +2125,8 @@ public class DataRegion implements IDataRegionForQuery {
synchronized (closeStorageGroupCondition) {
closeStorageGroupCondition.notifyAll();
}
+ TsFileMetricManager.getInstance()
+ .addFile(tsFileProcessor.getTsFileResource().getTsFileSize(), tsFileProcessor.isSequence());
logger.info("signal closing database condition in {}", databaseName + "-" + dataRegionId);
}
diff --git a/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java b/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
index 04ec438712..76755677cc 100644
--- a/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
+++ b/server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessor.java
@@ -28,7 +28,6 @@ import org.apache.iotdb.db.conf.IoTDBConfig;
import org.apache.iotdb.db.conf.IoTDBDescriptor;
import org.apache.iotdb.db.conf.adapter.CompressionRatio;
import org.apache.iotdb.db.engine.StorageEngine;
-import org.apache.iotdb.db.engine.TsFileMetricManager;
import org.apache.iotdb.db.engine.flush.CloseFileListener;
import org.apache.iotdb.db.engine.flush.FlushListener;
import org.apache.iotdb.db.engine.flush.FlushManager;
@@ -1270,7 +1269,6 @@ public class TsFileProcessor {
writer.getFile().length(),
closeEndTime - closeStartTime);
}
- TsFileMetricManager.getInstance().addFile(tsFileResource.getTsFile().length(), sequence);
writer = null;
}
diff --git a/server/src/main/java/org/apache/iotdb/db/service/metrics/FileMetrics.java b/server/src/main/java/org/apache/iotdb/db/service/metrics/FileMetrics.java
index 765e00452f..47aebe608a 100644
--- a/server/src/main/java/org/apache/iotdb/db/service/metrics/FileMetrics.java
+++ b/server/src/main/java/org/apache/iotdb/db/service/metrics/FileMetrics.java
@@ -49,6 +49,13 @@ public class FileMetrics implements IMetricSet {
private long unsequenceFileTotalSize = 0L;
private long unsequenceFileTotalCount = 0L;
+ private long innerSeqCompactionTempFileSize = 0L;
+ private long innerUnseqCompactionTempFileSize = 0L;
+ private long crossCompactionTempFileSize = 0L;
+ private long innerSeqCompactionTempFileNum = 0L;
+ private long innerUnseqCompactionTempFileNum = 0L;
+ private long crossCompactionTempFileNum = 0L;
+
@Override
public void bindTo(AbstractMetricService metricService) {
metricService.createAutoGauge(
@@ -93,6 +100,48 @@ public class FileMetrics implements IMetricSet {
FileMetrics::getUnsequenceFileTotalCount,
Tag.NAME.toString(),
"unseq");
+ metricService.createAutoGauge(
+ Metric.FILE_COUNT.toString(),
+ MetricLevel.IMPORTANT,
+ this,
+ FileMetrics::getInnerSeqCompactionTempFileNum,
+ Tag.NAME.toString(),
+ "inner-seq-temp-num");
+ metricService.createAutoGauge(
+ Metric.FILE_COUNT.toString(),
+ MetricLevel.IMPORTANT,
+ this,
+ FileMetrics::getInnerUnseqCompactionTempFileNum,
+ Tag.NAME.toString(),
+ "inner-unseq-temp-num");
+ metricService.createAutoGauge(
+ Metric.FILE_COUNT.toString(),
+ MetricLevel.IMPORTANT,
+ this,
+ FileMetrics::getCrossCompactionTempFileNum,
+ Tag.NAME.toString(),
+ "cross-temp-num");
+ metricService.createAutoGauge(
+ Metric.FILE_SIZE.toString(),
+ MetricLevel.IMPORTANT,
+ this,
+ FileMetrics::getInnerSeqCompactionTempFileSize,
+ Tag.NAME.toString(),
+ "inner-seq-temp-size");
+ metricService.createAutoGauge(
+ Metric.FILE_SIZE.toString(),
+ MetricLevel.IMPORTANT,
+ this,
+ FileMetrics::getInnerUnseqCompactionTempFileSize,
+ Tag.NAME.toString(),
+ "inner-unseq-temp-size");
+ metricService.createAutoGauge(
+ Metric.FILE_SIZE.toString(),
+ MetricLevel.IMPORTANT,
+ this,
+ FileMetrics::getCrossCompactionTempFileSize,
+ Tag.NAME.toString(),
+ "cross-temp-size");
// finally start to update the value of some metrics in async way
if (null == currentServiceFuture) {
@@ -128,6 +177,30 @@ public class FileMetrics implements IMetricSet {
MetricType.AUTO_GAUGE, Metric.FILE_COUNT.toString(), Tag.NAME.toString(), "seq");
metricService.remove(
MetricType.AUTO_GAUGE, Metric.FILE_COUNT.toString(), Tag.NAME.toString(), "unseq");
+ metricService.remove(
+ MetricType.AUTO_GAUGE,
+ Metric.FILE_COUNT.toString(),
+ Tag.NAME.toString(),
+ "inner-seq-temp-num");
+ metricService.remove(
+ MetricType.AUTO_GAUGE,
+ Metric.FILE_COUNT.toString(),
+ Tag.NAME.toString(),
+ "inner-unseq-temp-num");
+ metricService.remove(
+ MetricType.AUTO_GAUGE, Metric.FILE_COUNT.toString(), Tag.NAME.toString(), "cross-temp-num");
+ metricService.remove(
+ MetricType.AUTO_GAUGE,
+ Metric.FILE_SIZE.toString(),
+ Tag.NAME.toString(),
+ "inner-seq-temp-size");
+ metricService.remove(
+ MetricType.AUTO_GAUGE,
+ Metric.FILE_SIZE.toString(),
+ Tag.NAME.toString(),
+ "inner-unseq-temp-size");
+ metricService.remove(
+ MetricType.AUTO_GAUGE, Metric.FILE_SIZE.toString(), Tag.NAME.toString(), "cross-temp-size");
}
private void collect() {
@@ -162,4 +235,28 @@ public class FileMetrics implements IMetricSet {
public long getUnsequenceFileTotalCount() {
return unsequenceFileTotalCount;
}
+
+ public long getInnerSeqCompactionTempFileSize() {
+ return innerSeqCompactionTempFileSize;
+ }
+
+ public long getInnerUnseqCompactionTempFileSize() {
+ return innerUnseqCompactionTempFileSize;
+ }
+
+ public long getCrossCompactionTempFileSize() {
+ return crossCompactionTempFileSize;
+ }
+
+ public long getInnerSeqCompactionTempFileNum() {
+ return innerSeqCompactionTempFileNum;
+ }
+
+ public long getInnerUnseqCompactionTempFileNum() {
+ return innerUnseqCompactionTempFileNum;
+ }
+
+ public long getCrossCompactionTempFileNum() {
+ return crossCompactionTempFileNum;
+ }
}