You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by ji...@apache.org on 2019/09/03 01:10:55 UTC
[incubator-iotdb] branch log_tool updated: Rephrase document Change
group logic of visualizer Add sum to statistics Fix that statistic tab
cannot be removed Add a new plan
This is an automated email from the ASF dual-hosted git repository.
jiangtian pushed a commit to branch log_tool
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
The following commit(s) were added to refs/heads/log_tool by this push:
new 3ed3e5b Rephrase document Change group logic of visualizer Add sum to statistics Fix that statistic tab cannot be removed Add a new plan
3ed3e5b is described below
commit 3ed3e5bb9cfc787c28a4719bf94a3d1b3881089a
Author: jt <jt...@163.com>
AuthorDate: Tue Sep 3 09:01:06 2019 +0800
Rephrase document
Change group logic of visualizer
Add sum to statistics
Fix that statistic tab cannot be removed
Add a new plan
---
.../UserGuide/8-System Tools/3-Log Visualizer.md | 36 +++++++-------
.../tools/logVisualize/plans/mergeRate.plan | 10 ++++
.../iotdb/db/tools/logvisual/LogVisualizer.java | 55 +++++++++++-----------
.../db/tools/logvisual/TimeSeriesStatistics.java | 8 +++-
.../tools/logvisual/gui/LogVisualizationGui.java | 2 +-
5 files changed, 63 insertions(+), 48 deletions(-)
diff --git a/docs/Documentation-CHN/UserGuide/8-System Tools/3-Log Visualizer.md b/docs/Documentation-CHN/UserGuide/8-System Tools/3-Log Visualizer.md
index 185e001..d405aab 100644
--- a/docs/Documentation-CHN/UserGuide/8-System Tools/3-Log Visualizer.md
+++ b/docs/Documentation-CHN/UserGuide/8-System Tools/3-Log Visualizer.md
@@ -62,68 +62,68 @@
# 使用方法
## 准备工作
-在使用LogVisualizer之前，你需要对其进行构建。LogVisualizer被集成到server模块作为一个系统工具，因此你
-可以通过构建server模块来同时构建LogVisualizer。你可以使用以下命令：
+在使用LogVisualizer之前，您需要对其进行构建。LogVisualizer被集成到server模块作为一个系统工具，因此您
+可以通过构建server模块来同时构建LogVisualizer。您可以使用以下命令：
> mvn clean package -pl server -DskipTests
-之后你能在这个目录下找到LogVisualizer的启动脚本：
+之后您能在这个目录下找到LogVisualizer的启动脚本：
`server/target/iotdb-server-{project-version}-SNAPSHOT/tools/logVisualize`
## 启动图形界面
-你可以通过脚本`log-visualizer.sh` (在Windows下为`log-visualizer.bat`)来启动LogVisualizer. 这将会启动
+您可以通过脚本`log-visualizer.sh` (在Windows下为`log-visualizer.bat`)来启动LogVisualizer. 这将会启动
一个如下图所示的图形界面:
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/main_panel.png?raw=true">
## 选择日志文件
-首先，你需要点击图示的按钮选择一个含有待可视化日志的文件。注意：我们目前仅支持单行日志，多行日志无法
+首先，您需要点击图示的按钮选择一个含有待可视化日志的文件。注意：我们目前仅支持单行日志，多行日志无法
被正确解析而会被忽略。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/select_log_file.png?raw=true">
## 选择解析器配置文件
-然后，你还需要选择一个解析器配置文件，该文件将告诉LogVisualizer如何解析日志。对于IoTDB产生的日志，我们在
-`tools\default.log.pattern`里提供了一个例子。如果你需要解析的日志与IoTDB的日志具有相同的格式，你可以
-直接使用该文件。否则，你需要仔细阅读该样例文件，并根据你的日志结构进行相应的修改。
+然后，您还需要选择一个解析器配置文件，该文件将告诉LogVisualizer如何解析日志。对于IoTDB产生的日志，我们在
+`tools\default.log.pattern`里提供了一个例子。如果您需要解析的日志与IoTDB的日志具有相同的格式，您可以
+直接使用该文件。否则，您需要仔细阅读该样例文件，并根据您的日志结构进行相应的修改。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/select_log_parser_property.png?raw=true">
## 载入日志
之后，点击`Load logs`按钮来让LogVisualizer准备对这些日志进行解析。如果准备成功，按钮上方将显示
-`Logs are successfully loaded`。否则将会弹出一个对话框提示错误，你需要根据错误内容进行相应处理。
+`Logs are successfully loaded`。否则将会弹出一个对话框提示错误，您需要根据错误内容进行相应处理。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/load_logs.png?raw=true">
## 加载可视化计划
-接下来，你可以选择加载磁盘上已有的可视化计划或者新建计划。要加载已有的计划，只需要点击 `Load plans`
+接下来，您可以选择加载磁盘上已有的可视化计划或者新建计划。要加载已有的计划，只需要点击 `Load plans`
按钮，并在弹出的对话框中选择想要加载的计划文件或者包含计划文件的文件夹。注意，如果选择文件夹，该文件
夹必须只包含计划文件。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/load_plan.png?raw=true">
## 新建可视化计划
-现在，我们将展示如何使用图形界面创建一个新的可视化计划，你也可以参考已有的计划文件，使用文本编辑器编写。
+现在，我们将展示如何使用图形界面创建一个新的可视化计划，您也可以参考已有的计划文件，使用文本编辑器编写。
首先点击`Create plan`按钮并选择存放新计划的文件，新计划将出现在右边的列表。选择的文件名将作为新计划的
名称，该名称不能与已有的计划重复。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/create_plan.png?raw=true">
## 编辑可视化计划
-在创建了一个新的计划以后，你可以在右边的面板中对其进行编辑。我们在 `tools/logVisualize/plans/flushTimeConsumption.plan`
-提供了一个样例来帮助你理解计划的每一个字段的含义以及应该如何设置它们，我们强烈推荐你仔细阅读该样例。
-你也可以编辑其他已经加载了的计划。编辑结束后请务必点击`Save plan`按钮来保存您的修改。在下图的例子中，
+在创建了一个新的计划以后，您可以在右边的面板中对其进行编辑。我们在 `tools/logVisualize/plans/flushTimeConsumption.plan`
+提供了一个样例来帮助您理解计划的每一个字段的含义以及应该如何设置它们，我们强烈推荐您仔细阅读该样例。
+您也可以编辑其他已经加载了的计划。编辑结束后请务必点击`Save plan`按钮来保存您的修改。在下图的例子中，
我们给出了如何设计一个计划让它能对任务的完成时间进行可视化，并将任务按照其类别进行分组。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/edit_plan.png?raw=true">
## 删除可视化计划
-如果你不再想使用某个计划，你可以选中它并点击`Delete plan`按钮。这将永久地把该计划从磁盘上移除。
+如果您不再想使用某个计划，您可以选中它并点击`Delete plan`按钮。这将永久地把该计划从磁盘上移除。
## 执行可视化计划
-最后，选中一个计划并点击`Execute plan`就可以执行该计划。请确保你之前已经加载了日志。计划执行以后，
+最后，选中一个计划并点击`Execute plan`就可以执行该计划。请确保您之前已经加载了日志。计划执行以后，
主界面上将会多出两个页面，第一个页面包含了若干个时间序列图，每一个对应了计划里的一个分组并且包含了
-计划中的所有测点（measurement），第二个页面包含了对每一组的日志的每个测点的统计信息。当你结束对一个
-页面的浏览以后，你可以点击页面右上角的按钮将其关闭。
+计划中的所有测点（measurement），第二个页面包含了对每一组的日志的每个测点的统计信息。当您结束对一个
+页面的浏览以后，您可以点击页面右上角的按钮将其关闭。
<img style="width:100%; max-width:800px; max-height:600px; margin-left:auto; margin-right:auto; display:block;" src="https://github.com/jt2594838/PicHub/blob/master/log-visualizer/execute_plan.png?raw=true">
diff --git a/server/src/assembly/resources/tools/logVisualize/plans/mergeRate.plan b/server/src/assembly/resources/tools/logVisualize/plans/mergeRate.plan
new file mode 100644
index 0000000..a9ed2dd
--- /dev/null
+++ b/server/src/assembly/resources/tools/logVisualize/plans/mergeRate.plan
@@ -0,0 +1,10 @@
+#
+#Mon Sep 02 16:40:14 CST 2019
+start_date=2019 08 29 21 30
+legends=Time,byteRate(MB/s),seriesRate,fileRate,ptRate
+name=mergeRate
+min_level=DEBUG
+date_pattern=yyyy MM dd HH mm
+content_pattern=.* ends after (.*)s, byteRate\: (.*)MB/s, seriesRate (.*)/s, chunkRate\: 0.0/s, fileRate\: (.*)/s, ptRate\: (.*)/s
+end_date=2019 08 29 23 59
+measurement_positions=1,2,3,4,5
diff --git a/server/src/main/java/org/apache/iotdb/db/tools/logvisual/LogVisualizer.java b/server/src/main/java/org/apache/iotdb/db/tools/logvisual/LogVisualizer.java
index f94ac5e..77ff498 100644
--- a/server/src/main/java/org/apache/iotdb/db/tools/logvisual/LogVisualizer.java
+++ b/server/src/main/java/org/apache/iotdb/db/tools/logvisual/LogVisualizer.java
@@ -151,7 +151,8 @@ public class LogVisualizer {
}
collectLogs(plan);
groupLogs();
- Map<String,TimeSeriesCollection> taggedTimeSeries = createTimeSeries(plan);
+ // <measurementName, <tag, timeseries>>
+ Map<String,Map<String, TimeSeries>> taggedTimeSeries = createTimeSeries(plan);
charts = drawCharts(taggedTimeSeries, plan);
statisticsMap = genStatisticMap(taggedTimeSeries);
clearLogGroups();
@@ -198,15 +199,16 @@ public class LogVisualizer {
logger.info("Found {} different tags", logGroups.size());
}
- private Map<String, TimeSeriesCollection> createTimeSeries(VisualizationPlan plan) {
- Map<String, TimeSeriesCollection> ret = new HashMap<>();
+ // <measurementName, <tag, timeseries>>
+ private Map<String, Map<String, TimeSeries>> createTimeSeries(VisualizationPlan plan) {
+ Map<String, Map<String, TimeSeries>> ret = new HashMap<>();
for (Entry<List<String>, List<LogEntry>> entry : logGroups.entrySet()) {
List<String> tags = entry.getKey();
List<LogEntry> logs = entry.getValue();
// create a tag string for the log group as the key of the timeseries
String concatenatedTag;
if (tags.isEmpty()) {
- concatenatedTag = plan.getName();
+ concatenatedTag = plan.getName() + " ";
} else {
StringBuilder builder = new StringBuilder(plan.getName() + "-" + tags.get(0));
for (int i = 1; i < tags.size(); i++) {
@@ -216,44 +218,44 @@ public class LogVisualizer {
concatenatedTag = builder.toString();
}
- TimeSeriesCollection timeseriesGroup = new TimeSeriesCollection();
if (plan.getMeasurementPositions() != null) {
// the measurements are given, create a timeseries for each measurement
String[] legends = plan.getLegends();
- for (String legend : legends) {
- // name each timeseries with the legend of each measurement
- TimeSeries timeSeries = new TimeSeries(concatenatedTag + legend);
- timeseriesGroup.addSeries(timeSeries);
- }
// use the values in each log to build the timeseries
for (LogEntry logEntry : logs) {
List<Double> values = logEntry.getMeasurements();
for (int i = 0; i < values.size(); i++) {
- timeseriesGroup.getSeries(i).addOrUpdate(new Millisecond(logEntry.getDate()), values.get(i));
+ String legend = legends[i];
+ TimeSeries timeSeries = ret.computeIfAbsent(legend, leg -> new HashMap<>())
+ .computeIfAbsent(concatenatedTag, tag -> new TimeSeries(tag + legend));
+ timeSeries.addOrUpdate(new Millisecond(logEntry.getDate()), values.get(i));
}
}
} else {
// the measurement are not given, just record the time when each log happened
- TimeSeries happenedInstance = new TimeSeries(concatenatedTag + "HappenedInstance");
+ String legend = "TimeOfOccurrence";
+ TimeSeries happenedInstance = ret.computeIfAbsent(legend, tag -> new HashMap<>())
+ .computeIfAbsent(concatenatedTag, tag -> new TimeSeries( tag + legend));
for (LogEntry logEntry : logs) {
happenedInstance.addOrUpdate(new Millisecond(logEntry.getDate()), 1.0);
}
- timeseriesGroup.addSeries(happenedInstance);
}
- ret.put(concatenatedTag, timeseriesGroup);
}
return ret;
}
- private Map<String, JFreeChart> drawCharts(Map<String, TimeSeriesCollection> taggedTimeSeries,
+ private Map<String, JFreeChart> drawCharts(Map<String, Map<String, TimeSeries>> taggedTimeSeries,
VisualizationPlan plan) {
Map<String, JFreeChart> charts = new HashMap<>();
- for (Entry<String, TimeSeriesCollection> entry : taggedTimeSeries.entrySet()) {
- String tag = entry.getKey();
- TimeSeriesCollection timeSeriesList = entry.getValue();
+ for (Entry<String, Map<String, TimeSeries>> entry : taggedTimeSeries.entrySet()) {
+ String measurementName = entry.getKey();
+ TimeSeriesCollection timeSeriesList = new TimeSeriesCollection();
+ for (TimeSeries timeSeries : entry.getValue().values()) {
+ timeSeriesList.addSeries(timeSeries);
+ }
// contain the start time of the timeseries in the x-axis name
Date startDate = new Date((long) timeSeriesList.getDomainBounds(true).getLowerBound());
- JFreeChart chart = ChartFactory.createTimeSeriesChart(tag, "time-"+ startDate, "value",
+ JFreeChart chart = ChartFactory.createTimeSeriesChart(measurementName, "time-"+ startDate, measurementName,
timeSeriesList);
XYPlot xyPlot = chart.getXYPlot();
XYLineAndShapeRenderer xyLineAndShapeRenderer = ((XYLineAndShapeRenderer) xyPlot
@@ -265,7 +267,7 @@ public class LogVisualizer {
// do not draw lines if we only record the time instances of the logs
xyLineAndShapeRenderer.setDefaultLinesVisible(false);
}
- charts.put(tag, chart);
+ charts.put(measurementName, chart);
}
return charts;
}
@@ -282,19 +284,18 @@ public class LogVisualizer {
this.logFile = logFile;
}
- private Map<String, List<TimeSeriesStatistics>> genStatisticMap(Map<String,TimeSeriesCollection>
+ private Map<String, List<TimeSeriesStatistics>> genStatisticMap(Map<String, Map<String, TimeSeries>>
taggedTimeSeries) {
Map<String, List<TimeSeriesStatistics>> ret = new HashMap<>();
- for (Entry<String, TimeSeriesCollection> timeSeriesCollectionEntry : taggedTimeSeries.entrySet()) {
+ for (Entry<String, Map<String, TimeSeries>> timeSeriesEntry : taggedTimeSeries.entrySet()) {
// calculate the statistics of the logs in each group
- String tag = timeSeriesCollectionEntry.getKey();
- TimeSeriesCollection timeSeriesCollection = timeSeriesCollectionEntry.getValue();
+ String measurementName = timeSeriesEntry.getKey();
List<TimeSeriesStatistics> seriesStatistics = new ArrayList<>();
- for (int i = 0; i < timeSeriesCollection.getSeriesCount(); i++) {
- TimeSeries timeSeries = timeSeriesCollection.getSeries(i);
+ Map<String, TimeSeries> seriesMap = timeSeriesEntry.getValue();
+ for (TimeSeries timeSeries : seriesMap.values()) {
seriesStatistics.add(new TimeSeriesStatistics(timeSeries));
}
- ret.put(tag, seriesStatistics);
+ ret.put(measurementName, seriesStatistics);
}
return ret;
}
diff --git a/server/src/main/java/org/apache/iotdb/db/tools/logvisual/TimeSeriesStatistics.java b/server/src/main/java/org/apache/iotdb/db/tools/logvisual/TimeSeriesStatistics.java
index 71d36b5..eb993af 100644
--- a/server/src/main/java/org/apache/iotdb/db/tools/logvisual/TimeSeriesStatistics.java
+++ b/server/src/main/java/org/apache/iotdb/db/tools/logvisual/TimeSeriesStatistics.java
@@ -20,6 +20,7 @@
package org.apache.iotdb.db.tools.logvisual;
import java.util.Date;
+import org.apache.iotdb.tsfile.utils.StringContainer;
import org.jfree.data.time.TimeSeries;
import org.jfree.data.time.TimeSeriesDataItem;
@@ -29,7 +30,7 @@ import org.jfree.data.time.TimeSeriesDataItem;
public class TimeSeriesStatistics {
public static final String[] HEADER = new String[] {
- "name", "count", "meanInterval", "maxInterval", "minInterval", "meanVal", "maxVal", "minVal"
+ "name", "count", "meanInterval", "maxInterval", "minInterval", "meanVal", "maxVal", "minVal", "valSum"
};
private String name;
@@ -40,6 +41,7 @@ public class TimeSeriesStatistics {
private double meanVal = 0.0;
private double maxVal = Double.MIN_VALUE;
private double minVal = Double.MAX_VALUE;
+ private double valSum = 0.0;
TimeSeriesStatistics(TimeSeries timeSeries) {
Date lastDate = null;
@@ -60,6 +62,7 @@ public class TimeSeriesStatistics {
meanVal = (meanVal * size + value) / (size + 1);
maxVal = maxVal < value ? value : maxVal;
minVal = minVal < value ? minVal : value;
+ valSum += value;
size ++;
}
}
@@ -82,7 +85,8 @@ public class TimeSeriesStatistics {
ret[i++] = minInterval;
ret[i++] = meanVal;
ret[i++] = maxVal;
- ret[i] = minVal;
+ ret[i++] = minVal;
+ ret[i] = valSum;
return ret;
}
}
\ No newline at end of file
diff --git a/server/src/main/java/org/apache/iotdb/db/tools/logvisual/gui/LogVisualizationGui.java b/server/src/main/java/org/apache/iotdb/db/tools/logvisual/gui/LogVisualizationGui.java
index ba6b599..5df3fbd 100644
--- a/server/src/main/java/org/apache/iotdb/db/tools/logvisual/gui/LogVisualizationGui.java
+++ b/server/src/main/java/org/apache/iotdb/db/tools/logvisual/gui/LogVisualizationGui.java
@@ -119,7 +119,7 @@ public class LogVisualizationGui {
this::onTabClose);
ResultStatisticTab oldTableTab = resultTablePanels.get(tabName);
if (oldTableTab != null) {
- tabbedPane.remove(oldPlotTab);
+ tabbedPane.remove(oldTableTab);
}
resultTablePanels.put(tabName, resultStatisticTab);
tabbedPane.add(resultStatisticTab);