You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iotdb.apache.org by qi...@apache.org on 2020/07/22 14:01:15 UTC
[incubator-iotdb] branch add_file_list created (now a345f00)
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a change to branch add_file_list
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git.
at a345f00 add file list doc
This branch includes the following new commits:
new 9beddd9 Merge remote-tracking branch 'origin/master'
new a345f00 add file list doc
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[incubator-iotdb] 02/02: add file list doc
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch add_file_list
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
commit a345f00539699e30ddad3fc4454a28ff87fb0ef6
Author: qiaojialin <64...@qq.com>
AuthorDate: Wed Jul 22 22:00:39 2020 +0800
add file list doc
---
docs/zh/SystemDesign/StorageEngine/FileLists.md | 140 ++++++++++++++++++++++++
site/src/main/.vuepress/config.js | 1 +
2 files changed, 141 insertions(+)
diff --git a/docs/zh/SystemDesign/StorageEngine/FileLists.md b/docs/zh/SystemDesign/StorageEngine/FileLists.md
new file mode 100644
index 0000000..a236bd4
--- /dev/null
+++ b/docs/zh/SystemDesign/StorageEngine/FileLists.md
@@ -0,0 +1,140 @@
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+# 磁盘文件汇总
+
+## 系统文件目录 data/system
+
+* data/system/schema/mlog.txt (元数据日志)
+
+ http://iotdb.apache.org/SystemDesign/SchemaManager/SchemaManager.html#log-management-of-metadata
+
+* data/system/schema/system.properties (系统不可变参数)
+
+ ```
+ partition_interval=9223372036854775807
+ timestamp_precision=ms
+ tsfile_storage_fs=LOCAL
+ enable_partition=false
+ max_degree_of_index_node=1024
+ tag_attribute_total_size=700
+ iotdb_version=UNKNOWN
+ ```
+
+* data/system/schema/tlog.txt (标签、属性文件）
+
+ http://iotdb.apache.org/SystemDesign/SchemaManager/SchemaManager.html#tlog
+
+
+* data/system/storage_groups/{sg_name}/{partition}/Version-xxx （每个分区的版本控制文件，无内容）
+
+* data/system/storage_groups/upgrade/Version-xxx (升级过程的版本控制文件)
+
+* data/system/upgrade/upgrade.txt (升级日志文件)
+
+格式：
+
+```
+oldTsFilePath,UpgradeCheckStatus
+```
+
+* UpgradeCheckStatus: 表示文件的升级状态
+ * BEGIN_UPGRADE_FILE(1): 待升级的文件
+ * AFTER_UPGRADE_FILE(2): 升级后的文件已经封口
+ * UPGRADE_SUCCESS(3): 完成了旧文件删除，移动新文件
+
+示例：
+
+```
+xxx/root.group_11/upgrade/1587634378188-101-0.tsfile,1
+xxx/root.group_11/upgrade/1587634378188-101-0.tsfile,2
+xxx/root.group_11/upgrade/1587634378188-101-0.tsfile,3
+...
+```
+
+* data/system/users/{user_name}.profile (用户信息)
+
+```
+1. username (UTF-8)
+2. password (UTF-8)
+3. privilegeNumber n (int)
+4. n privileges (Privelege)
+5. rolNumber k (int)
+6. k role names (UTF-8)
+7. UseWaterMarkFlag (Boolean)
+```
+
+* data/system/roles/ (权限信息)
+
+```
+1. roleName (UTF-8)
+2. privilegeNumber n (int)
+3. n privileges (Privilege)
+```
+
+Privilege 格式
+
+```
+1. bindingPath (UTF-8)
+2. privilegeNumber n (int)
+3. n privilegeOrdinals (int)
+```
+
+## 数据文件目录 data/data
+
+* data/data/sequence/{storage_group}/{partition}/{timestamp}-{version}-{merge_times}.tsfile (数据文件)
+
+http://iotdb.apache.org/SystemDesign/TsFile/Format.html#_1-2-tsfile-overview
+
+* data/data/sequence/{storage_group}/{partition}/{tsfile}.resource (对应TsFile的设备时间索引)
+
+```
+deviceId(String),start_time(long)
+deviceId(String),start_time(long)
+deviceId(String),end_time(long)
+deviceId(String),end_time(long)
+```
+
+* data/data/sequence/{storage_group}/{partition}/{tsfile}.mods (数据修改操作标记)
+
+http://iotdb.apache.org/SystemDesign/DataQuery/QueryFundamentals.html#query-with-modifications
+
+* data/data/sequence/{storage_group}/{partition}/{tsfile}-{level}-{timestamp}.vm (未封口的 TsFile 文件)
+
+* data/data/sequence/{storage_group}/{partition}/{tsfile}.flush (TsFile持久化memtable时的日志文件，无内容)
+
+* data/data/sequence/{storage_group}/{partition}/{tsfile}.vm.log (合并 vm 的日志文件)
+
+```
+源文件
+文件path
+目标文件
+文件path
+device，offset （已经合并的设备和目标文件完整的ChunkGroup的offset）
+```
+
+## 写前日志目录 data/wal
+
+* data/wal/{storage_group}-{tsfile_name}/wal{x} (对应某个 TsFile 的写前日志文件)
+
+文件格式为写入执行计划的序列化内容: InsertRowPlan、InsertTabletPlan、DeletePlan
+
+
diff --git a/site/src/main/.vuepress/config.js b/site/src/main/.vuepress/config.js
index d609538..ec97b4b 100644
--- a/site/src/main/.vuepress/config.js
+++ b/site/src/main/.vuepress/config.js
@@ -1046,6 +1046,7 @@ var config = {
{
title: '存储引擎',
children: [
+ ['StorageEngine/FileLists','磁盘文件汇总'],
['StorageEngine/StorageEngine','存储引擎'],
['StorageEngine/WAL','写前日志'],
['StorageEngine/FlushManager','FlushManager'],
[incubator-iotdb] 01/02: Merge remote-tracking branch
'origin/master'
Posted by qi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
qiaojialin pushed a commit to branch add_file_list
in repository https://gitbox.apache.org/repos/asf/incubator-iotdb.git
commit 9beddd9de2218d598409ff94364f6af8b7c3bb39
Merge: 02c2bba be4d4b1
Author: qiaojialin <64...@qq.com>
AuthorDate: Wed Jul 22 21:57:56 2020 +0800
Merge remote-tracking branch 'origin/master'
client-py/readme.md | 8 +-
client-py/src/Session.py | 456 +++++++++++++++++++++
client-py/src/SessionExample.py | 105 +++++
client-py/src/client_example.py | 347 ----------------
client-py/src/utils/Field.py | 176 ++++++++
client-py/src/utils/IoTDBConstants.py | 52 +++
client-py/src/utils/IoTDBRpcDataSet.py | 219 ++++++++++
client-py/src/utils/RowRecord.py | 55 +++
client-py/src/utils/SessionDataSet.py | 101 +++++
client-py/src/utils/Tablet.py | 133 ++++++
client-py/src/utils/__init__.py | 18 +
.../main/java/org/apache/iotdb/SessionExample.java | 18 +-
.../iotdb/db/engine/flush/VmLogAnalyzer.java | 31 +-
.../org/apache/iotdb/db/engine/flush/VmLogger.java | 1 +
.../db/engine/storagegroup/TsFileProcessor.java | 38 +-
.../tsfile/common/constant/TsFileConstant.java | 1 -
.../apache/iotdb/tsfile/write/record/Tablet.java | 5 +-
17 files changed, 1357 insertions(+), 407 deletions(-)