[jira] [Commented] (IOTDB-698) Enhance management of TsFileResource

["WangChao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IOTDB-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183018#comment-17183018 ] 

WangChao commented on IOTDB-698:
--------------------------------

Hi, I have an idea about the index management of Device.

 

In the picture, we maintain two levels of index to manage the device index.

 

In the primary index, there are 100000 virtual devices, and each real device will be hashed to a virtual device.

In the secondary index, there are 1000 virtual devices. The virtual devices of each primary index will be hashed and mapped to a virtual device.

 


write:

1. Each time a tsfile is sealed, all device indexes [deviceid, (starttime, Endtime, tsfilepath)] are mapped and added to the primary index

2. When the primary index is full, the virtual device of the primary index will be hashed and mapped to the secondary index.

 

query:

1. According to the device name, calculate the virtual device corresponding to the primary index according to the above hash method, and then hash the virtual device corresponding to the secondary index according to the virtual device of the first level index. Then, according to the given time filtering conditions, the primary index file corresponding to the virtual device in the secondary index is filtered out

2. Open the primary index file and find the relevant tsfile according to the time filtering conditions and the corresponding primary index virtual device

3. Read and query data in tsfile

 

 

 

!https://code.bonc.com.cn/confluence/download/attachments/59935226/tsfileResouce%E8%AE%BE%E8%AE%A1.jpg?version=5&modificationDate=1597289697000&api=v2!

 

 

Feedback is welcome!

> Enhance management of TsFileResource 
> -------------------------------------
>
>                 Key: IOTDB-698
>                 URL: https://issues.apache.org/jira/browse/IOTDB-698
>             Project: Apache IoTDB
>          Issue Type: Improvement
>          Components: Core/Engine
>            Reporter: Jialin Qiao
>            Assignee: WangChao
>            Priority: Major
>
> We need a global TsFileResourceManager to manage TsFileResource in all StorageGroupProcessors.
>  
> When the memory of resources exceeds a threshold, we need to remove them from memory.
> The TsFileResourceManager acts like a cache.
>  
> We also need to design an index to quickly find resources for each device.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)