You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/03/17 09:19:52 UTC
[GitHub] [incubator-iotdb] SolomonAnn opened a new pull request #917:
[IOTDB-559] Optimize the working mechanism of MemTable
SolomonAnn opened a new pull request #917: [IOTDB-559] Optimize the working mechanism of MemTable
URL: https://github.com/apache/incubator-iotdb/pull/917
I've optimzing the working mechanism of MemTable by applying the sliding window approach. The basic idea is that when a MemTable is full, IoTDB will only flush parital data of the MemTable and keep the rest. Take the case in the [IOTDB-559](https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-559?filter=allopenissues) for example, we set the proportion of flushed data in the filled MemTable as 80%. When IoTDB receives <1,1> <2,2> <3,3> <7,7> <5,5> <6,6> <4,4>, one MemTable called **M1** will be filled with <1,1> <2,2> <3,3> <7,7> <5,5>. Then the following steps will be taken one by one:
1. **M1** will be sorted as <1,1> <2,2> <3,3> <5,5> <7,7>;
2. <1,1> <2,2> <3,3> <5,5> will be flushed and <7,7> will be kept into one new MemTable **M2**;
3. The latest flushed time will be updated to 5;
4. <6,6> will be inserted into **M2** and <4,4> will be regarded as an unsequence data point.
Compared the current mechanism, the number of generated unsequence data due to network transmission can be reduced. As the attached figure shown, when we set the proportion of flushed data in the filled MemTable as 80% and set the proportion of unsequence data in the inserted data as 10%, 20% and 30% respectively, it doesn't generate any unsequence data. However, we can also notice that the speed of inserting declined by 26%. Now I don't know how to carry on the further optimization and I need your help sincerely.
NOTICE: I mainly modified StorageGroupProcessor.java and TsFileProcessor.java.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [iotdb] JackieTien97 closed pull request #917: [IOTDB-559] Optimize the working mechanism of MemTable
Posted by GitBox <gi...@apache.org>.
JackieTien97 closed pull request #917:
URL: https://github.com/apache/iotdb/pull/917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org