Re: Out-of-Memory Analysis- reformat for reading

[江天 <jt...@163.com>]

Good summary, I would like to make some minor supplements.

1. I think the data of one series can be discarded as soon as its corresponding Chunk is generated, so you do not need to bother redesigning it, a simple optimization may do the trick.

2. Yeah, as long as you can estimate how much memory is occupied by a query. But I suggest we focus on insertion first.

3. OF COURSE NOT. Considering all intermediate objects and object overhead takes a long way to go, but we can adjust it little by little.

4. I have to point out that you may get BufferWrite working memtable, BufferWrite flushing memtable, Overflow working memtable, Overflow  flushing memtable at the same time, so a simple double may be just not enough for estimation.

> 在 2019年4月22日，下午12:10，kangr15 <ka...@mails.tsinghua.edu.cn> 写道：
> 
> Hi all:
> Sorry forthe text format, the follows are reorganized:
> 1. Flushing to disk may double the memory cost: A storage group maintains a list of ChunkGroups in memory and will be flushed to disk when its occupied memory exceeding the threshold (128MB by default).
> In the current implementation, when starting to flush data, a ChunkGroup is encoded in memory and thereby a new byte array is kept in memory. Until all ChunkGroups have been encoded in memory, their corresponding byte arrays can be released together. Since the byte array has a comparable size with original data (0.5× to 1×), the above strategy may double the memory in the worst case.
> Solution: It is needed to redesign the flush strategy. In TsFile, a Page is the minimal flush unit, where a ChunkGroup contains several Chunks and a Chunk contains several pages. Once a page is encoded into a byte array, we can flush the byte array to disk and then release it. In this case, the extra memory is a page size (64KB by default) at most. This modification involves a list of cascading change, including metadata format and writing process.
> 
> 2. Memory Control Strategy: It is needed to redesign the memory Control Strategy. For example, assigning 60% memory to the writing process and 30% memory to the querying process. The writing memory includes the memory table and the flush process. As an Insert coming, if its required memory exceeds TotalMem * 0.6 - MemTableUsage - FlushUsage, the Insert will be rejected.
> 3. Is the memory statistics accuracy? In current codes, the memory usage of a TSRecord Java Object, corresponding to an Insert SQL, is calculated by summating its DataPoints. e.g., "insert into root.a.b.c(timestamp,v1, v2) values(1L, true, 1.2f)", its usage is 8 + 1 + 4=13, which ignores the size of object head and others. It is needed to redesign the memory statistics accuracy carefully.
> 4. Is there still the memory leak? As shown in the log of the last crash due to the out of memory exception, we find out the actual JVM memory is 18G, whereas our memory statistic module only counts 8G. Besides the inaccuracy mentioned in Q3, we doubt there are still memory leak or other potential problems. We will continue to debug it.
> 
> 
> 
> 
> —
> 顺颂时祺
> 康荣
> 清华大学软件学院
> —
> Best Regards,
> Rong Kang
> School of Software, Tsinghua University
> 
> 
> 原始邮件
> 发件人:kangr15kangr15@mails.tsinghua.edu.cn
> 收件人:devdev@iotdb.apache.org
> 发送时间:2019年4月22日(周一) 12:01
> 主题:Out-of-Memory Analysis
> 
> 
> Hi all: Flushing to disk may double the memory cost: A storage group maintains a list of ChunkGroups in memory and will be flushed to disk when its occupied memory exceeding the threshold (128MB by default). In the current implementation, when starting to flush data, a ChunkGroup is encoded in memory and thereby a new byte array is kept in memory. Until all ChunkGroups have been encoded in memory, their corresponding byte arrays can be released together. Since the byte array has a comparable size with original data (0.5× to 1×), the above strategy may double the memory in the worst case. Solution: It is needed to redesign the flush strategy. In TsFile, a Page is the minimal flush unit, where a ChunkGroup contains several Chunks and a Chunk contains several pages. Once a page is encoded into a byte array, we can flush the byte array to disk and then release it. In this case, the extra memory is a page size (64KB by default) at most. This modification involves a list of cascading change, including metadata format and writing process. Memory Control Strategy: It is needed to redesign the memory Control Strategy. For example, assigning 60% memory to the writing process and 30% memory to the querying process. The writing memory includes the memory table and the flush process. As an Insert coming, if its required memory exceeds TotalMem * 0.6 - MemTableUsage - FlushUsage, the Insert will be rejected. Is the memory statistics accuracy? In current codes, the memory usage of a TSRecord Java Object, corresponding to an Insert SQL, is calculated by summating its DataPoints. e.g., "insert into root.a.b.c(timestamp,v1, v2) values(1L, true, 1.2f)", its usage is 8 + 1 + 4=13, which ignores the size of object head and others. It is needed to redesign the memory statistics accuracy carefully. Is there still the memory leak? As shown in the log of the last crash due to the out of memory exception, we find out the actual JVM memory is 18G, whereas our memory statistic module only counts 8G. Besides the inaccuracy mentioned in Q3, we doubt there are still memory leak or other potential problems. We will continue to debug it. — 顺颂时祺 康荣 清华大学软件学院 — Best Regards, Rong Kang School of Software, Tsinghua University