You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/10/26 14:06:57 UTC
[GitHub] [incubator-doris] weizuo93 opened a new issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
weizuo93 opened a new issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797
MemTable flush can be activated by three situations：
A. The size of MemTable reach to the flush threshold `config::write_buffer_size`;
B. Data load is finished and DeltaWriter needs to be closed；
C. Memory consumption exceeds the limit and MemTable needs to flush to reduce memory usage.
A large number of small segment files would be generated due to "situation B" above when small batch data (far less than `config::write_buffer_size`) are loaded frequently , which will lead to lower efficiency for scan operations.
We can optimize MemTable flush mechanism like this:
(1) Maintain a `vector<MemTable>` for each tablet;
(2) When close DeltaWriter and flush MemTable for a tablet, do not reset the Memtable if there is no flush operation before for this tablet in the data load and push the MemTable into `vector<MemTable>`;
(3) When next flush operation for this tablet comming, judge whether the flush is activated by :
a. "situation A": Merge all MemTable in `vector<MemTable>` into current MemTable , flush the merged MemTable , delete all the rowset corresponding to MemTable in `vector<MemTable>` and clear `vector<MemTable>`;
b. "situation B": If the total size of MemTable in `vector<MemTable>`and current MemTable reach threshold `config::write_buffer_size`, merge all MemTable in `vector<MemTable>` into current MemTable, flush the merged MemTable, delete all the rowset corresponding to MemTable in `vector<MemTable>` and clear `vector<MemTable>`; If the total size of MemTable in `vector<MemTable>`and current MemTable is less than the threshold `config::write_buffer_size`, push the MemTable into `vector<MemTable>` and only flush the current MemTable;
c. "situation C": flush the current MemTable 、reset all the MemTable in `vector<MemTable>` and clear `vector<MemTable>`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716918696
I understand this is equivalent to handing part of the cumulative compaction work to the load stage to complete?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716924361
Currently, MemTable isn't queryable. If you buffer a small Memtable for a long time, which means the user couldn't real-time analyze.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716940441
@weizuo93 I think this is a very big change. You should consider the read process and replication/clone process, and maybe you should also consider a Write-Ahead-Log. Could you please provide more information on WAL and clone and read?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716935452
> In fact. you are doing the in-memory compaction, but we will also do disk compaction, which is repetitive. How do you avoid this issue?
Sometimes, disk compaction may not be done in time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei edited a comment on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
yiguolei edited a comment on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716940441
@weizuo93 I think this is a very big change. You should consider the read process and replication/clone process, and maybe you should also consider a Write-Ahead-Log. Could you please provide more information on WAL and clone and read?
And also It does not mean load process finished when memtable is generated because load is a two phase procedure. There may be many useless memtable in vector<MemTable>.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 closed issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
weizuo93 closed issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-717150935
I encountered some troubles when I optimize MemTable flush mechanism to solve the problem of to many small segment files. I'll try other solution to deal with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716929333
In fact. you are doing the in-memory compaction, but we will also do disk compaction, which is repetitive. How do you avoid this issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen removed a comment on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
kangkaisen removed a comment on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716924361
Currently, MemTable isn't queryable. If you buffer a small Memtable for a long time, which means the user couldn't real-time analyze.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716933891
> Currently, MemTable isn't queryable. If you buffer a small Memtable for a long time, which means the user couldn't real-time analyze.
All the MemTables in `vector<MemTable>` have been flushed after each load job finished. These MemTables are buffered only for the purpose of merging with subsequent MemTable to flush a large segment file. After merge operation finished and flush a large segment file, all the previous rowsets corresponding to MemTable in `vector<MemTable>` will be deleted. So, it will not have an effect on real-time analyze.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716938157
> all the previous rowsets corresponding to MemTable in `vector<MemTable>` will be deleted
This is not easy to implement. the rowsets which already been flushed to the disk may be merged by cumulative or base compaction, so that you can not find all origin rowsets.
If you implement what you designed, you need to consider the existing compaction logic. As far as I can think of, I am not sure whether the additional memory and IO overhead brought by this approach will have other effects.
In fact, this is an implementation of "Group Commit". And I think a readable Memtable and WAL may be a better way.
I remember that @vagetablechicken and @decster was try to implement "Memory Engine #3382". Can we go that way?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] weizuo93 commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
weizuo93 commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716933515
> I understand this is equivalent to handing part of the cumulative compaction work to the load stage to complete?
Absolutely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #4797: [Proposal] Optimize MemTable flush mechanism to avoid too many small segment files for data load
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #4797:
URL: https://github.com/apache/incubator-doris/issues/4797#issuecomment-716938770
> > In fact. you are doing the in-memory compaction, but we will also do disk compaction, which is repetitive. How do you avoid this issue?
>
> Sometimes, disk compaction may not be done in time.
You shouldn't do this assumption always, we should the mechanism to avoid one tablet does in-memory compaction and disk compaction at the same time.
Besides, you should consider memory usage. If one table load involves 1000 tablets, at most, you will use 1000 * 100MB = 100GB memory. How do you avoid this issue?
Does the vector<MemTable> respect to `DeltaWriter::flush_memtable_and_wait`? If so, I think it isn't very simple.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org