You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/30 07:11:42 UTC
[GitHub] [incubator-doris] decster opened a new issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
decster opened a new issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442
To add MemTablet type to meta, there are 2 ways:
1. Create new meta kind, like BaseTablet and MemTablet
2. Use original meta, add new fields.
Unlike original Tablet, MemTablet doesn't have components like rowsets and may not have del_predicates, alter tasks etc. Those are vectors/ptrs so it won't use much memory(or pb space) if they are empty. Also it's worth to mention now MemTablet can handle multiple schema(schema change) inside MemTablet, it's won't be necessary to create a new tablet after schema change, but we can also keep the old behavior, depending on which way is easier.
To make things simple, we can first add a field, e.g. `bool is_memory [default=false]` to mark this tablet is a MemTablet, and latter add new fields if needed.
One potential issue is there already exists a term `in_memory` in TabletSchema when creating tablet, which means "keep all pages in page caches at best", causes confusion.
Another option is to add an `enum StorageEngine`, with types of MEMORY and DISK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] decster commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
decster commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-621681381
> so these TabletMeta in RocksDB need a gc mechanism? For example, cleanup on BE restart?
"cleanup" meaning "delete tablet" or "truncate tablet" ? In an early implementation, I wouldn't expect actual usage except for testing, just choose the most simple method, maybe "delete tablet"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-622355933
@decster Hi, how do we expect our user to create the real in memory table (different from early `in memory` OLAP table cache )? I think if the user interface is decided，this issue will be determinate with it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] decster commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
decster commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-623112875
> i.e. olap engine have aggr/uniq/duplicate model, while this storage engine only have standard table+primary key semantics
More about how the memory engine data model map to the original aggr/uniq/duplicate model
In the current design, memory engine only maps to uniq data model, all key column is primary key and all none-key column is REPLACE.
In the future, aggr column can be treated as updatable column, which should be simple to support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-621690223
> > so these TabletMeta in RocksDB need a gc mechanism? For example, cleanup on BE restart?
>
> "cleanup" meaning "delete tablet" or "truncate tablet" ? In an early implementation, I wouldn't expect actual usage except for testing, just choose the most simple method, maybe "delete tablet"
I mean after BE restart, the data of MemTablet is losing. So the TabletMeta of these MemTablet in RocksDB become orphans. We need a way to clean them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-623223611
>we may need more discussions about how to make the table data model and create table syntax/semantics more consistent.
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] decster edited a comment on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
decster edited a comment on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-621681381
> so these TabletMeta in RocksDB need a gc mechanism? For example, cleanup on BE restart?
"cleanup" meaning "delete tablet" or "truncate tablet" ? In an early implementation, I wouldn't expect actual usage except for testing, just choose the most simple method, maybe "delete tablet"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] decster edited a comment on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
decster edited a comment on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-621703329
> So the TabletMeta of these MemTablet in RocksDB become orphans. We need a way to clean them.
"delete tablet" does exactly this, another way is only allowing 1 replica and "truncate tablet"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] decster commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
decster commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-623109292
The original design is to reuse the existing property `storage_medium = memory` and data is persistent on SSD, this introduces little change, it won't change the syntax or add a new property, we can use this as a test entry point at least, and defer the user interface discussion later.
Also, in my opinion, the whole table data model and create table syntax is strange/messy compared to other OLAP/MPP systems, i.e. olap engine have aggr/uniq/duplicate model, while this storage engine only have standard table+primary key semantics(which is more common in classic database domain), we may need more discussions about how to make the table data model and create table syntax/semantics more consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] decster commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
decster commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-621703329
> So the TabletMeta of these MemTablet in RocksDB become orphans. We need a way to clean them.
"delete tablet" does exactly this, another way is only allowing 1 replica and "truncate tablet"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli closed issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
chaoyli closed issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3442: [Memory Engine] Add MemTablet type to TabletMeta
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3442:
URL: https://github.com/apache/incubator-doris/issues/3442#issuecomment-621661593
I prefer `enum StorageEngine` (or maybe `enum TabletType`?), which is more clearly and extensible than a `bool`.
And another question, in the early implementation, the persistence of memtablet data is not supported, so these TabletMeta in RocksDB need a gc mechanism? For example, cleanup on BE restart?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org