You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/14 12:24:08 UTC
[GitHub] [incubator-doris] Lchangliang opened a new issue, #10135: [Feature] Lightweight schema change of add/drop column
Lchangliang opened a new issue, #10135:
URL: https://github.com/apache/incubator-doris/issues/10135
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Background
Add/drop column are heightly operators. They will create new tablet and rewrite the data from old tablet. When add/drop column frequently, a lot of time is wasted waiting. So we need a new way to optimize the process.
Improvement
This improvement involves three aspects, read, writer, compaction. In original impl, BE will hold the tablet schema, set unique id for each column. When read/writer/compaction, BE can get the schema from tablet meta. The core of the modification is
that get the schema from FE when read/writer. And Every rowset will hold its schema. Using the newest schema (largest verison rowset) when doing compaction.
Modification
1. Generate Unique ID by FE.
2. When reading/inserting data, FE will send the newest schema to BE.
3. When inserting, BE will persistent the schema with rowset meta.
4. When doing compaction, BE will choose newest schema from compation rowsets and make it persistent with new rowset meta after compaction.
5. The improvement is only acting on add/drop value. If add/drop key, it will be done by the old way.
6. It will compatible with old table. Old table is mean that system already has tables before the upgrade. But old table will always do the change by old way although the column is value.
Result
When add/drop value column, they will be lightweight operators. They don't need rewrite the data and complete quickly.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring closed issue #10135: [Feature] Lightweight schema change of add/drop column
Posted by GitBox <gi...@apache.org>.
dataroaring closed issue #10135: [Feature] Lightweight schema change of add/drop column
URL: https://github.com/apache/doris/issues/10135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Lchangliang commented on issue #10135: [Feature] Lightweight schema change of add/drop column
Posted by GitBox <gi...@apache.org>.
Lchangliang commented on issue #10135:
URL: https://github.com/apache/doris/issues/10135#issuecomment-1181392847
TODO:
1. FE synchronization unique_id from BE for old table.
2. optimize that too more tablet_schemas are in memory.
3. About flink connector. Support Light Schema Change and optimize that remove data use streaming load must need head.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org