You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2023/06/25 14:18:17 UTC
[GitHub] [doris] freemandealer opened a new issue, #18476: [Feature] load code refactoring
freemandealer opened a new issue, #18476:
URL: https://github.com/apache/doris/issues/18476
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Description
# Motivation
Doris now supports many load methods. The implementation code of these methods is intertwined and the attributes overlap. To improve maintainability, we plan to refactor the code.
# Goals
long-term: unify all data loading methods (brokerload, insert, etc) into one
short-term:
1) unify the attribute specification method and parsing method and default values of various loads;
2) reuse single code base as much as possible and delete duplicate code;
# Plan Overview
We plan to use insert stmt as the common backend for all load methods. Various loads retain their original user interface, but will eventually be converted into insert stmt for planning and subsequent execution.
# Task Breakdown
Phase1: make S3/Broker load work on `insert stmt`
1) #18477
2) #18480
3) #18478
Phase2: make all other load method work on `insert stmt`
1) run streamload with insert stmt planner
2) support mysql load
3) support routine load
...
Phase3: fully support all use cases in all load method
1) insert stmt support writing multi-tables in single transaction
2) insert stmt support sequence column v1
...
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring closed issue #18476: [Feature] load code refactoring
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring closed issue #18476: [Feature] load code refactoring
URL: https://github.com/apache/doris/issues/18476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on issue #18476: [Feature] load code refactoring
Posted by "TangSiyang2001 (via GitHub)" <gi...@apache.org>.
TangSiyang2001 commented on issue #18476:
URL: https://github.com/apache/doris/issues/18476#issuecomment-1505489366
May I have a try ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring closed issue #18476: [Feature] load code refactoring
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring closed issue #18476: [Feature] load code refactoring
URL: https://github.com/apache/doris/issues/18476
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org