You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/08/10 06:35:07 UTC
[GitHub] [doris] compasses opened a new issue, #11640: [Feature] New BulkLoad, which support build segment file isolate from Doris cluster
compasses opened a new issue, #11640:
URL: https://github.com/apache/doris/issues/11640
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
Currently, there are several ways to load data into Doris, like broker load, stream load etc. But all these ways have some kind of shortcomings from our perspective, for example:
1. High resource cost, because each tablet has multiple replica, and each replica will do the data load separately.
2. Hard to ensure cluster stable and performance, load jobs may lead to high load of resource competition.
3. Cause query latency climbing up and down, this kind of issue may came across again and again.
So we may want a new way to do data load, like lightweight read / write splitting, which can extremely keep high-throughput write and read.
Here we just have a very rough design, and many details need clarify.
The overall flow：
The new bulk load may have some connect with function like backup / restore, broker load etc.
1. FE issue the bulk load command, and BE will write tablet meta to HDFS.
2. Then FE will schedule a spark / flink job to run segment builder, which will read HDFS data file and build segment file to local, and upload these segment file to HDFS when build finish.
3. Then FE will start to load these segment from HDFS, mainly each BE do the real job.
4. Last the FE need publish this transaction like the broker load.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Feature] New BulkLoad, which support build segment file isolate from Doris cluster [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed issue #11640: [Feature] New BulkLoad, which support build segment file isolate from Doris cluster
URL: https://github.com/apache/doris/issues/11640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org