You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/29 11:24:39 UTC
[GitHub] [incubator-doris] wyb opened a new issue #3433: [Spark load] Doris support Spark load
wyb opened a new issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433
For many users who want to load data into doris for the first time, they have large amount of data, about 10G+，it is hard to support to load so large data into doris at one time using Broker load or Stream load. To resolve this problem, We proposal a new solution to load data by using spark cluster.
Spark clusters are used to preprocess data (bitmap global dict build, partition, sort, aggregation) in spark load, which can improve Doris load performance of large data volume and save the computing resources of Doris.
Spark load is mainly used for the initial migration from other systems or loading large amounts of data into Doris.
```
+
| 0. User create spark load job
+----v----+
| FE |---------------------------------+
+----+----+ |
| 3. FE send push tasks |
| 5. FE publish version |
+------------+------------+ |
| | | |
+---v---+ +---v---+ +---v---+ |
| BE | | BE | | BE | |1. FE submit Spark ETL job
+---^---+ +---^---+ +---^---+ |
|4. BE push with broker | |
+---+---+ +---+---+ +---+---+ |
|Broker | |Broker | |Broker | |
+---^---+ +---^---+ +---^---+ |
| | | |
+---+------------+------------+---+ 2.ETL +-------------v---------------+
| HDFS +-------> Spark cluster |
| <-------+ |
+---------------------------------+ +-----------------------------+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/6] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/6] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/6] Fe create job
#3819 [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt
#3716 [Spark load][Fe 5/6] Fe submit spark etl job
#3717 [Spark load][Fe 6/6] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
xy720 edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-636846173
**Be handle push task**
#3742 [Spark load][Be 1/1] Be handle push task
**Optimize**
#3878 [Spark load][broker load]Optimize reading parquet format file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb removed a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb removed a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635181748
Add resource manager #3418
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
#3418 [Spark load] Add resource manager (finished)
#3712 [Spark load] Add spark etl job config
#3718 [Spark load] Update push task thrift interface
#3715 [Spark load] Fe create job
#3716 [Spark load] Fe submit spark etl job
#3717 [Spark load] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xy720 commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
xy720 commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-636846173
**Be handle push task**
#3742 [Spark load][Be 1/1] Be handle push task
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wangbo commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-626104019
**Count Distinct Module**
#3319 Support Java Verision HyperLogLog(REVIEWING)
#3061 Doris Support Using Hive Table to Build Global Dict(TESTING)
#3088 Support Java version 64 bits Integers for BITMAP type(MERGED)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
[Spark load] Add spark etl job config #3712
[Spark load] Update push task thrift interface #3718
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wangbo edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-626104019
**Count Distinct Module**
#3319 Support Java Verision HyperLogLog(REVIEWING)
#3061 Doris Support Using Hive Table to Build Global Dict(TESTING)
#3088 Support Java version 64 bits Integers for BITMAP type(MERGED)
**Spark DPP Module**
#3726 [Spark Load] Rollup Tree Builder
#3728 [Spark Load] Using SparkDpp to complete some calculation in Spark Load
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635469749
[Spark load] Fe submit spark etl job #3716
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/5] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/5] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/5] Fe create job
#3716 [Spark load][Fe 4/5] Fe submit spark etl job
#3717 [Spark load][Fe 5/5] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-621142283
design doc
#2855 [Proposal] support spark load
#2887 [Proposal] Support Spark Convert Doris Segment
#3010 Spark load interface
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
#3418 [Spark load] Add resource manager (Merged)
#3712 [Spark load] Add spark etl job config
#3718 [Spark load] Update push task thrift interface
#3715 [Spark load] Fe create job
#3716 [Spark load] Fe submit spark etl job
#3717 [Spark load] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
Add spark etl job config #3712
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load] Add spark etl job config
#3718 [Spark load] Update push task thrift interface
#3715 [Spark load] Fe create job
#3716 [Spark load] Fe submit spark etl job
#3717 [Spark load] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/6] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/6] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/6] Fe create job (Merged)
#3819 [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt (Merged)
#3716 [Spark load][Fe 5/6] Fe submit spark etl job (Merged)
#3717 [Spark load][Fe 6/6] Fe process etl and loading state job (Merged)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635181748
Add resource manager #3418
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/6] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/6] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/6] Fe create job (Merged)
#3819 [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt
#3716 [Spark load][Fe 5/6] Fe submit spark etl job
#3717 [Spark load][Fe 6/6] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/6] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/6] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/6] Fe create job (Merged)
#3819 [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt (Merged)
#3716 [Spark load][Fe 5/6] Fe submit spark etl job
#3717 [Spark load][Fe 6/6] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635482036
[Spark load] Fe process etl and loading state job #3717
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb removed a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb removed a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635310967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/6] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/6] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/6] Fe create job (Merged)
#3819 [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt (Merged)
#3716 [Spark load][Fe 5/6] Fe submit spark etl job (Merged)
#3717 [Spark load][Fe 6/6] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-621172837
1. can refer this issue in related PR and issue.
2. I will create a project "Spark Load" to track this feature.
3. You can create an issue for each part of this project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-621142283
**Design doc**
#2855 [Proposal] support spark load
#2887 [Proposal] Support Spark Convert Doris Segment
#3010 Spark load interface
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb commented on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb commented on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-621142283
design doc
[Proposal] support spark load #2855
[Proposal] Support Spark Convert Doris Segment #2887
Spark load etl interface #3010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wyb edited a comment on issue #3433: [Spark load] Doris support Spark load
Posted by GitBox <gi...@apache.org>.
wyb edited a comment on issue #3433:
URL: https://github.com/apache/incubator-doris/issues/3433#issuecomment-635180109
**Resource manager**
#3418 [Spark load] Add resource manager (Merged)
**Fe schedule job execution**
#3712 [Spark load][Fe 1/5] Add spark etl job config
#3718 [Spark load][Fe 2/5] Update push task thrift interface
#3715 [Spark load][Fe 3/5] Fe create job
#3716 [Spark load][Fe 4/5] Fe submit spark etl job
#3717 [Spark load][Fe 5/5] Fe process etl and loading state job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org