You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "lvshaokang (via GitHub)" <gi...@apache.org> on 2023/05/12 03:35:14 UTC
[GitHub] [doris] lvshaokang opened a new pull request, #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
lvshaokang opened a new pull request, #19571:
URL: https://github.com/apache/doris/pull/19571
# Proposed changes
Issue Number: close #18476
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1558949921
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1550627656
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1555961456
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1552385738
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1558690110
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1201650492
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/UnifiedLoadStmt.java:
##########
@@ -55,6 +42,31 @@ public void init() {
proxyStmt.setUserInfo(getUserInfo());
}
+ public static UnifiedLoadStmt buildMysqlLoadStmt(DataDescription dataDescription, Map<String, String> properties,
+ String comment) {
+ StatementBase proxyStmt;
+ final ConnectContext connectContext = ConnectContext.get();
+ if (connectContext != null && connectContext.getSessionVariable().isEnableUnifiedLoad()) {
+ proxyStmt = new MysqlLoadStmt(dataDescription, properties, comment);
+ } else {
+ proxyStmt = new LoadStmt(dataDescription, properties, comment);
+ }
+ return new UnifiedLoadStmt(proxyStmt);
+ }
+
+ public static UnifiedLoadStmt buildBrokerLoadStmt(LabelName label, List<DataDescription> dataDescriptions,
+ BrokerDesc brokerDesc,
+ Map<String, String> properties, String comment) {
+ StatementBase proxyStmt;
+ final ConnectContext connectContext = ConnectContext.get();
+ if (connectContext != null && connectContext.getSessionVariable().isEnableUnifiedLoad()) {
+ proxyStmt = new BrokerLoadStmt(label, dataDescriptions, brokerDesc, properties, comment);
+ } else {
+ proxyStmt = new LoadStmt(label, dataDescriptions, brokerDesc, properties, comment);
+ }
+ return new UnifiedLoadStmt(proxyStmt);
+ }
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] CalvinKirs commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1201615009
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/UnifiedLoadStmt.java:
##########
@@ -55,6 +42,31 @@ public void init() {
proxyStmt.setUserInfo(getUserInfo());
}
+ public static UnifiedLoadStmt buildMysqlLoadStmt(DataDescription dataDescription, Map<String, String> properties,
+ String comment) {
+ StatementBase proxyStmt;
+ final ConnectContext connectContext = ConnectContext.get();
+ if (connectContext != null && connectContext.getSessionVariable().isEnableUnifiedLoad()) {
+ proxyStmt = new MysqlLoadStmt(dataDescription, properties, comment);
+ } else {
+ proxyStmt = new LoadStmt(dataDescription, properties, comment);
+ }
+ return new UnifiedLoadStmt(proxyStmt);
+ }
+
+ public static UnifiedLoadStmt buildBrokerLoadStmt(LabelName label, List<DataDescription> dataDescriptions,
+ BrokerDesc brokerDesc,
+ Map<String, String> properties, String comment) {
+ StatementBase proxyStmt;
+ final ConnectContext connectContext = ConnectContext.get();
+ if (connectContext != null && connectContext.getSessionVariable().isEnableUnifiedLoad()) {
+ proxyStmt = new BrokerLoadStmt(label, dataDescriptions, brokerDesc, properties, comment);
+ } else {
+ proxyStmt = new LoadStmt(label, dataDescriptions, brokerDesc, properties, comment);
+ }
+ return new UnifiedLoadStmt(proxyStmt);
+ }
+
Review Comment:
Generally LGTM
But it would be better if the code style can be optimized. Similar to this code, we can return directly without passing the else judgment again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1558835722
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1545380413
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1545062842
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1545086584
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.69 seconds
stream load tsv: 422 seconds loaded 74807831229 Bytes, about 169 MB/s
stream load json: 24 seconds loaded 2358488459 Bytes, about 93 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230512041810_clickbench_pr_142590.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1194505471
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/MysqlLoadManager.java:
##########
@@ -209,6 +214,11 @@ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, Load
return loadResult;
}
+ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, InsertStmt insertStmt, String loadId)
Review Comment:
right, only rewrite func for unified`insertstmt`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1194503520
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/MysqlLoadStmt.java:
##########
@@ -69,7 +81,7 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
for (String path : dataDescription.getFilePaths()) {
if (Config.mysql_load_server_secure_path.isEmpty()) {
throw new AnalysisException("Load local data from fe local is not enabled. If you want to use it,"
- + " plz set the `mysql_load_server_secure_path` for FE to be a right path.");
+ + " plz set the `mysql_load_server_secure_path` for FE to be a right path.");
Review Comment:
may be checkstyle auto format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1556209762
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #19571:
URL: https://github.com/apache/doris/pull/19571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1193712366
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/MysqlLoadStmt.java:
##########
@@ -69,7 +81,7 @@ public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
for (String path : dataDescription.getFilePaths()) {
if (Config.mysql_load_server_secure_path.isEmpty()) {
throw new AnalysisException("Load local data from fe local is not enabled. If you want to use it,"
- + " plz set the `mysql_load_server_secure_path` for FE to be a right path.");
+ + " plz set the `mysql_load_server_secure_path` for FE to be a right path.");
Review Comment:
unnecessary change
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/MysqlLoadManager.java:
##########
@@ -209,6 +214,11 @@ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, Load
return loadResult;
}
+ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, InsertStmt insertStmt, String loadId)
Review Comment:
duplicated with line 50?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1193287403
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InsertStmt.java:
##########
@@ -117,7 +117,7 @@ public static class Properties {
// ---------------------------- for old insert stmt ----------------------------
public boolean isValuesOrConstantSelect() {
- throw new UnsupportedOperationException("only invoked in NativeInsertStmt");
+ return false;
Review Comment:
beacuse of the insert stmt as the abstract class, execute to this point[1] will throw exception when i tested. But I think this isn't correct.
[1] https://github.com/apache/doris/blob/master/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java#L324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1556439735
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1552416723
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1193361105
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/MysqlLoadStmt.java:
##########
@@ -31,11 +31,23 @@
import java.io.IOException;
import java.util.Collections;
import java.util.List;
+import java.util.Map;
public class MysqlLoadStmt extends InsertStmt {
private DataDescription dataDescription;
+ public MysqlLoadStmt(DataDescription dataDescription, Map<String, String> properties, String comments) {
+ this.label = new LabelName();
+ this.dataDescription = dataDescription;
+ this.properties = properties;
+ if (comments != null) {
+ this.comments = comments;
+ } else {
+ this.comments = "";
+ }
Review Comment:
we should handle there in base class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on a diff in pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on code in PR #19571:
URL: https://github.com/apache/doris/pull/19571#discussion_r1193276178
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadManagerAdapter.java:
##########
@@ -31,15 +37,25 @@
*/
public class LoadManagerAdapter {
- public void startLoadFromInsertStmt(InsertStmt insertStmt) throws DdlException {
+ private String mysqlLoadId;
+
+ public void submitLoadFromInsertStmt(ConnectContext context, InsertStmt insertStmt)
+ throws UserException, IOException {
final LoadType loadType = insertStmt.getLoadType();
final LoadManager loadManager = Env.getCurrentEnv().getLoadManager();
switch (loadType) {
case BROKER_LOAD:
loadManager.createLoadJobFromStmt(insertStmt);
+ context.getState().setOk();
break;
case MYSQL_LOAD:
// TODO: implement
Review Comment:
useless code
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InsertStmt.java:
##########
@@ -117,7 +117,7 @@ public static class Properties {
// ---------------------------- for old insert stmt ----------------------------
public boolean isValuesOrConstantSelect() {
- throw new UnsupportedOperationException("only invoked in NativeInsertStmt");
+ return false;
Review Comment:
why change this?
##########
fe/fe-core/src/main/java/org/apache/doris/load/loadv2/MysqlLoadManager.java:
##########
@@ -148,9 +149,13 @@ public MysqlLoadManager(TokenManager tokenManager) {
public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, LoadStmt stmt, String loadId)
throws IOException, UserException {
+ return executeMySqlLoadJobFromStmt(context, stmt.getDataDescriptions().get(0), loadId);
+ }
+
+ public LoadJobRowResult executeMySqlLoadJobFromStmt(ConnectContext context, DataDescription dataDesc, String loadId)
+ throws IOException, UserException {
LoadJobRowResult loadResult = new LoadJobRowResult();
// Mysql data load only have one data desc
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1558625096
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1558724055
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 33.76 seconds
stream load tsv: 420 seconds loaded 74807831229 Bytes, about 169 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 76.9 seconds inserted 10000000 Rows, about 130K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523075038_clickbench_pr_148793.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] lvshaokang commented on pull request #19571: [feature](load-refactor) Unfied mysql load use InsertStmt
Posted by "lvshaokang (via GitHub)" <gi...@apache.org>.
lvshaokang commented on PR #19571:
URL: https://github.com/apache/doris/pull/19571#issuecomment-1559034999
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org