You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zzzzzzzs (via GitHub)" <gi...@apache.org> on 2023/10/06 09:34:48 UTC

[PR] [Enhancement](Load) Nereids supports http_stream [doris]

zzzzzzzs opened a new pull request, #25049:
URL: https://github.com/apache/doris/pull/25049

   ## Proposed changes
   
   Nereids supports http_stream.
   
   Due to the possibility of errors in Nereids, it will fall back to the legacy planner. Due to previous `ctx->need_schema` is a global variable, and Nereids has read column information. `ctx->need_schema` has been set to true. If fall back to the legacy planner, because `ctx->need_schema` is true and cannot retrieve column information again. To solve this problem, PR https://github.com/apache/doris/pull/24978/  needs to be merged first.
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762887321

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1351613931


##########
fe/fe-core/src/main/java/org/apache/doris/planner/external/TVFScanNode.java:
##########
@@ -65,6 +65,10 @@ public TVFScanNode(PlanNodeId id, TupleDescriptor desc, boolean needCheckColumnP
         tableValuedFunction = (ExternalFileTableValuedFunction) table.getTvf();
     }
 
+    public String getTvfName() {

Review Comment:
   Is this method useful?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1352763361


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -166,7 +175,13 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
         state.addTableIndexes(physicalOlapTableSink.getTargetTable());
 
         executor.setProfileType(ProfileType.LOAD);
+        executor.setPlanner(planner);

Review Comment:
   moved to `generateStreamLoadNereidsPlan` function



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762921535

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1757538207

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1757599311

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 44.65 seconds
    stream load tsv:          561 seconds loaded 74807831229 Bytes, about 127 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17162288946 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762928165

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.61 seconds
    stream load tsv:          548 seconds loaded 74807831229 Bytes, about 130 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.8 seconds inserted 10000000 Rows, about 347K ops/s
    storage size: 17161988313 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762927542

   TeamCity be ut coverage result:
    Function Coverage: 36.27% (8145/22456) 
    Line Coverage: 28.50% (65301/229131)
    Region Coverage: 27.18% (33818/124419)
    Branch Coverage: 23.99% (17259/71952)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/16647e770a5c72ddc8c4ef1352bcb7858dac6bfd_16647e770a5c72ddc8c4ef1352bcb7858dac6bfd/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1817513645

   Due to a long absence of merge, there was a problem during rebase, so a new PR https://github.com/apache/doris/pull/27233 was created


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1754916547

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.3 seconds
    stream load tsv:          573 seconds loaded 74807831229 Bytes, about 124 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          64 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.6 seconds inserted 10000000 Rows, about 349K ops/s
    storage size: 17162331545 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1351618491


##########
fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java:
##########
@@ -2047,51 +2064,41 @@ private void httpStreamPutImpl(TStreamLoadPutRequest request, TStreamLoadPutResu
             checkPasswordAndPrivs(cluster, request.getUser(), request.getPasswd(), request.getDb(), request.getTbl(),
                     request.getUserIp(), PrivPredicate.LOAD);
         }
+        ctx.setThreadLocalInfo();
         ctx.setEnv(Env.getCurrentEnv());
         ctx.setQueryId(request.getLoadId());
         ctx.setCluster(SystemInfoService.DEFAULT_CLUSTER);
         ctx.setCurrentUserIdentity(UserIdentity.ROOT);

Review Comment:
   why use root???
   Should use the actual username because we need check if current user has the related privilege!!!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762785224

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762814534

   TeamCity be ut coverage result:
    Function Coverage: 36.27% (8144/22454) 
    Line Coverage: 28.49% (65282/229119)
    Region Coverage: 27.18% (33814/124409)
    Branch Coverage: 23.98% (17251/71944)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/8821e80bafbcc5726f5eb6e3abffbeb65bd9ee43_8821e80bafbcc5726f5eb6e3abffbeb65bd9ee43/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs closed pull request #25049: [Enhancement](Load) Nereids supports http_stream
URL: https://github.com/apache/doris/pull/25049


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1754849478

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1351620362


##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2735,6 +2737,81 @@ private List<ResultRow> convertResultBatchToResultRows(TResultBatch batch) {
         return resultRows;
     }
 
+    public void generateStreamLoadNereidsPlan(TUniqueId queryId) throws Exception {
+        LOG.info("TUniqueId: {} generate stream load plan", queryId);
+        context.setQueryId(queryId);
+        context.setStartTime();
+        profile.getSummaryProfile().setQueryBeginTime();
+        context.setStmtId(STMT_ID_GENERATOR.incrementAndGet());
+
+        parseByNereids();
+        Preconditions.checkState(parsedStmt instanceof LogicalPlanAdapter,
+                "Nereids only process LogicalPlanAdapter, but parsedStmt is " + parsedStmt.getClass().getName());
+        context.getState().setNereids(true);
+        InsertIntoTableCommand insert = (InsertIntoTableCommand) ((LogicalPlanAdapter) parsedStmt).getLogicalPlan();
+        try {
+            insert.initPlan(context, this);
+            if (context.getTxnEntry() == null) {
+                TransactionEntry transactionEntry =
+                        new TransactionEntry(new TTxnParams().setTxnId(insert.getTxn().getTxnId()),
+                        insert.getTxn().getDatabase(), insert.getTxn().getTable());
+                transactionEntry.setLabel(insert.getTxn().getLabelName());
+                context.setTxnEntry(transactionEntry);
+            }
+            // Determine if it can be converted to TVFScanNode
+            PlanNode planRoot = planner.getFragments().get(0).getPlanRoot();
+            Preconditions.checkState(planRoot instanceof TVFScanNode,
+                    "Nereids' planNode cannot be converted to " + planRoot.getClass().getName());
+        } catch (QueryStateException e) {
+            LOG.debug("Command(" + originStmt.originStmt + ") process failed.", e);
+            context.setState(e.getQueryState());
+            throw new NereidsException("Command(" + originStmt.originStmt + ") process failed",
+                    new AnalysisException(e.getMessage(), e));
+        } catch (UserException e) {
+            // Return message to info client what happened.
+            LOG.debug("Command(" + originStmt.originStmt + ") process failed.", e);
+            context.getState().setError(e.getMysqlErrorCode(), e.getMessage());
+            throw new NereidsException("Command (" + originStmt.originStmt + ") process failed",
+                    new AnalysisException(e.getMessage(), e));
+        } catch (Exception e) {
+            // Maybe our bug
+            LOG.debug("Command (" + originStmt.originStmt + ") process failed.", e);
+            context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, e.getMessage());
+            throw new NereidsException("Command (" + originStmt.originStmt + ") process failed.",
+                    new AnalysisException(e.getMessage(), e));
+        }
+    }
+
+    public void getStreamLoadPlan(TUniqueId queryId) throws Exception {
+        try {
+            generateStreamLoadNereidsPlan(queryId);
+        } catch (NereidsException | ParseException e) {
+            if (context.getMinidump() != null) {
+                MinidumpUtils.saveMinidumpString(context.getMinidump(), DebugUtil.printId(context.queryId()));
+            }
+            // try to fall back to legacy planner
+            LOG.debug("nereids cannot process statement\n" + originStmt.originStmt
+                    + "\n because of " + e.getMessage(), e);
+            if (e instanceof NereidsException && !context.getSessionVariable().enableFallbackToOriginalPlanner) {
+                LOG.warn("Analyze failed. {}", context.getQueryIdentifier(), e);
+                throw ((NereidsException) e).getException();
+            }
+            LOG.debug("fall back to legacy planner on statement:\n{}", originStmt.originStmt);
+            // Due to executing Nereids, it needs to be reset
+            parsedStmt = null;
+            planner = null;
+            context.getState().setNereids(false);
+            context.setTxnEntry(null);
+            context.setQueryId(queryId);
+            context.setStartTime();
+            profile.getSummaryProfile().setQueryBeginTime();

Review Comment:
   context.setStartTime();
               profile.getSummaryProfile().setQueryBeginTime();
   useless



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1812120427

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1754848388

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1352761542


##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2736,6 +2738,77 @@ private List<ResultRow> convertResultBatchToResultRows(TResultBatch batch) {
         return resultRows;
     }
 
+    public void generateStreamLoadNereidsPlan(TUniqueId queryId) throws Exception {
+        LOG.info("TUniqueId: {} generate stream load plan", queryId);

Review Comment:
   make some iconic log



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -136,15 +145,15 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
 
         if (isOverwrite) {
             dealOverwrite(ctx, executor, physicalOlapTableSink);
-            return;
+            return null;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762887035

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1352275870


##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2736,6 +2738,77 @@ private List<ResultRow> convertResultBatchToResultRows(TResultBatch batch) {
         return resultRows;
     }
 
+    public void generateStreamLoadNereidsPlan(TUniqueId queryId) throws Exception {

Review Comment:
   private



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -166,7 +169,18 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
         state.addTableIndexes(physicalOlapTableSink.getTargetTable());
 
         executor.setProfileType(ProfileType.LOAD);
-
+        if (ctx.getTxnEntry() == null) {
+            TransactionEntry transactionEntry = new TransactionEntry(new TTxnParams().setTxnId(txn.getTxnId()),
+                    physicalOlapTableSink.getDatabase(), physicalOlapTableSink.getTargetTable());
+            transactionEntry.setLabel(label);
+            ctx.setTxnEntry(transactionEntry);
+        }
+        executor.setPlanner(planner);
+        TVFScanNode planRoot = (TVFScanNode) planner.getFragments().get(0).getPlanRoot();
+        if ("HttpStreamTableValuedFunction".equals(planRoot.getTvfName())) {
+            LOG.info("exec http_stream by nereids, query id: {}, txn id: {}", ctx.queryId(), txn.getTxnId());
+            return;
+        }

Review Comment:
   cannot understand this code block, please explain it



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -136,15 +145,15 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
 
         if (isOverwrite) {
             dealOverwrite(ctx, executor, physicalOlapTableSink);
-            return;
+            return null;

Review Comment:
   so stream load could not do insert overwrite?



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -166,7 +175,13 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
         state.addTableIndexes(physicalOlapTableSink.getTargetTable());
 
         executor.setProfileType(ProfileType.LOAD);
+        executor.setPlanner(planner);

Review Comment:
   why set planner here?



##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2736,6 +2738,77 @@ private List<ResultRow> convertResultBatchToResultRows(TResultBatch batch) {
         return resultRows;
     }
 
+    public void generateStreamLoadNereidsPlan(TUniqueId queryId) throws Exception {
+        LOG.info("TUniqueId: {} generate stream load plan", queryId);

Review Comment:
   debug?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762893283

   TeamCity be ut coverage result:
    Function Coverage: 36.28% (8147/22456) 
    Line Coverage: 28.51% (65322/229131)
    Region Coverage: 27.20% (33838/124419)
    Branch Coverage: 24.00% (17269/71952)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/2f18595384af4cba5e78a180e45d8534deace314_2f18595384af4cba5e78a180e45d8534deace314/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1754849401

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1781183519

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1750302140

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1352759265


##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2736,6 +2738,77 @@ private List<ResultRow> convertResultBatchToResultRows(TResultBatch batch) {
         return resultRows;
     }
 
+    public void generateStreamLoadNereidsPlan(TUniqueId queryId) throws Exception {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1351610768


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -100,8 +101,14 @@ public NereidsPlanner getPlanner() {
         return planner;
     }
 
-    @Override
-    public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+    public Transaction getTxn() {
+        return txn;
+    }
+
+    /**
+     * init plan

Review Comment:
   init plan is too simple, the comment is same to the method name, nobody could know the reason.
   please add more comment to explain why we need modify this. Then other people could modify it or refactor the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #25049:
URL: https://github.com/apache/doris/pull/25049#discussion_r1351616237


##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2735,6 +2737,81 @@ private List<ResultRow> convertResultBatchToResultRows(TResultBatch batch) {
         return resultRows;
     }
 
+    public void generateStreamLoadNereidsPlan(TUniqueId queryId) throws Exception {
+        LOG.info("TUniqueId: {} generate stream load plan", queryId);
+        context.setQueryId(queryId);
+        context.setStartTime();

Review Comment:
   context.setStartTime();
   profile.getSummaryProfile().setQueryBeginTime();
   Is these useful?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762920651

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "zzzzzzzs (via GitHub)" <gi...@apache.org>.
zzzzzzzs commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1762783523

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Enhancement](Load) Nereids supports http_stream [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25049:
URL: https://github.com/apache/doris/pull/25049#issuecomment-1784023117

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org