You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "924060929 (via GitHub)" <gi...@apache.org> on 2026/10/10 10:19:01 UTC

[PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation [doris]

924060929 opened a new pull request, #68881:
URL: https://github.com/apache/doris/pull/68881

   ### What problem does this PR solve?
   
   Issue Number: N/A
   
   Related PR: #68849 (backport), #35210, #53486 (introducing changes)
   
   Problem Summary:
   
   Backport #68849 to branch-4.1. Two concurrent `CREATE TABLE IF NOT EXISTS ... AS SELECT` statements can both observe an absent table. Only one registers its table, but the losing creator does not propagate the atomic registration result to CTAS. Both statements then insert into the winning table, producing duplicate rows.
   
   Propagate the existing-table result under the database write lock so the losing CTAS skips INSERT. After releasing that lock, recycle the losing cloud indexes through the existing FE `dropMaterializedIndex` API; report cleanup errors rather than returning success. Preserve the winning table and existing FE cleanup.
   
   Resolve the backport conflicts using branch-4.1's `getIndexIdList()` API and retain only the tests relevant to this fix. This changes four FE files only; no MetaService, recycler, or protobuf changes are included.
   
   ### Release note
   
   Fix duplicate rows when concurrent CREATE TABLE IF NOT EXISTS AS SELECT statements target the same internal table, and recycle cloud indexes allocated by the losing creator.
   
   ### Check List (For Author)
   
   - Test: Unit Test; FE Checkstyle; git diff --check.
   - Behavior changed: Yes. The losing CTAS skips INSERT; cloud cleanup failure is reported.
   - Does this need documentation: No.
   
   Validation on branch-4.1 (JDK 17):
   
   - `bash run-fe-ut.sh --run org.apache.doris.catalog.CreateTableTest,org.apache.doris.nereids.trees.plans.commands.CreateTableCommandTest,org.apache.doris.cloud.datasource.CloudInternalCatalogTest`: 23 tests passed, 0 failures/errors/skips (18 + 3 + 2).
   - `cd fe && mvn checkstyle:check -pl fe-core -Dmaven.build.cache.enabled=false`: BUILD SUCCESS.
   - `git diff --check`: passed.
   
   The cloud RPC boundary is mocked. No live MetaService deployment was tested for this backport. Earlier deployed CTAS validation is documented in #68849.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation #68849 [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6097709447

   # FE UT Coverage Report
   Increment line coverage `  100.00% (6/6)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68881_7c61950830304152bca6cda5d3c4f46af3d829ff/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68881_7c61950830304152bca6cda5d3c4f46af3d829ff/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation #68849 [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #68881:
URL: https://github.com/apache/doris/pull/68881


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation #68849 [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6099026572

   # FE Regression Coverage Report
   Increment line coverage `  7.69% (1/13)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/68881_7c61950830304152bca6cda5d3c4f46af3d829ff_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/68881_7c61950830304152bca6cda5d3c4f46af3d829ff_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6096498270

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation #68849 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68881:
URL: https://github.com/apache/doris/pull/68881#discussion_r4237316461


##########
fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java:
##########
@@ -1033,6 +1033,11 @@ public void removeSchemaChangeJob(long jobId, long dbId, long tableId, long inde
         }
     }
 
+    @Override
+    public void onCreateTableConflict(long dbId, OlapTable table) throws DdlException {
+        dropMaterializedIndex(dbId, table.getId(), table.getIndexIdList(), true);

Review Comment:
   [P2] Recycle the table version for a losing table with no partitions. `AUTO PARTITION BY RANGE (...) ()` can enter this race with an empty partition list, but `commit_index` still creates the loser's table-version key. This hook queues only its indexes for recycling. Both cloud version cleanup paths discover tables through partition keys, so after a successful drop there is no partition key from which to find and remove that version key. Each such conflict leaves permanent cloud metadata; explicitly clean up the empty table's version or give the recycler another discovery path.



##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -3322,6 +3328,11 @@ private boolean createOlapTable(Database db, CreateTableInfo createTableInfo) th
                 db.writeUnlock();
             }
             try {
+                if (result.second) {
+                    // Cloud indexes have already been committed. Recycle them outside the DB lock,
+                    // including when a CREATE without IF NOT EXISTS must report a duplicate error.
+                    onCreateTableConflict(db.getId(), olapTable);

Review Comment:
   [P2] Preserve a cleanup path when dropping the losing cloud indexes fails. In this registration race, `commit_index` has already removed the prepared recycle keys. If `onCreateTableConflict` gets a non-OK `dropIndex` response or exhausts RPC retries, the catch below calls `dropTable(db, loserId, ...)`, but that table was never registered, so the lookup fails and the recycler never sees a dropped-index marker. The loser's committed index/tablet metadata can remain indefinitely. Persist the loser IDs for retry or otherwise make the cleanup recoverable after a failed RPC.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation #68849 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6118650485

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6096499779

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation [doris]

Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6096498995

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] branch-4.1: [fix](fe) Skip insert after losing concurrent CTAS table creation #68849 [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68881:
URL: https://github.com/apache/doris/pull/68881#issuecomment-6118650118

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org