You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2023/10/18 07:34:30 UTC

[PR] (enhencement)(RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

ByteYue opened a new pull request, #25578:
URL: https://github.com/apache/doris/pull/25578

   ## Proposed changes
   
   Issue Number: close #xxx
   
   The files of corresponding remote file system shouldn't be deleted when the rowset writers's dtor is called, instead they should be deleted asynchronously by the remote file system. Otherwise the dtor would do S3 io which might cost a lot of time.
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1768634950

   TeamCity be ut coverage result:
    Function Coverage: 37.02% (8282/22371) 
    Line Coverage: 29.08% (66426/228461)
    Region Coverage: 27.73% (34490/124384)
    Branch Coverage: 24.33% (17521/72012)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/0140eacd3c19354dcf04d75e3aafcf0ca2ddbeec_0140eacd3c19354dcf04d75e3aafcf0ca2ddbeec/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #25578:
URL: https://github.com/apache/doris/pull/25578


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1767923600

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.66 seconds
    stream load tsv:          556 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          32 seconds loaded 861443392 Bytes, about 25 MB/s
    insert into select:          28.7 seconds inserted 10000000 Rows, about 348K ops/s
    storage size: 17162201175 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1767930473

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1767892702

   TeamCity be ut coverage result:
    Function Coverage: 37.01% (8279/22369) 
    Line Coverage: 29.08% (66402/228337)
    Region Coverage: 27.73% (34464/124263)
    Branch Coverage: 24.35% (17510/71918)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/8afa1d00f455d4f85a11e70b67350abf4690bf03_8afa1d00f455d4f85a11e70b67350abf4690bf03/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "JackDrogon (via GitHub)" <gi...@apache.org>.
JackDrogon commented on code in PR #25578:
URL: https://github.com/apache/doris/pull/25578#discussion_r1363859753


##########
be/src/olap/rowset/vertical_beta_rowset_writer.cpp:
##########
@@ -56,8 +55,7 @@ VerticalBetaRowsetWriter::~VerticalBetaRowsetWriter() {
             // Even if an error is encountered, these files that have not been cleaned up
             // will be cleaned up by the GC background. So here we only print the error
             // message when we encounter an error.
-            WARN_IF_ERROR(fs->delete_file(path),
-                          strings::Substitute("Failed to delete file=$0", path));
+            WARN_IF_ERROR(fs->delete_file(path), fmt::format("Failed to delete file={}", path));

Review Comment:
   add `#include <fmt/format.h>`



##########
be/src/olap/rowset/beta_rowset_writer.cpp:
##########
@@ -94,7 +93,7 @@ BetaRowsetWriter::~BetaRowsetWriter() {
             // will be cleaned up by the GC background. So here we only print the error
             // message when we encounter an error.
             WARN_IF_ERROR(fs->delete_file(seg_path),
-                          strings::Substitute("Failed to delete file=$0", seg_path));
+                          fmt::format("Failed to delete file={}", seg_path));

Review Comment:
   add `#include <fmt/format.h>`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1768472862

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (enhencement)(RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1767843973

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1768493428

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1768555332

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1768732788

   (From new machine)TeamCity pipeline, clickbench performance test result:
    the sum of best hot time: 45.95 seconds
    stream load tsv:          554 seconds loaded 74807831229 Bytes, about 128 MB/s
    stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
    stream load orc:          65 seconds loaded 1101869774 Bytes, about 16 MB/s
    stream load parquet:          33 seconds loaded 861443392 Bytes, about 24 MB/s
    insert into select:          28.9 seconds inserted 10000000 Rows, about 346K ops/s
    storage size: 17162338434 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1767930381

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #25578:
URL: https://github.com/apache/doris/pull/25578#discussion_r1363904083


##########
be/src/olap/rowset/beta_rowset_writer.cpp:
##########
@@ -20,6 +20,7 @@
 #include <assert.h>
 // IWYU pragma: no_include <bthread/errno.h>
 #include <errno.h> // IWYU pragma: keep
+#include <fmt/format.h>

Review Comment:
   warning: inclusion of deprecated C++ header 'stdio.h'; consider using 'cstdio' instead [modernize-deprecated-headers]
   
   ```suggestion
   #include <cstdio>
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [enhencement](RowsetWriter) Don't delete files when beta rowset writer destructed [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25578:
URL: https://github.com/apache/doris/pull/25578#issuecomment-1767868634

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org