You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2023/06/04 04:37:36 UTC
[GitHub] [doris] HHoflittlefish777 opened a new pull request, #20404: [Fix]fix lazy open commit info lose
HHoflittlefish777 opened a new pull request, #20404:
URL: https://github.com/apache/doris/pull/20404
## Proposed changes
**problem**
when open enable_lazy_open_partition config, there is a possibility that the open partition rpc will perform the open operation after the txn commit, which will cause the commit to fail and the load fail.
**What is this pr task**
- Therefore, to solve this problem,When sending the last RPC to obtain the commit information, it is necessary to wait for all lazy opens to end.
- a more comprehensive case has been added for lazy open testing.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20404: [Fix]fix lazy open commit info lose
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20404:
URL: https://github.com/apache/doris/pull/20404#issuecomment-1575479019
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #20404: [Fix]fix lazy open commit info lose
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #20404:
URL: https://github.com/apache/doris/pull/20404#discussion_r1216280175
##########
be/src/vec/sink/vtablet_sink.h:
##########
@@ -302,6 +304,7 @@ class VNodeChannel {
int _rpc_timeout_ms = 60000;
int64_t _next_packet_seq = 0;
MonotonicStopWatch _timeout_watch;
+ MonotonicStopWatch _lazy_open_timeout_watch;
Review Comment:
use local var instead of member?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #20404: [Fix]fix lazy open commit info lose
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #20404:
URL: https://github.com/apache/doris/pull/20404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20404: [Fix]fix lazy open commit info lose
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20404:
URL: https://github.com/apache/doris/pull/20404#issuecomment-1575437037
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HHoflittlefish777 commented on pull request #20404: [Fix]fix lazy open commit info lose
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #20404:
URL: https://github.com/apache/doris/pull/20404#issuecomment-1575436274
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20404: [Fix]fix lazy open commit info lose
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20404:
URL: https://github.com/apache/doris/pull/20404#issuecomment-1575383911
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #20404: [Fix]fix lazy open commit info lose
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #20404:
URL: https://github.com/apache/doris/pull/20404#discussion_r1216277572
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -320,6 +320,9 @@ Status VNodeChannel::init(RuntimeState* state) {
_rpc_timeout_ms = state->execution_timeout() * 1000;
_timeout_watch.start();
+ if (config::enable_lazy_open_partition) {
+ _lazy_open_timeout_watch.start();
+ }
Review Comment:
no need to start stop watch here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #20404: [Fix]fix lazy open commit info lose
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #20404:
URL: https://github.com/apache/doris/pull/20404#discussion_r1216277793
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -516,7 +519,7 @@ Status VNodeChannel::open_wait() {
}
void VNodeChannel::open_partition(int64_t partition_id) {
- _timeout_watch.reset();
+ _lazy_open_timeout_watch.reset();
Review Comment:
use stop watch as local var and start here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] freemandealer commented on a diff in pull request #20404: [Fix]fix lazy open commit info lose
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #20404:
URL: https://github.com/apache/doris/pull/20404#discussion_r1216191454
##########
regression-test/suites/rollup_p0/test_materialized_view_lazy_open.groovy:
##########
@@ -70,6 +75,58 @@ suite("test_materialized_view_lazy_open", "rollup") {
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.buckets" = "32",
+ "light_schema_change" = "false",
+ "replication_num"="1"
+ );
+ """
+
+ sql "DROP TABLE IF EXISTS ${tbName3}"
+ sql """
+ CREATE TABLE IF NOT EXISTS ${tbName4}(
+ k1 DATE,
+ k2 DECIMAL(10, 2),
+ k3 CHAR(10),
+ k4 INT NOT NULL
+ )
+ DUPLICATE KEY(k1, k2)
+ PARTITION BY RANGE(k1)
+ (
+ PARTITION p1 VALUES LESS THAN ("2000-01-01"),
+ PARTITION p2 VALUES LESS THAN ("2010-01-01"),
+ PARTITION p3 VALUES LESS THAN ("2020-01-01")
+ )
+ DISTRIBUTED BY HASH(k1) BUCKETS 32
+
+ properties(
+ "light_schema_change" = "true",
Review Comment:
Even though we enable schema change here, we have not triggered schema change yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20404: [Fix]fix lazy open commit info lose
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20404:
URL: https://github.com/apache/doris/pull/20404#issuecomment-1575437432
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HHoflittlefish777 commented on pull request #20404: [Fix]fix lazy open commit info lose
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #20404:
URL: https://github.com/apache/doris/pull/20404#issuecomment-1575383335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org