You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2026/10/08 03:23:41 UTC
(doris) 10/35: [fix](regression) Backport regression stability fixes to branch-4.1 (#68449) (#68618)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a8f3b9dd7dc6dae943247fda7aea588929545e84
Author: chunping <zh...@selectdb.com>
AuthorDate: Tue Sep 29 21:36:07 2026 +0800
[fix](regression) Backport regression stability fixes to branch-4.1 (#68449) (#68618)
### What problem does this PR solve?
Issue Number: None
Related PR: #68449
Problem Summary:
Backport #68449 (`50a39819e185f54fbb3a82e67182b1ec8f78433b`) to
`branch-4.1`. The changes stabilize regression clients and orchestration
for TLS/mTLS, Docker isolation, multi-cluster state cleanup, cloud
cache/compaction, asynchronous metadata visibility, and delete-bitmap
restart/concurrency tests. Doris Compose uses exact container-name
matching and retries concurrent removal races.
Resolved 27 conflict paths with these branch adaptations:
- Keep the `nereids_p0` suite paths, generated database names, planner
settings, existing SHOW columns and 4.1-only test logic.
- Preserve configured-master JDBC/HTTP endpoints in the
insert-visible-timeout suite, including mapped ports from #65340.
- Keep the routine-load WHERE expectation `(k00 = 8)`: 4.1's
`PlanUtils.ExpressionToExpr` stores unquoted slot labels, unlike master.
Retain the source topic fix, diagnostic fields and independent polling
counters.
- Keep `nereids_p0/hbo` in the non-concurrent exclusions while excluding
all `cloud_p0` suites as intended by the source fix.
- Combine the shared TLS/compaction helpers with 4.1's existing
framework behavior; do not import unrelated scenarios present only in
the source commit's parent.
- Omit `test_partition_cloud_add_build_index_validation.groovy`: the
partition index-format rollout feature from #66475 and its required
properties/configuration are absent from 4.1.
- Reuse the original PR's result files without manually creating
expected output; retain the deterministic variant ordering fix.
The resulting backport changes 350 files; no FE implementation,
protocol, or storage-format change is added.
### Release note
None
### Check List (For Author)
- Test:
- [x] Regression test on an existing 4.1.4 cloud cluster (1 FE / 3 BE):
127 passed, 3 failed, 186 skipped across all 316 changed suites.
- Retained failures: `test_create_view_nereids` (SHOW VIEW column-name
quoting mismatch), `test_streaming_insert_job_priv` (one successful task
and one pending task after the 300-second wait), and
`test_load_channel_profile` (S3 TVF query reports unknown column `c1`).
Their failed assertions/queries are unchanged by this backport. Root
causes are not established by this run.
- Python support is disabled: 19 attempts hit that prerequisite and 23
more suites were skipped. One JDBC fixture was unavailable, and the
single-BE Java static-load case returned early on this three-BE cluster;
these are counted as skipped, not passes.
- Other skipped suites require dedicated Docker/multi-compute-group
fixtures, restarts, TLS/external services, global configuration/fault
injection, or stress capacity. Four packed-file injection suites were
excluded before launch.
- Arrow Flight authentication passed after correcting only the
task-local endpoint configuration. Three small-table TTL/row-store
compaction follow-ups also passed.
- The failed streaming job and its test user were cleaned up. All FE/BE
nodes remained alive and HTTP health returned 200 after execution.
- [x] Unit Test: `run-regression-test.sh --compile` completed on the
remote validation host; all 35 regression-framework tests passed (0
failures, 0 errors, 0 skips).
- [x] Manual test:
- Diff whitespace and conflict-marker checks passed.
- All 340 changed existing Groovy files passed conversion-phase parsing
with Groovy 4.0.19 / JDK 17; both changed Python files passed
`py_compile`.
- `build-support/check-format.sh` passed with clang-format 16.0.6 (4,400
C/C++ files).
- Compared conflict resolutions with the source patch and branch-4.1
implementations, including RPC/debug-point dependencies and routine-load
expression serialization.
- BE/FE module builds, deployment and clang-tidy were not run, per the
requested existing-cluster validation scope. The live enterprise cluster
is not a build of this backport, so BE debug-point additions and
excluded scenarios still require CI.
- Behavior changed: Yes. Regression framework/test behavior and Doris
Compose discovery match the applicable source fixes; BE additions are
gated debug points.
- Does this need documentation: No.
<details>
<summary>Backport self-review checkpoints</summary>
| Checkpoint | Conclusion |
| --- | --- |
| Goal and tests | Applicable source fixes preserved; syntax/static
checks and 35 framework unit tests pass. Existing-cluster suite outcomes
and remaining CI scope are listed above. |
| Scope | Source patch plus documented branch adaptations; unsupported
feature suite omitted. |
| Concurrency and locking | Existing production lock order retained;
debug-only post-calculation barrier releases the rowset lock before
waiting. Suite worker errors propagate to their owners. |
| Lifecycle | Source connection/process cleanup and shared-state
restoration retained. |
| Configuration | Existing TLS and debug configuration used; referenced
cloud sync settings exist in 4.1. |
| Compatibility | Target RPC/HTTP/debug APIs verified; branch endpoint
mappings and result schemas retained. No wire/storage format changes. |
| Parallel paths | Relevant plaintext/TLS and local/cloud helper paths
retained. |
| Conditional checks | Source bounded polling and negative assertions
retained; branch adaptations checked against implementation. |
| Test coverage | Source coverage retained; 35 framework tests and 127
existing-cluster suites passed, with 3 retained failures and 186
documented skips. |
| Expected results | Source result files reused; no hand-written golden
output. |
| Observability | Source timeout/last-state diagnostics and
token/tablet/transaction debug logs retained. |
| Persistence and failover | No production persistence or replay change.
|
| Writes and atomicity | Production write behavior unchanged outside
enabled test debug points. |
| FE/BE propagation | Existing target sync-load RPC and metadata
endpoints support new helpers. |
| Performance | Production additions are gated debug checks; regression
polling is bounded. |
| Other findings | Focused framework/BE, TLS-suite and cloud-suite
reviews found no unresolved backport-specific blocker. |
</details>
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases and CI results
- [ ] Confirm document
- [ ] Add branch pick label
---
.../cloud/cloud_engine_calc_delete_bitmap_task.cpp | 26 +
be/src/cloud/cloud_meta_mgr.cpp | 10 +
docker/runtime/doris-compose/utils.py | 43 +-
.../cache/ttl/st04_alter_ttl_n_to_0_runtime.out | 7 +-
.../cache/ttl/st06_warmup_ttl_type_assert.out | 4 +-
.../data/cloud_p0/cache/ttl/test_show_ttl.out | 16 -
.../test_double_write_when_schema_change.out | 4 +-
.../org/apache/doris/regression/Config.groovy | 139 ++--
.../doris/regression/action/ProfileAction.groovy | 38 +-
.../doris/regression/suite/ScriptContext.groovy | 11 +
.../org/apache/doris/regression/suite/Suite.groovy | 231 ++++++-
.../doris/regression/suite/SuiteContext.groovy | 55 +-
.../apache/doris/regression/util/DebugPoint.groovy | 12 +-
.../org/apache/doris/regression/util/Http.groovy | 32 +-
.../regression/util/MultiClusterStateGuard.groovy | 195 ++++++
.../doris/regression/util/MySqlClient.groovy | 70 ++
.../doris/regression/util/RowsetMetaUtils.groovy | 79 +++
.../regression/util/WarmupMetricsUtils.groovy | 96 ++-
.../regression/ConfigArrowFlightSqlUrlTest.groovy | 61 ++
.../doris/regression/ConfigRootPasswordTest.groovy | 61 ++
.../suite/CompactionPluginCompatibilityTest.groovy | 154 +++++
.../regression/suite/SuiteSshExecTest.groovy} | 24 +-
.../regression/util/HttpAuthenticationTest.groovy | 72 ++
.../util/MultiClusterStateGuardTest.groovy | 256 ++++++++
.../doris/regression/util/MySqlClientTest.groovy | 96 +++
.../regression/util/RowsetMetaUtilsTest.groovy | 81 +++
.../nonConcurrent/conf/regression-conf.groovy | 2 +-
.../plugins/cloud_multi_cluster_state.groovy | 136 ++++
regression-test/plugins/plugin_compaction.groovy | 134 ++--
.../plugins/plugin_curl_requester.groovy | 3 +-
regression-test/plugins/plugin_index_change.groovy | 5 +-
.../suites/account_p0/test_auth_show.groovy | 4 +-
.../suites/account_p0/test_grant_priv.groovy | 10 +-
.../account_p0/test_grant_priv_resource.groovy | 4 +-
.../account_p0/test_grant_priv_workload.groovy | 4 +-
.../account_p0/test_nereids_authentication.groovy | 14 +-
.../account_p0/test_nereids_row_policy.groovy | 4 +-
.../suites/account_p0/test_set_password.groovy | 6 +-
.../suites/account_p0/test_view_row_policy.groovy | 6 +-
.../ann_topn_small_candidate_fallback.groovy | 21 +-
.../arrow_flight_sql_p0/test_auth_remote_ip.groovy | 3 +-
.../suites/audit/test_audit_log_queue_time.groovy | 212 +++---
.../auth_call/test_dml_select_udf_auth.groovy | 1 +
.../test_dml_stream_load_be_auth_docker.groovy | 49 +-
.../auth_p0/test_role_mapping_system_table.groovy | 2 +-
.../cloud_p0/auth/test_set_default_cluster.groovy | 6 +-
.../balance/test_cross_cg_bvar_metrics.groovy | 27 +-
.../multi_cluster/read_write/sync_insert.groovy | 330 ++++++----
.../read_write/test_multi_stale_rowset.groovy | 70 +-
.../cluster/test_warm_up_cluster_batch.groovy | 409 +++++++-----
...est_warm_up_cluster_event_cancel_expired.groovy | 92 ++-
.../cluster/test_warm_up_compute_group.groovy | 209 +++---
.../warm_up/table/test_warm_up_partition.groovy | 399 +++++++-----
.../test_warm_up_same_table_multi_times.groovy | 43 +-
.../warm_up/table/test_warm_up_table.groovy | 200 ++++--
.../warm_up/table/test_warm_up_tables.groovy | 486 ++++++++------
.../test_clear_file_cache_on_load_failure.groovy | 121 ++--
.../cloud_p0/cache/test_file_cache_info.groovy | 318 +++++----
.../suites/cloud_p0/cache/ttl/alter_ttl_2.groovy | 231 +++----
.../suites/cloud_p0/cache/ttl/alter_ttl_4.groovy | 489 ++++++++------
.../cloud_p0/cache/ttl/alter_ttl_random.groovy | 21 +-
.../cloud_p0/cache/ttl/alter_ttl_seconds.groovy | 170 -----
.../cache/ttl/st04_alter_ttl_n_to_0_runtime.groovy | 233 ++++---
.../cache/ttl/st06_warmup_ttl_type_assert.groovy | 231 ++++---
.../suites/cloud_p0/cache/ttl/test_show_ttl.groovy | 57 ++
.../suites/cloud_p0/cache/ttl/test_show_ttl.sql | 40 --
.../suites/cloud_p0/cache/ttl/test_ttl.groovy | 433 +++++++-----
...st_file_cache_warmup_read_metrics_docker.groovy | 4 +-
.../cache/warm_up/test_warmup_table_docker.groovy | 6 +-
...test_cloud_alter_disable_auto_compaction.groovy | 8 +-
.../async_copy_into/async_load.groovy | 10 +-
.../multi_cluster/copy_into/sync_laod.groovy | 124 ++--
.../cloud_p0/multi_cluster/default_cluster.groovy | 38 +-
.../multi_cluster_s3_load/load.groovy | 66 +-
.../cloud_p0/multi_cluster/rename_cluster.groovy | 10 +-
.../routine_load/test_routine_load.groovy | 6 +-
...test_materialized_view_with_drop_cluster.groovy | 8 +-
...est_materialized_view_with_readd_cluster.groovy | 8 +-
.../test_rollup_with_drop_cluster.groovy | 8 +-
.../test_rollup_with_readd_cluster.groovy | 8 +-
.../test_schema_change_with_drop_cluster.groovy | 8 +-
.../test_schema_change_with_readd_cluster.groovy | 8 +-
.../smooth_upgrade/smooth_upgrade.groovy | 10 +
.../multi_cluster/stream_load/stream_load.groovy | 67 +-
.../stream_load/stream_load_2pc.groovy | 3 +
.../stream_load/stream_load_lb.groovy | 12 +-
.../cloud_p0/multi_cluster/table_rebalance.groovy | 30 +-
.../test_apsaradb_internal_stage.groovy | 31 +-
.../multi_cluster/test_cloud_cluster.groovy | 19 +-
.../multi_cluster/test_drop_cluster.groovy | 98 +--
.../cloud_p0/multi_cluster/test_drop_userid.groovy | 2 +-
.../test_group_commit_multi_cluster.groovy | 12 +-
.../multi_cluster/test_overdue_instance.groovy | 54 +-
.../suites/cloud_p0/multi_cluster/test_tvf.groovy | 11 +-
.../default_vcg_auto_failover.groovy | 19 +-
.../use_vcg_read_write_routine_load.groovy | 722 +++++++++++----------
.../cloud_p0/snapshot/test_snapshot_command.groovy | 12 +-
.../test_clean_tablet_when_drop_force_table.groovy | 144 +++-
.../test_clean_tablet_when_rebalance.groovy | 109 ++--
.../test_create_partition_idempotence.groovy | 3 +-
.../test_insert_overwrite_idempotence.groovy | 3 +-
.../test_schema_change_with_compaction10.groovy | 66 +-
.../test_schema_change_with_compaction11.groovy | 91 +--
.../test_schema_change_with_compaction5.groovy | 25 +-
.../test_schema_change_with_compaction6.groovy | 67 +-
.../test_schema_change_with_compaction9.groovy | 24 +-
.../test_inverted_index_sc_version1_race.groovy | 2 +-
.../test_sc_compaction_cross_v1_retry.groovy | 73 ++-
...c_compaction_cross_v1_stale_base_refresh.groovy | 30 +-
.../test_sc_compaction_optimization.groovy | 50 +-
...est_sc_compaction_optimization_with_load.groovy | 52 +-
.../compaction_width_array_column.groovy | 2 +-
.../compaction/test_admin_compact_tablet.groovy | 88 ++-
...paction_with_dup_key_max_file_size_limit.groovy | 148 ++---
.../test_compaction_uniq_keys_row_store.groovy | 4 +-
.../test_compaction_uniq_keys_row_store_ck.groovy | 4 +-
...test_compaction_update_big_delete_bitmap.groovy | 17 +-
.../test_compaction_with_visible_version.groovy | 3 +
.../test_config_prune_delete_sign.groovy | 129 ++--
...ilecache_with_base_compaction_thresthold.groovy | 150 +++--
.../test_mow_compact_multi_segments.groovy | 48 +-
...paction_agg_and_remove_pre_delete_bitmap.groovy | 17 +-
...test_mow_compaction_and_rowset_not_exist.groovy | 17 +-
.../test_mow_compaction_and_schema_change.groovy | 17 +-
.../test_mow_stale_rowset_delete_bitmap.groovy | 17 +-
.../test_non_mow_key_bounds_aggregation.groovy | 25 +-
.../test_create_view_nereids.groovy | 1 +
.../suites/ddl_p0/test_drop_view_nereids.groovy | 1 +
.../delete/batch-delete-manual.md.groovy | 12 +-
.../import-way/group-commit-manual.md.groovy | 11 +-
.../import/import-way/mysql-load-manual.md.groovy | 20 +-
.../update/unique-update-transaction.md.groovy | 6 +-
.../update/update-of-aggregate-model.md.groovy | 2 +-
.../update/update-of-unique-model.md.groovy | 2 +-
.../test_iceberg_arrow_flight_split_source.groovy | 3 +-
.../select/test_mysql_varbinary_with_udf.groovy | 1 +
.../test_cloud_full_compaction_do_lease.groovy | 142 ++--
.../test_cloud_mow_delete_bitmap_lock_case.groovy | 412 +++++++++---
.../test_cloud_mow_partial_update_retry.groovy | 99 ++-
..._cloud_mow_stale_resp_load_load_conflict.groovy | 105 ++-
.../test_delete_bitmap_lock_with_restart.groovy | 545 ++++++++--------
...est_cold_data_compaction_fault_injection.groovy | 146 +++--
...ion_clone_missing_rowset_fault_injection.groovy | 29 +-
.../test_incomplete_commit_info.groovy | 117 ++--
...st_skip_index_compaction_fault_injection.groovy | 90 +--
.../http_rest_api/get/test_schema_api.groovy | 4 +-
.../http_rest_api/post/test_query_stmt.groovy | 3 +-
.../insert_p0/insert_group_commit_into.groovy | 10 +-
.../mow_insert_with_partition_drop.groovy | 24 +-
.../test_insert_visible_timeout_return_mode.groovy | 16 +-
regression-test/suites/insert_p0/test_jdbc.groovy | 4 +-
.../transaction/test_publish_exception.groovy | 5 +
.../suites/insert_p0/transaction/txn_insert.groovy | 5 +-
.../operate_internal_schema.groovy | 2 +-
.../show_data/test_show_index_data_p2.groovy | 104 +--
.../collect/test_javaudf_groupcount.groovy | 1 +
.../collect/test_javaudf_murmurhash3.groovy | 1 +
.../collect/test_javaudf_sessionize.groovy | 2 +-
.../javaudf_p0/date/test_javaudf_adddays.groovy | 1 +
.../date/test_javaudf_addisoperioud.groovy | 1 +
.../javaudf_p0/date/test_javaudf_daydiff.groovy | 1 +
.../function_meta/function_query_test.groovy | 10 +-
.../suites/javaudf_p0/function_meta/load.groovy | 19 +-
.../json/test_javaudf_convertfromcamelcase.groovy | 1 +
.../json/test_javaudf_converttocamelcase.groovy | 1 +
regression-test/suites/javaudf_p0/null/load.groovy | 1 +
.../sanity/test_javaudf_assertequal.groovy | 1 +
.../sanity/test_javaudf_assertlessthan.groovy | 1 +
.../sanity/test_javaudf_assertudf.groovy | 1 +
.../javaudf_p0/sketch/test_javaudf_md5.groovy | 1 +
.../test_javaudaf_my_date_datetime.groovy | 1 +
.../test_javaudaf_mygroupconcat_string.groovy | 1 +
.../javaudf_p0/test_javaudaf_mysum_array.groovy | 1 +
.../javaudf_p0/test_javaudaf_mysum_decimal.groovy | 1 +
.../test_javaudaf_mysum_float_double.groovy | 1 +
.../javaudf_p0/test_javaudaf_mysum_int.groovy | 1 +
.../javaudf_p0/test_javaudaf_null_test.groovy | 1 +
.../javaudf_p0/test_javaudaf_return_map.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_agg_map.groovy | 1 +
.../javaudf_p0/test_javaudf_all_types.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_array.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_auth.groovy | 5 +-
.../suites/javaudf_p0/test_javaudf_boolean.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_case.groovy | 1 +
.../javaudf_p0/test_javaudf_const_test.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_date.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_decimal.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_float.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_int.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_map.groovy | 1 +
.../javaudf_p0/test_javaudf_multi_evaluate.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_no_input.groovy | 2 +-
.../suites/javaudf_p0/test_javaudf_override.groovy | 1 +
.../suites/javaudf_p0/test_javaudf_ret_map.groovy | 1 +
.../test_javaudf_static_load_test.groovy | 2 +-
.../suites/javaudf_p0/test_javaudf_string.groovy | 1 +
.../javaudf_p0/test_javaudf_struct_type.groovy | 1 +
.../javaudf_p0/test_javaudf_with_decimal.groovy | 1 +
.../javaudf_p0/test_javaudtf_all_types.groovy | 1 +
.../javaudf_p0/test_javaudtf_arrayint.groovy | 1 +
.../suites/javaudf_p0/test_javaudtf_decimal.groovy | 1 +
.../suites/javaudf_p0/test_javaudtf_double.groovy | 1 +
.../suites/javaudf_p0/test_javaudtf_int.groovy | 1 +
.../suites/javaudf_p0/test_javaudtf_string.groovy | 1 +
.../test_streaming_insert_job_priv.groovy | 8 +-
.../test_insert_statistic_after_fe_restart.groovy | 6 +-
.../test_routine_load_advanced_mapping.groovy | 56 +-
.../routine_load/test_routine_load_with_udf.groovy | 1 +
.../scripts/stream_load_redirect_chunked_e2e.py | 16 +-
.../test_packed_file_stream_load_case1.groovy | 2 +-
.../test_packed_file_stream_load_case2.groovy | 2 +-
.../test_packed_file_stream_load_case3.groovy | 2 +-
.../test_packed_file_stream_load_case4.groovy | 2 +-
.../test_packed_file_stream_load_case5.groovy | 11 +-
.../test_packed_file_stream_load_case6.groovy | 11 +-
.../test_packed_file_stream_load_case7.groovy | 11 +-
.../test_packed_file_stream_load_case8.groovy | 11 +-
.../load_p0/stream_load/test_sink_tolerate.groovy | 3 +-
.../load_p0/stream_load/test_stream_load.groovy | 38 +-
.../stream_load/test_stream_load_endpoint.groovy | 141 ++--
...test_stream_load_fe_redirect_chunked_e2e.groovy | 10 +-
.../stream_load/test_stream_load_with_udf.groovy | 1 +
.../suites/manager/test_manager_interface_3.groovy | 317 +++------
.../suites/mtmv_p0/test_expand_star_mtmv.groovy | 1 +
.../mysql_ssl_p0/test_mysql_connection.groovy | 66 +-
.../cache/parse_sql_from_sql_cache.groovy | 21 +-
.../insert_into_table/insert_auth.groovy | 6 +-
.../javaudf/collect/test_javaudf_groupcount.groovy | 1 +
.../collect/test_javaudf_murmurhash3.groovy | 1 +
.../javaudf/collect/test_javaudf_sessionize.groovy | 2 +-
.../javaudf/date/test_javaudf_adddays.groovy | 1 +
.../javaudf/date/test_javaudf_addisoperioud.groovy | 1 +
.../javaudf/date/test_javaudf_daydiff.groovy | 1 +
.../json/test_javaudf_convertfromcamelcase.groovy | 1 +
.../json/test_javaudf_converttocamelcase.groovy | 1 +
.../javaudf/sanity/test_javaudf_assertequal.groovy | 1 +
.../sanity/test_javaudf_assertlessthan.groovy | 1 +
.../javaudf/sanity/test_javaudf_assertudf.groovy | 1 +
.../javaudf/sketch/test_javaudf_md5.groovy | 1 +
.../javaudf/test_javaudaf_my_date_datetime.groovy | 1 +
.../test_javaudaf_mygroupconcat_string.groovy | 1 +
.../javaudf/test_javaudaf_mysum_decimal.groovy | 1 +
.../test_javaudaf_mysum_float_double.groovy | 1 +
.../javaudf/test_javaudaf_mysum_int.groovy | 1 +
.../javaudf/test_javaudaf_null_test.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_boolean.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_case.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_date.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_decimal.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_float.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_int.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_ip.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_null.groovy | 1 +
.../nereids_p0/javaudf/test_javaudf_string.groovy | 1 +
.../agg_state/test_outfile_agg_state.groovy | 2 +-
.../agg_state_array/test_outfile_agg_array.groovy | 2 +-
.../test_outfile_agg_state_bitmap.groovy | 2 +-
.../nereids_p0/outfile/hll/test_outfile_hll.groovy | 2 +-
.../test_outfile_quantile_state.groovy | 2 +-
.../show/test_nereids_show_functions.groovy | 13 +-
.../stats/col_stats/hot_value_analyze_sync.groovy | 2 +-
.../test_partition_create_tablet_rr.groovy | 1 +
.../auto_partition/sql/multi_thread_load.groovy | 30 +-
.../sql/stress_test_insert_into.groovy | 22 +-
.../diff_data/stress_test_diff_date_list.groovy | 16 +-
.../stress_test_high_concurrency_load.groovy | 19 +-
.../same_data/stress_test_same_date_range.groovy | 19 +-
.../stress_test_two_stream_load.groovy | 19 +-
.../suites/point_query_p0/test_point_query.groovy | 10 +-
.../point_query_p0/test_point_query_ck.groovy | 17 +-
.../test_point_query_partition.groovy | 6 +-
.../suites/point_query_p0/test_rowstore.groovy | 14 +-
.../suites/point_query_p0/test_rowstore_ck.groovy | 14 +-
.../test_python_raise_error_propagation.groovy | 3 +
.../pythonudaf_p0/test_pythonudaf_basic.groovy | 1 +
...st_pythonudaf_complex_aggregation_module.groovy | 1 +
..._pythonudaf_complex_state_objects_module.groovy | 1 +
.../pythonudaf_p0/test_pythonudaf_drop.groovy | 2 +
.../test_pythonudaf_forbidden_module.groovy | 1 +
.../test_pythonudaf_pkg_isolation.groovy | 2 +
.../pythonudaf_p0/test_pythonudaf_simple.groovy | 1 +
.../test_pythonudaf_window_advanced_module.groovy | 1 +
.../test_python_udaf_complex.groovy | 1 +
.../test_python_udf_business_logic.groovy | 1 +
.../test_python_udf_external_api.groovy | 3 +-
.../test_python_udf_nlp_chinese.groovy | 1 +
.../test_python_udtf_complex.groovy | 1 +
.../sanity/test_pythonudf_assertequal.groovy | 1 +
.../sanity/test_pythonudf_assertlessthan.groovy | 1 +
.../pythonudf_p0/test_pythonudf_array.groovy | 1 +
.../test_pythonudf_base_data_type.groovy | 1 +
.../pythonudf_p0/test_pythonudf_boolean.groovy | 1 +
.../test_pythonudf_complex_data_type.groovy | 1 +
.../suites/pythonudf_p0/test_pythonudf_drop.groovy | 2 +
.../test_pythonudf_file_protocol.groovy | 1 +
.../pythonudf_p0/test_pythonudf_float.groovy | 1 +
.../test_pythonudf_forbidden_module.groovy | 1 +
.../suites/pythonudf_p0/test_pythonudf_int.groovy | 1 +
.../suites/pythonudf_p0/test_pythonudf_map.groovy | 1 +
.../pythonudf_p0/test_pythonudf_module.groovy | 1 +
.../test_pythonudf_module_advanced.groovy | 1 +
.../test_pythonudf_module_scalar.groovy | 1 +
.../test_pythonudf_module_vector.groovy | 1 +
.../test_pythonudf_pkg_isolation.groovy | 2 +
.../pythonudf_p0/test_pythonudf_ret_map.groovy | 1 +
.../pythonudf_p0/test_pythonudf_string.groovy | 1 +
.../test_pythonudtf_basic_module.groovy | 1 +
.../test_pythonudtf_data_types_module.groovy | 1 +
.../pythonudtf_p0/test_pythonudtf_drop.groovy | 2 +
.../test_pythonudtf_edge_cases_module.groovy | 1 +
.../test_pythonudtf_exceptions_module.groovy | 1 +
.../test_pythonudtf_forbidden_module.groovy | 1 +
.../test_pythonudtf_io_patterns_module.groovy | 1 +
.../test_pythonudtf_pkg_isolation.groovy | 2 +
.../test_pythonudtf_sql_integration_module.groovy | 1 +
.../query_p0/cache/query_cache_with_context.groovy | 3 +-
.../query_p0/system/test_partitions_schema.groovy | 6 +-
.../query_p0/system/test_table_options.groovy | 6 +-
.../query_p0/system/test_table_properties.groovy | 6 +-
.../suites/query_p0/test_row_policy.groovy | 4 +-
.../query_p2/big_join_build/big_join_build.groovy | 2 +-
.../test_set_operation_large_string.groovy | 2 +-
.../suites/query_profile/profile_safe.groovy | 5 +-
.../query_profile/s3_load_profile_test.groovy | 55 +-
.../query_profile/test_load_channel_profile.groovy | 6 +-
...e_column_with_delete_drop_column_dup_key.groovy | 82 +--
...olumn_with_delete_drop_column_unique_key.groovy | 63 +-
.../test_filecache_with_alter_table.groovy | 36 +-
.../test_alter_table_column_rename.groovy | 2 +-
.../test_schema_change_fail.groovy | 11 +-
...test_show_create_table_and_views_nereids.groovy | 89 ++-
.../suites/show_p0/test_show_data.groovy | 2 +-
.../suites/show_p0/test_show_processlist.groovy | 6 +-
.../suites/statistics/test_analyze_all_null.groovy | 2 +-
.../suites/statistics/test_hot_value.groovy | 2 +-
.../test_skip_agg_table_value_column.groovy | 2 +-
.../test_partial_update_2pc_schema_change.groovy | 38 +-
.../legacy/test_f_2pc_schema_change.groovy | 38 +-
.../publish/test_f_auto_inc_compaction.groovy | 35 +-
.../test_f_seq_publish_read_from_old.groovy | 33 +-
...xble_partial_update_publish_conflict_seq.groovy | 35 +-
...flexible_partial_update_publish_conflict.groovy | 35 +-
.../test_key_bounds_truncation_basic.groovy | 27 +-
...est_key_bounds_truncation_read_scenarios.groovy | 39 +-
.../test_partial_update_publish_all_del.groovy | 35 +-
.../publish/test_partial_update_publish_seq.groovy | 35 +-
...partial_update_delete_sign_with_conflict.groovy | 33 +-
.../test_partial_update_multi_stmt.groovy | 13 +-
.../test_double_write_when_schema_change.groovy | 4 +-
.../suites/variant_p0/variant_with_rowstore.groovy | 4 +-
350 files changed, 8633 insertions(+), 5157 deletions(-)
diff --git a/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp b/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
index ffa7529d172..32c96474be7 100644
--- a/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
+++ b/be/src/cloud/cloud_engine_calc_delete_bitmap_task.cpp
@@ -213,6 +213,15 @@ Status CloudTabletCalcDeleteBitmapTask::handle(int64_t queue_time_us) const {
}
int64_t t3 = MonotonicMicros();
+ DBUG_EXECUTE_IF("CloudEngineCalcDeleteBitmapTask.handle.block_for_restart", {
+ if (dp->param<int64_t>("tablet_id", -1) == _tablet_id) {
+ // The restart regression pairs this marker with the persisted MS lock owner.
+ LOG(INFO) << "delete bitmap restart barrier token="
+ << dp->param<std::string>("token", "") << " lock_id=" << _transaction_id
+ << " tablet_id=" << _tablet_id;
+ DBUG_BLOCK;
+ }
+ });
DBUG_EXECUTE_IF("CloudEngineCalcDeleteBitmapTask.handle.inject_sleep", {
auto p = dp->param("percent", 0.01);
// 100s > Config.calculate_delete_bitmap_task_timeout_seconds = 60s
@@ -275,6 +284,23 @@ Status CloudTabletCalcDeleteBitmapTask::handle(int64_t queue_time_us) const {
DCHECK(invisible_rowsets.size() == i + 1 - empty_rowset_count);
}
}
+ DBUG_EXECUTE_IF("CloudTabletCalcDeleteBitmapTask.handle.block_after_calc", {
+ auto target_tablet_id = dp->param<int64_t>("tablet_id", 0);
+ auto target_transaction_id = dp->param<int64_t>("transaction_id", 0);
+ if (status.ok() && target_tablet_id == _tablet_id &&
+ (target_transaction_id == 0 || target_transaction_id == _transaction_id)) {
+ // Delay the response after calculation without preventing another transaction from
+ // calculating the delete bitmap on the same tablet.
+ wrlock.unlock();
+ LOG(INFO) << "start debug block after calculating delete bitmap"
+ << ", tablet_id=" << _tablet_id << ", txn_id=" << _transaction_id;
+ while (DebugPoints::instance()->is_enable(DP_NAME)) {
+ std::this_thread::sleep_for(std::chrono::milliseconds(50));
+ }
+ LOG(INFO) << "finish debug block after calculating delete bitmap"
+ << ", tablet_id=" << _tablet_id << ", txn_id=" << _transaction_id;
+ }
+ });
DBUG_EXECUTE_IF("CloudCalcDbmTask.handle.return.block",
auto target_tablet_id = dp->param<int64_t>("tablet_id", 0);
if (target_tablet_id == tablet->tablet_id()) {DBUG_BLOCK});
diff --git a/be/src/cloud/cloud_meta_mgr.cpp b/be/src/cloud/cloud_meta_mgr.cpp
index d6293997e77..f9cf4b31609 100644
--- a/be/src/cloud/cloud_meta_mgr.cpp
+++ b/be/src/cloud/cloud_meta_mgr.cpp
@@ -2338,6 +2338,16 @@ Status CloudMetaMgr::get_delete_bitmap_update_lock(const CloudTablet& tablet, in
"initiator {}",
tablet.table_id(), lock_id, initiator);
}
+ DBUG_EXECUTE_IF("CloudMetaMgr.get_delete_bitmap_update_lock.block_for_restart", {
+ if (st.ok() && res.status().code() == MetaServiceCode::OK &&
+ dp->param<int64_t>("table_id", -1) == tablet.table_id()) {
+ // Pause only a successful acquisition, never a conflict or failed RPC.
+ LOG(INFO) << "delete bitmap restart barrier token="
+ << dp->param<std::string>("token", "") << " lock_id=" << lock_id
+ << " tablet_id=" << tablet.tablet_id() << " initiator=" << initiator;
+ DBUG_BLOCK;
+ }
+ });
return st;
}
diff --git a/docker/runtime/doris-compose/utils.py b/docker/runtime/doris-compose/utils.py
index 0f41728d3f2..ff7f5e4b033 100644
--- a/docker/runtime/doris-compose/utils.py
+++ b/docker/runtime/doris-compose/utils.py
@@ -21,6 +21,7 @@ import jsonpickle
import logging
import os
import pwd
+import re
import socket
import subprocess
import sys
@@ -145,24 +146,46 @@ def is_container_running(container):
# return all doris containers when cluster_names is empty
def get_doris_containers(cluster_names):
- if cluster_names:
- if type(cluster_names) == type(""):
- filter_names = "{}{}-*".format(DORIS_PREFIX, cluster_names)
- else:
- filter_names = "|".join([
- "{}{}-*".format(DORIS_PREFIX, name) for name in cluster_names
- ])
+ import cluster
+ if isinstance(cluster_names, str):
+ requested_names = {cluster_names} if cluster_names else set()
else:
- filter_names = "{}*".format(DORIS_PREFIX)
+ requested_names = set(cluster_names or [])
+
+ # Docker's name filter matches substrings, so anchor the complete service
+ # name before the SDK inspects each matching container. Otherwise a lookup
+ # for "test_vcg" also inspects "test_vcg_warmup..." containers.
+ name_pattern = ("|".join(re.escape(name) for name in sorted(requested_names))
+ if requested_names else ".+")
+ node_pattern = "|".join(re.escape(node_type)
+ for node_type in cluster.Node.TYPE_ALL)
+ filter_names = r"^/{}({})-({})-[0-9]+$".format(
+ re.escape(DORIS_PREFIX), name_pattern, node_pattern)
clusters = {}
client = docker.client.from_env()
- containers = client.containers.list(filters={"name": filter_names})
+ max_list_attempts = 3
+ for attempt in range(1, max_list_attempts + 1):
+ try:
+ # A container may be removed between Docker's list and inspect.
+ containers = client.containers.list(filters={"name": filter_names},
+ ignore_removed=True)
+ break
+ except docker.errors.NotFound as err:
+ if attempt == max_list_attempts:
+ raise
+ LOG.warning(
+ "Docker returned 404 while listing Doris containers; "
+ "retrying enumeration (%d/%d): %s",
+ attempt,
+ max_list_attempts,
+ err,
+ )
for container in containers:
cluster_name, _, _ = parse_service_name(container.name)
if not cluster_name:
continue
- if cluster_names and cluster_name not in cluster_names:
+ if requested_names and cluster_name not in requested_names:
continue
if cluster_name not in clusters:
clusters[cluster_name] = []
diff --git a/regression-test/data/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.out b/regression-test/data/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.out
index 475b9ff01e9..eaa6918709a 100644
--- a/regression-test/data/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.out
+++ b/regression-test/data/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.out
@@ -1,4 +1,7 @@
-- This file is automatically generated. You should know what you did if you want to edit this
--- !sql --
-400
+-- !before_alter --
+400 79800 3490
+
+-- !after_alter --
+400 79800 3490
diff --git a/regression-test/data/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.out b/regression-test/data/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.out
index a8419e02ed9..0db2a386610 100644
--- a/regression-test/data/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.out
+++ b/regression-test/data/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.out
@@ -1,7 +1,7 @@
-- This file is automatically generated. You should know what you did if you want to edit this
-- !source_preheat --
-200
+19900 2690
-- !target_query --
-200
+19900 2690
diff --git a/regression-test/data/cloud_p0/cache/ttl/test_show_ttl.out b/regression-test/data/cloud_p0/cache/ttl/test_show_ttl.out
deleted file mode 100644
index 1d2ad674a77..00000000000
--- a/regression-test/data/cloud_p0/cache/ttl/test_show_ttl.out
+++ /dev/null
@@ -1,16 +0,0 @@
--- This file is automatically generated. You should know what you did if you want to edit this
--- !test_show_ttl --
-0
-
--- !test_show_ttl_2 --
-0
-
--- !test_show_ttl_3 --
-customer_ttl CREATE TABLE `customer_ttl` (\n `C_CUSTKEY` int NOT NULL,\n `C_NAME` varchar(25) NOT NULL,\n `C_ADDRESS` varchar(40) NOT NULL,\n `C_NATIONKEY` int NOT NULL,\n `C_PHONE` char(15) NOT NULL,\n `C_ACCTBAL` decimal(15,2) NOT NULL,\n `C_MKTSEGMENT` char(10) NOT NULL,\n `C_COMMENT` varchar(117) NOT NULL\n) ENGINE=OLAP\nDUPLICATE KEY(`C_CUSTKEY`, `C_NAME`)\nDISTRIBUTED BY HASH(`C_CUSTKEY`) BUCKETS 32\nPROPERTIES (\n"file_cache_ttl_seconds" = "300",\n"is_being_synced" = "fals [...]
-
--- !test_show_ttl_4 --
-0
-
--- !test_show_ttl_5 --
-customer_ttl CREATE TABLE `customer_ttl` (\n `C_CUSTKEY` int NOT NULL,\n `C_NAME` varchar(25) NOT NULL,\n `C_ADDRESS` varchar(40) NOT NULL,\n `C_NATIONKEY` int NOT NULL,\n `C_PHONE` char(15) NOT NULL,\n `C_ACCTBAL` decimal(15,2) NOT NULL,\n `C_MKTSEGMENT` char(10) NOT NULL,\n `C_COMMENT` varchar(117) NOT NULL\n) ENGINE=OLAP\nDUPLICATE KEY(`C_CUSTKEY`, `C_NAME`)\nDISTRIBUTED BY HASH(`C_CUSTKEY`) BUCKETS 32\nPROPERTIES (\n"file_cache_ttl_seconds" = "0",\n"is_being_synced" = "false" [...]
-
diff --git a/regression-test/data/variant_p0/schema_change/test_double_write_when_schema_change.out b/regression-test/data/variant_p0/schema_change/test_double_write_when_schema_change.out
index 1190a5f30bd..37aba8eb64c 100644
--- a/regression-test/data/variant_p0/schema_change/test_double_write_when_schema_change.out
+++ b/regression-test/data/variant_p0/schema_change/test_double_write_when_schema_change.out
@@ -28,7 +28,7 @@ PullRequestEvent 2489395770 2015-01-01T01:00:02Z
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/1310570?","gravatar_id":"","id":1310570,"login":"soumith","url":"https://api.github.com/users/soumith"},"created_at":"2015-01-01T01:00:00Z","id":"2489395767","payload":{"before":"2d06657267b32e0c8e193c617039da200f710195","commits":[{"sha":"dbd68d30ee1f7b60d404553fc1c6226ebb374c8e","author":{"email":"88de463b5797707cf3425f85a415c3d869db732b@gmail.com","name":"Soumith Chintala"},"message":"back to old structure, except lua f [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/6799218?","gravatar_id":"","id":6799218,"login":"gyuho","url":"https://api.github.com/users/gyuho"},"created_at":"2015-01-01T02:00:00Z","id":"2489418154","payload":{"before":"79d9868b8a1afbc5c6c42d33fcc7b97524f888ff","commits":[{"sha":"cf79996e894954b28db73a6bed54ba490c63f174","author":{"email":"0dfbe0912134938594f432abfeb9cb2f2e9092c9@gmail.com","name":"Gyu-Ho Lee"},"message":"Commit with markdownx at 2014-12-31 17:59:58" [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/4188511?","gravatar_id":"","id":4188511,"login":"iganari","url":"https://api.github.com/users/iganari"},"created_at":"2015-01-01T03:00:01Z","id":"2489436569","payload":{"action":"opened","number":44,"pull_request":{"_links":{"comments":{"href":"https://api.github.com/repos/iganari/mrtg/issues/44/comments"},"commits":{"href":"https://api.github.com/repos/iganari/mrtg/pulls/44/commits"},"html":{"href":"https://github.com/iga [...]
-1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
+1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
@@ -40,7 +40,7 @@ PullRequestEvent 2489395770 2015-01-01T01:00:02Z
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/1310570?","gravatar_id":"","id":1310570,"login":"soumith","url":"https://api.github.com/users/soumith"},"created_at":"2015-01-01T01:00:00Z","id":"2489395767","payload":{"before":"2d06657267b32e0c8e193c617039da200f710195","commits":[{"sha":"dbd68d30ee1f7b60d404553fc1c6226ebb374c8e","author":{"email":"88de463b5797707cf3425f85a415c3d869db732b@gmail.com","name":"Soumith Chintala"},"message":"back to old structure, except lua f [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/6799218?","gravatar_id":"","id":6799218,"login":"gyuho","url":"https://api.github.com/users/gyuho"},"created_at":"2015-01-01T02:00:00Z","id":"2489418154","payload":{"before":"79d9868b8a1afbc5c6c42d33fcc7b97524f888ff","commits":[{"sha":"cf79996e894954b28db73a6bed54ba490c63f174","author":{"email":"0dfbe0912134938594f432abfeb9cb2f2e9092c9@gmail.com","name":"Gyu-Ho Lee"},"message":"Commit with markdownx at 2014-12-31 17:59:58" [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/4188511?","gravatar_id":"","id":4188511,"login":"iganari","url":"https://api.github.com/users/iganari"},"created_at":"2015-01-01T03:00:01Z","id":"2489436569","payload":{"action":"opened","number":44,"pull_request":{"_links":{"comments":{"href":"https://api.github.com/repos/iganari/mrtg/issues/44/comments"},"commits":{"href":"https://api.github.com/repos/iganari/mrtg/pulls/44/commits"},"html":{"href":"https://github.com/iga [...]
-1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
+1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
1 {"actor":{"avatar_url":"https://avatars.githubusercontent.com/u/100067519?","display_login":"savorfamily","gravatar_id":"123","id":100067519,"login":"savorfamily","url":"https://api.github.com/users/savorfamily"},"created_at":"2022-11-07T02:00:00Z","id":"25061216922","payload":{"before":"2bf823a4febcf809da126828ecef7617c8cc48ea","commits":[{"sha":"81106d369f763cb729d9d77610ace252c9db53f0","author":{"email":"73812536+upptime-bot@users.noreply.github.com","name":"Upptime Bot"},"message": [...]
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/Config.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/Config.groovy
index b4be1a88c21..85fa5a3ef4e 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/Config.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/Config.groovy
@@ -1145,7 +1145,9 @@ class Config {
void fetchCloudMode() {
if (runMode == RunMode.UNKNOWN) {
try {
- def result = JdbcUtils.executeToMapArray(getRootConnection(), "SHOW FRONTEND CONFIG LIKE 'cloud_unique_id'")
+ def result = getRootConnection().withCloseable { conn ->
+ JdbcUtils.executeToMapArray(conn, "SHOW FRONTEND CONFIG LIKE 'cloud_unique_id'")
+ }
runMode = result[0].Value.toString().isEmpty() ? RunMode.NOT_CLOUD : RunMode.CLOUD
} catch (Throwable t) {
throw new IllegalStateException("Fetch server config 'cloud_unique_id' failed, jdbcUrl: ${jdbcUrl}", t)
@@ -1155,7 +1157,9 @@ class Config {
boolean isClusterKeyEnabled() {
try {
- def result = JdbcUtils.executeToMapArray(getRootConnection(), "SHOW FRONTEND CONFIG LIKE 'random_add_order_by_keys_for_mow'")
+ def result = getRootConnection().withCloseable { conn ->
+ JdbcUtils.executeToMapArray(conn, "SHOW FRONTEND CONFIG LIKE 'random_add_order_by_keys_for_mow'")
+ }
log.info("show random_add_order_by_keys_for_mow config: ${result}".toString())
return result[0].Value.toString().equalsIgnoreCase("true")
} catch (Throwable t) {
@@ -1199,8 +1203,12 @@ class Config {
return DriverManager.getConnection(jdbcUrl, jdbcUser, jdbcPassword)
}
+ String getRootPassword() {
+ return (jdbcUser == 'root' && jdbcPassword) ? jdbcPassword : ''
+ }
+
Connection getRootConnection() {
- return DriverManager.getConnection(jdbcUrl, 'root', '')
+ return DriverManager.getConnection(jdbcUrl, 'root', getRootPassword())
}
Connection getConnectionByDbName(String dbName) {
@@ -1219,14 +1227,7 @@ class Config {
Class.forName("org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver")
String arrowFlightSqlHost = otherConfigs.get("extArrowFlightSqlHost")
String arrowFlightSqlPort = otherConfigs.get("extArrowFlightSqlPort")
- String arrowFlightSqlUrl = "jdbc:arrow-flight-sql://${arrowFlightSqlHost}:${arrowFlightSqlPort}" +
- "/?useServerPrepStmts=false&useSSL=false&useEncryption=false"
- // Arrow 17.0.0-rc03 support jdbc:arrow-flight-sql connect db
- // https://github.com/apache/arrow/issues/41947
- if (dbName?.trim()) {
- arrowFlightSqlUrl = "jdbc:arrow-flight-sql://${arrowFlightSqlHost}:${arrowFlightSqlPort}" +
- "/catalog=" + dbName + "?useServerPrepStmts=false&useSSL=false&useEncryption=false"
- }
+ String arrowFlightSqlUrl = getArrowFlightSqlJdbcUrl(arrowFlightSqlHost, arrowFlightSqlPort, dbName)
tryCreateDbIfNotExist(dbName)
log.info("connect to ${arrowFlightSqlUrl}".toString())
String arrowFlightSqlJdbcUser = otherConfigs.get("extArrowFlightSqlUser")
@@ -1234,6 +1235,27 @@ class Config {
return DriverManager.getConnection(arrowFlightSqlUrl, arrowFlightSqlJdbcUser, arrowFlightSqlJdbcPassword)
}
+ String getArrowFlightSqlJdbcUrl(String host, String port, String dbName = null) {
+ // Arrow 17.0.0-rc03 supports selecting the database with /catalog=<dbName>.
+ // Flight JDBC uses useEncryption and PEM paths rather than MySQL's useSSL/key stores.
+ String catalog = dbName?.trim() ? "catalog=${dbName}" : ""
+ boolean enableTls = otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true") ?: false
+ String url = "jdbc:arrow-flight-sql://${host}:${port}/${catalog}?useServerPrepStmts=false&useEncryption=${enableTls}"
+ if (enableTls) {
+ String verifyMode = otherConfigs.get("tlsVerifyMode")?.toString()?.toLowerCase() ?: "strict"
+ if (verifyMode == "none") {
+ url += "&disableCertificateVerification=true"
+ } else if (otherConfigs.get("trustCACert")) {
+ url += "&tlsRootCerts=${otherConfigs.get('trustCACert')}"
+ }
+ if (otherConfigs.get("trustCert") && otherConfigs.get("trustCAKey")) {
+ url += "&clientCertificate=${otherConfigs.get('trustCert')}" +
+ "&clientKey=${otherConfigs.get('trustCAKey')}"
+ }
+ }
+ return url
+ }
+
Connection getDownstreamConnection() {
return DriverManager.getConnection(ccrDownstreamUrl, ccrDownstreamUser, ccrDownstreamPassword)
}
@@ -1300,24 +1322,17 @@ class Config {
return jdbcUrl
}
- String urlWithDb = jdbcUrl
- String urlWithoutSchema = jdbcUrl.substring(jdbcUrl.indexOf("://") + 3)
- if (urlWithoutSchema.indexOf("/") >= 0) {
- if (jdbcUrl.contains("?")) {
- // e.g: jdbc:mysql://locahost:8080/?a=b
- urlWithDb = jdbcUrl.substring(0, jdbcUrl.lastIndexOf("?"))
- urlWithDb = urlWithDb.substring(0, urlWithDb.lastIndexOf("/"))
- urlWithDb += ("/" + dbName) + jdbcUrl.substring(jdbcUrl.lastIndexOf("?"))
- } else {
- // e.g: jdbc:mysql://locahost:8080/
- urlWithDb += dbName
- }
- } else {
- // e.g: jdbc:mysql://locahost:8080
- urlWithDb += ("/" + dbName)
+ int authorityStart = jdbcUrl.indexOf("://") + 3
+ int queryStart = jdbcUrl.indexOf("?", authorityStart)
+ int pathStart = jdbcUrl.indexOf("/", authorityStart)
+ if (pathStart >= 0 && (queryStart < 0 || pathStart < queryStart)) {
+ return jdbcUrl.substring(0, pathStart + 1) + dbName +
+ (queryStart >= 0 ? jdbcUrl.substring(queryStart) : "")
}
-
- return urlWithDb
+ if (queryStart >= 0) {
+ return jdbcUrl.substring(0, queryStart) + "/" + dbName + jdbcUrl.substring(queryStart)
+ }
+ return jdbcUrl + "/" + dbName
}
public static String buildUrlWithDb(String jdbcUrl, String dbName) {
@@ -1330,8 +1345,7 @@ class Config {
public static String buildUrlWithDb (String jdbcUrl, String dbName, String keyStorePath, String keyStorePassword, String trustStorePath, String trustStorePassword) {
String urlWithDb = buildUrlWithDbImpl(jdbcUrl, dbName);
- urlWithDb = addTlsUrl(urlWithDb, keyStorePath, keyStorePassword, trustStorePath, trustStorePassword);
- urlWithDb = addTimeoutUrl(urlWithDb);
+ urlWithDb = buildTlsJdbcUrl(urlWithDb, keyStorePath, keyStorePassword, trustStorePath, trustStorePassword);
return urlWithDb
}
@@ -1343,8 +1357,27 @@ class Config {
return url
}
+ public static String buildUrlWithDb(String host, int queryPort, String dbName,
+ String keyStorePath, String keyStorePassword,
+ String trustStorePath, String trustStorePassword) {
+ def url = String.format(
+ "jdbc:mysql://%s:%s/?useLocalSessionState=true&allowLoadLocalInfile=false",
+ host, queryPort)
+ url = buildUrlWithDb(url, dbName, keyStorePath, keyStorePassword, trustStorePath, trustStorePassword)
+ return url
+ }
+
+ public static String buildTlsJdbcUrl(String jdbcUrl, String keyStorePath, String keyStorePassword,
+ String trustStorePath, String trustStorePassword) {
+ String tlsUrl = addTlsUrl(jdbcUrl, keyStorePath, keyStorePassword, trustStorePath, trustStorePassword)
+ return addTimeoutUrl(tlsUrl)
+ }
+
private static String addSslUrl(String url) {
- if (url.contains("TLS")) {
+ // A configured TLS URL already has its own verification policy and key stores.
+ // Appending the optional-SSL defaults would override requireSSL and certificate verification.
+ if (url.contains("useSSL=") || url.contains("sslMode=") ||
+ url.contains("clientCertificateKeyStoreUrl=") || url.contains("trustCertificateKeyStoreUrl=")) {
return url
}
// ssl-mode = PREFERRED
@@ -1366,25 +1399,33 @@ class Config {
}
private static String addTlsUrl(String url, String keyStorePath, String keyStorePassword, String trustStorePath, String trustStorePassword) {
- // ssl-mode = PREFERRED
- String useSsl = "true"
- String requireSsl = "true"
- String useSslconfig = "useSSL=" + useSsl + "&requireSSL=" + requireSsl + "&verifyServerCertificate=true"
- String clientCAKey = "clientCertificateKeyStoreUrl=file:" + keyStorePath
- String clientCAPwd = "clientCertificateKeyStorePassword=" + keyStorePassword
- String trustCAKey = "trustCertificateKeyStoreUrl=file:" + trustStorePath
- String trustCAPwd = "trustCertificateKeyStorePassword=" + trustStorePassword
- String tlsUrl = useSslconfig + "&" + clientCAKey + "&" + clientCAPwd + "&" + trustCAKey + "&" + trustCAPwd
- // e.g: jdbc:mysql://locahost:8080/dbname?
- if (url.charAt(url.length() - 1) == '?') {
- return url + tlsUrl
- // e.g: jdbc:mysql://locahost:8080/dbname?a=b
- } else if (url.contains('?')) {
- return url + '&' + tlsUrl
- // e.g: jdbc:mysql://locahost:8080/dbname
- } else {
- return url + '?' + tlsUrl
+ int queryStart = url.indexOf('?')
+ String baseUrl = queryStart >= 0 ? url.substring(0, queryStart) : url
+ List<String> options = new ArrayList<>()
+ Set<String> tlsOptions = new HashSet<>(Arrays.asList(
+ "sslMode", "useSSL", "requireSSL", "verifyServerCertificate",
+ "clientCertificateKeyStoreUrl", "clientCertificateKeyStorePassword",
+ "trustCertificateKeyStoreUrl", "trustCertificateKeyStorePassword"))
+ if (queryStart >= 0) {
+ for (String option : url.substring(queryStart + 1).split('&')) {
+ if (option.isEmpty()) {
+ continue
+ }
+ int equals = option.indexOf('=')
+ String name = equals >= 0 ? option.substring(0, equals) : option
+ if (!tlsOptions.contains(name)) {
+ options.add(option)
+ }
+ }
}
+ options.add("useSSL=true")
+ options.add("requireSSL=true")
+ options.add("verifyServerCertificate=true")
+ options.add("clientCertificateKeyStoreUrl=file:" + keyStorePath)
+ options.add("clientCertificateKeyStorePassword=" + keyStorePassword)
+ options.add("trustCertificateKeyStoreUrl=file:" + trustStorePath)
+ options.add("trustCertificateKeyStorePassword=" + trustStorePassword)
+ return baseUrl + "?" + String.join("&", options)
}
private static String addTimeoutUrl(String url) {
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/ProfileAction.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/ProfileAction.groovy
index b73ef8e196c..40deeec8930 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/ProfileAction.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/ProfileAction.groovy
@@ -36,6 +36,7 @@ class ProfileAction implements SuiteAction {
private Runnable runCallback
private Closure<String> check
private SuiteContext context
+ private InetSocketAddress profileAddress
ProfileAction(SuiteContext context, String tag) {
this.context = context
@@ -47,6 +48,13 @@ class ProfileAction implements SuiteAction {
this.tag = null
}
+ /** Query profiles are local to the FE that executed the SQL. */
+ ProfileAction(SuiteContext context, InetSocketAddress profileAddress) {
+ this.context = context
+ this.tag = null
+ this.profileAddress = Objects.requireNonNull(profileAddress, "profileAddress can not be null")
+ }
+
void run(@ClosureParams(value = FromString, options = []) Runnable run) {
runCallback = run
}
@@ -58,17 +66,13 @@ class ProfileAction implements SuiteAction {
List getProfileList() {
def httpCli = new HttpCliAction(context)
- def addr = context.getFeHttpAddress()
+ def addr = profileAddress ?: context.getFeHttpAddress()
httpCli.endpoint("${addr.hostString}:${addr.port}")
httpCli.uri("/rest/v1/query_profile")
httpCli.op("get")
httpCli.printResponse(false)
- if (context.config.isCloudMode()) {
- httpCli.basicAuthorization(context.config.feCloudHttpUser, context.config.feCloudHttpPassword)
- } else {
- httpCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
- }
+ httpCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
List profileData = []
httpCli.check { code, body ->
if (code != 200) {
@@ -104,17 +108,13 @@ class ProfileAction implements SuiteAction {
String getProfile(String profileId) {
def profileCli = new HttpCliAction(context)
- def addr = context.getFeHttpAddress()
+ def addr = profileAddress ?: context.getFeHttpAddress()
profileCli.endpoint("${addr.hostString}:${addr.port}")
profileCli.uri("/rest/v1/query_profile/${profileId}")
profileCli.op("get")
profileCli.printResponse(false)
- if (context.config.isCloudMode()) {
- profileCli.basicAuthorization(context.config.feCloudHttpUser, context.config.feCloudHttpPassword)
- } else {
- profileCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
- }
+ profileCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
def result = [text: ""]
profileCli.check { profileCode, profileResp ->
if (profileCode != 200) {
@@ -222,17 +222,13 @@ class ProfileAction implements SuiteAction {
}
def httpCli = new HttpCliAction(context)
- def addr = context.getFeHttpAddress()
+ def addr = profileAddress ?: context.getFeHttpAddress()
httpCli.endpoint("${addr.hostString}:${addr.port}")
httpCli.uri("/rest/v1/query_profile")
httpCli.op("get")
httpCli.printResponse(false)
- if (context.config.isCloudMode()) {
- httpCli.basicAuthorization(context.config.feCloudHttpUser, context.config.feCloudHttpPassword)
- } else {
- httpCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
- }
+ httpCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
httpCli.check { code, body ->
if (code != 200) {
throw new IllegalStateException("Get profile list failed, code: ${code}, body:\n${body}")
@@ -252,11 +248,7 @@ class ProfileAction implements SuiteAction {
profileCli.op("get")
profileCli.printResponse(false)
- if (context.config.isCloudMode()) {
- profileCli.basicAuthorization(context.config.feCloudHttpUser, context.config.feCloudHttpPassword)
- } else {
- profileCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
- }
+ profileCli.basicAuthorization(context.config.feHttpUser, context.config.feHttpPassword)
profileCli.check { profileCode, profileResp ->
if (profileCode != 200) {
throw new IllegalStateException("Get profile failed, url: ${"/rest/v1/query_profile/${profileId}"}, code: ${profileCode}, body:\n${profileResp}")
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/ScriptContext.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/ScriptContext.groovy
index c8b64fb9807..16790e43309 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/ScriptContext.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/ScriptContext.groovy
@@ -21,6 +21,7 @@ import groovy.transform.CompileStatic
import groovy.util.logging.Slf4j
import org.apache.doris.regression.Config
import org.apache.doris.regression.suite.event.EventListener
+import org.apache.doris.regression.util.Http
import java.lang.reflect.UndeclaredThrowableException
import java.util.concurrent.ExecutorService
@@ -62,6 +63,16 @@ class ScriptContext implements Closeable {
def outputRelativePath = path.substring(0, path.lastIndexOf(".")) + ".out"
this.outputFile = new File(new File(config.dataPath), outputRelativePath)
this.dataPath = this.outputFile.getParentFile().getCanonicalFile()
+ Http.configure(
+ (config.otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true")) ?: false,
+ config.otherConfigs.get("tlsVerifyMode")?.toString()?.toLowerCase() ?: "strict",
+ config.otherConfigs.get("trustStorePath")?.toString(),
+ config.otherConfigs.get("trustStorePassword")?.toString(),
+ config.otherConfigs.get("trustStoreType")?.toString() ?: "PKCS12",
+ config.otherConfigs.get("keyStorePath")?.toString(),
+ config.otherConfigs.get("keyStorePassword")?.toString(),
+ config.otherConfigs.get("keyStoreType")?.toString() ?: "PKCS12"
+ )
}
private final synchronized Suite newSuite(String suiteName, String group) {
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
index d280c880874..9899da208c1 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
@@ -35,6 +35,10 @@ import groovy.json.JsonSlurper
import groovy.util.logging.Slf4j
import org.apache.commons.lang3.ObjectUtils
import org.apache.doris.regression.Config
+import org.apache.doris.regression.suite.client.BackendClientImpl
+import org.apache.doris.regression.util.RowsetMetaUtils
+import org.apache.doris.thrift.TNetworkAddress
+import org.apache.doris.thrift.TSyncLoadForTabletsRequest
import org.apache.doris.regression.RegressionTest
import org.apache.doris.regression.action.FlightRecordAction
import org.apache.doris.regression.action.BenchmarkAction
@@ -134,6 +138,30 @@ class Suite implements GroovyInterceptable {
return getConf("suites." + name + "." + key, defaultValue)
}
+ boolean isDorisTlsEnabled() {
+ return Boolean.parseBoolean(getConf("enableTLS", "false"))
+ }
+
+ String getDorisHttpScheme() {
+ return isDorisTlsEnabled() ? "https" : "http"
+ }
+
+ String getDorisCurlTlsOptions() {
+ if (!isDorisTlsEnabled()) {
+ return ""
+ }
+ return " --cert ${quoteShellArgument(getConf('trustCert'))}" +
+ " --key ${quoteShellArgument(getConf('trustCAKey'))}" +
+ " --cacert ${quoteShellArgument(getConf('trustCACert'))}"
+ }
+
+ private static String quoteShellArgument(String value) {
+ if (!value) {
+ throw new IllegalArgumentException("Missing TLS certificate path for curl")
+ }
+ return "'" + value.replace("'", "'\"'\"'") + "'"
+ }
+
List<String> getDorisConnectorTlsArgs() {
if (!Boolean.parseBoolean(getConf("enableTLS", "false"))) {
return Collections.emptyList()
@@ -299,6 +327,21 @@ class Suite implements GroovyInterceptable {
return context.connect(user, password, url, actionSupplier)
}
+ public <T> T connectToDoris(String user = context.config.jdbcUser, String password = context.config.jdbcPassword,
+ String url = context.getJdbcUrl(), Closure<T> actionSupplier) {
+ return context.connectToDoris(user, password, url, actionSupplier)
+ }
+
+ public <T> T connectWithDockerCluster(
+ SuiteCluster cluster,
+ Boolean connectToFollower = false,
+ String user = context.config.jdbcUser,
+ String password = context.config.jdbcPassword,
+ Closure<T> actionSupplier) {
+ def jdbcUrl = cluster.getJdbcUrl(connectToFollower)
+ return context.connect(user, password, jdbcUrl, actionSupplier)
+ }
+
public <T> T connectInDocker(String user = context.config.jdbcUser, String password = context.config.jdbcPassword,
Closure<T> actionSupplier) {
def connInfo = context.threadLocalConn.get()
@@ -314,6 +357,119 @@ class Suite implements GroovyInterceptable {
// }
// )
// }
+ /** Wait for continuous version coverage on each tablet's serving BE, with lazy commit enabled. */
+ void syncAndWaitTabletVersion(Collection<Map> tablets, long version, int timeoutSeconds = 60) {
+ Assertions.assertFalse(tablets.isEmpty(), "no tablets to synchronize")
+ List<Map> tabletList = tablets.toList()
+ Map<String, List<Map>> tabletGroups = tabletList.groupBy { it.BackendId.toString() }
+ Map<String, BackendClientImpl> backendClients = [:]
+ Set<Integer> ready = [] as Set
+ Map<String, Object> lastStates = [:]
+ int pollCount = 0
+ try {
+ if (isCloudMode()) {
+ def backendById = sql_return_maparray("SHOW BACKENDS").collectEntries {
+ [(it.BackendId.toString()): it]
+ }
+ tabletGroups.keySet().each { backendId ->
+ def backend = backendById[backendId]
+ Assertions.assertNotNull(backend,
+ "backend ${backendId} for tablets ${tabletGroups[backendId]*.TabletId} was not found")
+ backendClients[backendId] = new BackendClientImpl(
+ new TNetworkAddress(backend.Host.toString(), backend.BePort as int),
+ backend.HttpPort as int)
+ }
+ }
+
+ awaitUntil(timeoutSeconds, 0.5) {
+ // The BE RPC is asynchronous. Retry it periodically in case an earlier queued
+ // task observed no advancement while Meta Service was finalizing lazy commit.
+ if (!backendClients.isEmpty() && pollCount++ % 10 == 0) {
+ tabletGroups.each { backendId, backendTablets ->
+ backendClients[backendId].client.syncLoadForTablets(
+ new TSyncLoadForTabletsRequest(
+ backendTablets.collect { it.TabletId as long }))
+ }
+ }
+ tabletList.eachWithIndex { tablet, index ->
+ if (!ready.contains(index)) {
+ def status = Http.GET(tablet.CompactionStatus.toString(), true, false,
+ context.config.feHttpUser, context.config.feHttpPassword)
+ lastStates["${tablet.TabletId}@${tablet.BackendId}"] = status
+ if (RowsetMetaUtils.coversVersion(status, version)) {
+ ready.add(index)
+ }
+ }
+ }
+ return ready.size() == tabletList.size()
+ }
+ } catch (org.awaitility.core.ConditionTimeoutException e) {
+ throw new IllegalStateException(
+ "Waiting for tablet version ${version} timed out; last BE states: ${lastStates}", e)
+ } finally {
+ backendClients.values().each { it.close() }
+ }
+ }
+
+ /** Fetch an exact physical rowset after the serving BE has caught up. */
+ Map syncAndWaitCloudRowsetMeta(Map tablet, long version, int timeoutSeconds = 60) {
+ syncAndWaitTabletVersion([tablet], version, timeoutSeconds)
+ return getRowsetMetaAtVersion(tablet, version)
+ }
+
+ /**
+ * Call after syncAndWaitTabletVersion. Cloud headers omit rs_metas, so read the committed
+ * MS rowset key instead. The caller must prevent compaction from removing the exact version.
+ */
+ Map getRowsetMetaAtVersion(Map tablet, long version) {
+ if (!isCloudMode()) {
+ String metaUrl = tablet.MetaUrl.toString()
+ metaUrl += (metaUrl.contains('?') ? '&' : '?') + 'byte_to_base64=true'
+ def header = Http.GET(metaUrl, true, false,
+ context.config.feHttpUser, context.config.feHttpPassword)
+ Assertions.assertTrue(header.rs_metas instanceof List, "tablet header is missing rs_metas")
+ def meta = header.rs_metas.find { (it.end_version as long) == version }
+ Assertions.assertNotNull(meta, "rowset not found: tablet=${tablet.TabletId}, version=${version}")
+ return RowsetMetaUtils.decodeKeyBounds(meta)
+ }
+ def endpoint = context.config.metaServiceHttpAddress
+ def token = context.config.metaServiceToken
+ // CI deployment writes multiClusterInstanceId as a custom property; the legacy
+ // multiClusterInstance can still contain the template's default_instance_id.
+ def instanceId = context.config.otherConfigs.get("multiClusterInstanceId")?.toString()?.trim() ?:
+ context.config.multiClusterInstance?.trim()
+ Assertions.assertTrue(endpoint?.trim() && token?.trim() && instanceId?.trim(),
+ "metaServiceHttpAddress, metaServiceToken and multiClusterInstanceId (or multiClusterInstance) must be configured")
+ def params = [token: token, key_type: "MetaRowsetKey", instance_id: instanceId,
+ tablet_id: tablet.TabletId, version: version]
+ def query = params.collect { key, value ->
+ "${key}=${java.net.URLEncoder.encode(value.toString(), 'UTF-8')}"
+ }.join('&')
+ // Do not use Http.GET here: it logs the URL, including the MS token.
+ def baseUrl = endpoint.contains('://') ? endpoint : "http://${endpoint}"
+ HttpURLConnection conn = new URL("${baseUrl}/MetaService/http/get_value?${query}").openConnection()
+ conn.connectTimeout = 5000
+ conn.readTimeout = 10000
+ conn.instanceFollowRedirects = false
+ try {
+ int code = conn.responseCode
+ Assertions.assertEquals(200, code,
+ "MS rowset read failed: instance=${instanceId}, tablet=${tablet.TabletId}, version=${version}")
+ def meta = new JsonSlurper().parseText(conn.inputStream.getText('UTF-8'))
+ Assertions.assertNotNull(meta.end_version,
+ "MS response is not rowset metadata: tablet=${tablet.TabletId}, version=${version}, code=${meta.code}")
+ Assertions.assertEquals(version, meta.end_version as long)
+ Assertions.assertEquals(tablet.TabletId as long, meta.tablet_id as long)
+ return RowsetMetaUtils.decodeKeyBounds(meta)
+ } catch (IOException e) {
+ // Network exception messages may contain the credential-bearing URL.
+ throw new IOException("MS rowset read failed: tablet=${tablet.TabletId}, version=${version}, " +
+ "error=${e.class.simpleName}")
+ } finally {
+ conn.disconnect()
+ }
+ }
+
public void awaitUntil(int atMostSeconds, double intervalSecond = 1, Closure actionSupplier) {
Awaitility
.with().pollInSameThread()
@@ -1453,8 +1609,12 @@ class Suite implements GroovyInterceptable {
return outBuf.toString()
}
+ static String buildSshCommand(String username, String host, String cmd) {
+ return "ssh -o StrictHostKeyChecking=no ${username}@${host} '${cmd}'"
+ }
+
void sshExec(String username, String host, String cmd, boolean alert=true) {
- String command = "ssh ${username}@${host} '${cmd}'"
+ String command = buildSshCommand(username, host, cmd)
def cmds = ["/bin/bash", "-c", command]
logger.info("Execute: ${cmds}".toString())
Process p = cmds.execute()
@@ -1980,27 +2140,34 @@ class Suite implements GroovyInterceptable {
}
String getServerPrepareJdbcUrl(String jdbcUrl, String database, boolean useMasterIp) {
- String urlWithoutSchema = jdbcUrl.substring(jdbcUrl.indexOf("://") + 3)
- def sql_ip = useMasterIp ? getMasterIp() : urlWithoutSchema.substring(0, urlWithoutSchema.indexOf(":"))
- def sql_port
- if (urlWithoutSchema.indexOf("/") >= 0) {
- // e.g: jdbc:mysql://locahost:8080/?a=b
- sql_port = urlWithoutSchema.substring(urlWithoutSchema.indexOf(":") + 1, urlWithoutSchema.indexOf("/"))
- } else {
- // e.g: jdbc:mysql://locahost:8080
- sql_port = urlWithoutSchema.substring(urlWithoutSchema.indexOf(":") + 1)
- }
- String tlsUrl = ""
- // set server side prepared statement url
+ String scheme = "jdbc:mysql://"
+ if (!jdbcUrl.startsWith(scheme)) {
+ throw new IllegalArgumentException("Expected a MySQL JDBC URL")
+ }
+ String endpointAndPath = jdbcUrl.substring(scheme.length())
+ int pathStart = endpointAndPath.indexOf("/")
+ int queryStart = endpointAndPath.indexOf("?")
+ int endpointEnd = pathStart >= 0 && (queryStart < 0 || pathStart < queryStart)
+ ? pathStart : (queryStart >= 0 ? queryStart : endpointAndPath.length())
+ String endpoint = endpointAndPath.substring(0, endpointEnd)
+ int portStart = endpoint.lastIndexOf(":")
+ if (portStart < 0) {
+ throw new IllegalArgumentException("MySQL JDBC URL has no port")
+ }
+ String host = useMasterIp ? getMasterIp() : endpoint.substring(0, portStart)
+ String suffix = endpointAndPath.substring(endpointEnd)
+ if (!suffix.startsWith("/")) {
+ suffix = "/" + suffix
+ }
+ String url = Config.buildUrlWithDbImpl(scheme + host + endpoint.substring(portStart) + suffix, database)
if ((context.config.otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true")) ?: false) {
- String useSslconfig = "useSSL=true&requireSSL=true&verifyServerCertificate=true"
- String clientCAKey = "clientCertificateKeyStoreUrl=file:" + context.config.otherConfigs.get("keyStorePath")
- String clientCAPwd = "clientCertificateKeyStorePassword=" + context.config.otherConfigs.get("keyStorePassword")
- String trustCAKey = "trustCertificateKeyStoreUrl=file:" + context.config.otherConfigs.get("trustStorePath")
- String trustCAPwd = "trustCertificateKeyStorePassword=" + context.config.otherConfigs.get("trustStorePassword")
- tlsUrl = "&" + useSslconfig + "&" + clientCAKey + "&" + clientCAPwd + "&" + trustCAKey + "&" + trustCAPwd
+ url = Config.buildTlsJdbcUrl(url,
+ context.config.otherConfigs.get("keyStorePath")?.toString(),
+ context.config.otherConfigs.get("keyStorePassword")?.toString(),
+ context.config.otherConfigs.get("trustStorePath")?.toString(),
+ context.config.otherConfigs.get("trustStorePassword")?.toString())
}
- return "jdbc:mysql://" + sql_ip + ":" + sql_port + "/" + database + "?&useServerPrepStmts=true" + tlsUrl
+ return url + (url.contains("?") ? "&" : "?") + "useServerPrepStmts=true"
}
DebugPoint GetDebugPoint() {
@@ -3394,7 +3561,7 @@ class Suite implements GroovyInterceptable {
endpoint feEndPoint
uri "/rest/v1/query_profile"
check check_func
- basicAuthorization "${context.config.feCloudHttpUser}","${context.config.feCloudHttpPassword}"
+ basicAuthorization "${context.config.feHttpUser}","${context.config.feHttpPassword}"
}
}
@@ -3417,7 +3584,7 @@ class Suite implements GroovyInterceptable {
endpoint feEndPoint
uri "/api/profile?query_id=${query_id}"
check check_func
- basicAuthorization "${context.config.feCloudHttpUser}","${context.config.feCloudHttpPassword}"
+ basicAuthorization "${context.config.feHttpUser}","${context.config.feHttpPassword}"
}
}
@@ -3626,6 +3793,26 @@ class Suite implements GroovyInterceptable {
}
}
+ def scp_udf_file_to_all_fe = { udf_file_path ->
+ def udf_file = new File(udf_file_path).absoluteFile
+ assertTrue(udf_file.isFile(), "UDF file does not exist: ${udf_file}")
+ def fe_hosts = sql_return_maparray("SHOW FRONTENDS").collect { it.Host }.unique()
+ assertTrue(!fe_hosts.isEmpty(), "No frontend found to copy UDF file to")
+ if (fe_hosts.size() == 1) {
+ def feAddress = java.net.InetAddress.getByName(fe_hosts[0].toString())
+ if (feAddress.isAnyLocalAddress() || feAddress.isLoopbackAddress() ||
+ java.net.NetworkInterface.getByInetAddress(feAddress) != null) {
+ logger.info("Only one local frontend, skip scp udf file")
+ return
+ }
+ }
+
+ fe_hosts.each { fe_host ->
+ sshExec("root", fe_host, "mkdir -p ${udf_file.parent}")
+ scpFiles("root", fe_host, udf_file.path, udf_file.path, false)
+ }
+ }
+
def check_fold_consistency = { test_sql ->
def re_fe = order_sql "select /*+SET_VAR(enable_fold_constant_by_be=false)*/ ${test_sql}"
def re_be = order_sql "select /*+SET_VAR(enable_fold_constant_by_be=true)*/ ${test_sql}"
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteContext.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteContext.groovy
index dd360b353b8..fb33da6865d 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteContext.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/SuiteContext.groovy
@@ -160,6 +160,13 @@ class SuiteContext implements Closeable {
Connection getConnectionByDbName(String dbName) {
def jdbcUrl = getJdbcUrl()
+ if (isTlsEnabled()) {
+ jdbcUrl = Config.buildTlsJdbcUrl(jdbcUrl,
+ config.otherConfigs.get("keyStorePath")?.toString(),
+ config.otherConfigs.get("keyStorePassword")?.toString(),
+ config.otherConfigs.get("trustStorePath")?.toString(),
+ config.otherConfigs.get("trustStorePassword")?.toString())
+ }
def jdbcConn = DriverManager.getConnection(jdbcUrl, config.jdbcUser, config.jdbcPassword)
try {
String sql = "CREATE DATABASE IF NOT EXISTS ${dbName}"
@@ -170,7 +177,9 @@ class SuiteContext implements Closeable {
} catch (Throwable t) {
throw new IllegalStateException("Create database failed, jdbcUrl: ${jdbcUrl}", t)
}
- def dbUrl = Config.buildUrlWithDb(jdbcUrl, dbName)
+ def dbUrl = isTlsEnabled()
+ ? Config.buildUrlWithDbImpl(jdbcUrl, dbName)
+ : Config.buildUrlWithDb(jdbcUrl, dbName)
log.info("connect to ${dbUrl}".toString())
return DriverManager.getConnection(dbUrl, config.jdbcUser, config.jdbcPassword)
}
@@ -362,6 +371,22 @@ class SuiteContext implements Closeable {
}
}
+ // Use this explicitly for Doris FE connections. Generic connect must also support external
+ // databases and TLS tests that intentionally supply their own (or no) certificates.
+ public <T> T connectToDoris(String user, String password, String url, Closure<T> actionSupplier) {
+ if (!url.startsWith("jdbc:mysql:")) {
+ throw new IllegalArgumentException("Doris FE connections require a MySQL JDBC URL")
+ }
+ if (isTlsEnabled()) {
+ url = Config.buildTlsJdbcUrl(url,
+ config.otherConfigs.get("keyStorePath")?.toString(),
+ config.otherConfigs.get("keyStorePassword")?.toString(),
+ config.otherConfigs.get("trustStorePath")?.toString(),
+ config.otherConfigs.get("trustStorePassword")?.toString())
+ }
+ return connect(user, password, url, actionSupplier)
+ }
+
public <T> T connect(String user, String password, String url, Closure<T> actionSupplier) {
def originConnection = threadLocalConn.get()
if ((config.otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true")) ?: false) {
@@ -403,6 +428,10 @@ class SuiteContext implements Closeable {
}
}
+ private boolean isTlsEnabled() {
+ return (config.otherConfigs.get("enableTLS")?.toString()?.equalsIgnoreCase("true")) ?: false
+ }
+
Connection getMasterConnectionByDbName(String dbName) {
def result = JdbcUtils.executeToMapArray(getConnection(), "SHOW FRONTENDS")
def master = null
@@ -414,7 +443,16 @@ class SuiteContext implements Closeable {
}
if (master) {
log.info("master found: ${master.Host}:${master.HttpPort}")
- def url = Config.buildUrlWithDb(master.Host as String, master.QueryPort as Integer, dbName)
+ def url = isTlsEnabled()
+ ? Config.buildUrlWithDb(
+ master.Host as String,
+ master.QueryPort as Integer,
+ dbName,
+ config.otherConfigs.get("keyStorePath")?.toString(),
+ config.otherConfigs.get("keyStorePassword")?.toString(),
+ config.otherConfigs.get("trustStorePath")?.toString(),
+ config.otherConfigs.get("trustStorePassword")?.toString())
+ : Config.buildUrlWithDb(master.Host as String, master.QueryPort as Integer, dbName)
def username = config.jdbcUser
def password = config.jdbcPassword
@@ -436,7 +474,16 @@ class SuiteContext implements Closeable {
}
if (master) {
log.info("master found: ${master.Host}:${master.HttpPort}")
- def url = Config.buildUrlWithDb(master.Host as String, master.QueryPort as Integer, dbName)
+ def url = isTlsEnabled()
+ ? Config.buildUrlWithDb(
+ master.Host as String,
+ master.QueryPort as Integer,
+ dbName,
+ config.otherConfigs.get("keyStorePath")?.toString(),
+ config.otherConfigs.get("keyStorePassword")?.toString(),
+ config.otherConfigs.get("trustStorePath")?.toString(),
+ config.otherConfigs.get("trustStorePassword")?.toString())
+ : Config.buildUrlWithDb(master.Host as String, master.QueryPort as Integer, dbName)
ConnectionInfo connInfo = threadLocalConn.get()
def userName = null
def userPass = null
@@ -447,7 +494,7 @@ class SuiteContext implements Closeable {
userName = config.jdbcUser
userPass = config.jdbcPassword
}
- connectTo(url, connInfo.username, connInfo.password)
+ connectTo(url, userName, userPass)
log.info("Successfully reconnected to the master")
} else {
throw new Exception("No master found to reconnect")
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/DebugPoint.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/DebugPoint.groovy
index cbf4cda6f8d..58ca44abdf5 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/DebugPoint.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/DebugPoint.groovy
@@ -38,6 +38,11 @@ class DebugPoint {
this.suite = suite
}
+ private static String buildDebugPointUrl(String host, int httpPort, String path) {
+ String scheme = Http.enableTls ? "https" : "http"
+ return "${scheme}://${host}:${httpPort}${path}"
+ }
+
/* Enable debug point in regression
* Note: set BE config::enable_debug_points = true to take effect
* Parameters:
@@ -48,7 +53,7 @@ class DebugPoint {
* params: timeout, execute, or other customized input params
*/
static def enableDebugPoint(String host, int httpPort, NodeType type, String name, Map<String, String> params = null) {
- def url = 'http://' + host + ':' + httpPort + '/api/debug_point/add/' + name
+ def url = buildDebugPointUrl(host, httpPort, '/api/debug_point/add/' + name)
if (params != null && params.size() > 0) {
url += '?' + params.collect((k, v) -> k + '=' + v).join('&')
}
@@ -64,7 +69,7 @@ class DebugPoint {
* name: debug point name
*/
static def disableDebugPoint(String host, int httpPort, NodeType type, String name) {
- def url = 'http://' + host + ':' + httpPort + '/api/debug_point/remove/' + name
+ def url = buildDebugPointUrl(host, httpPort, '/api/debug_point/remove/' + name)
def result = Http.POST(url, null, true)
Http.checkHttpResult(result, type)
}
@@ -76,7 +81,7 @@ class DebugPoint {
* type: NodeType.BE or NodeType.FE
*/
static def clearDebugPoints(String host, int httpPort, NodeType type) {
- def url = 'http://' + host + ':' + httpPort + '/api/debug_point/clear'
+ def url = buildDebugPointUrl(host, httpPort, '/api/debug_point/clear')
def result = Http.POST(url, null, true)
Http.checkHttpResult(result, type)
}
@@ -197,4 +202,3 @@ class DebugPoint {
}
}
}
-
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/Http.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/Http.groovy
index 8ee33b5ecba..e902a345a66 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/Http.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/Http.groovy
@@ -66,6 +66,9 @@ class Http {
keyStorePath = keyPath
keyStorePassword = keyPassword
keyStoreType = keystoreType
+ if (!enableTls) {
+ return
+ }
initSSLContext()
}
@@ -110,13 +113,26 @@ class Http {
}
}
- static Object GET(url, isJson = false, printText = true) {
- if (enableTls) {
- url = url.replace("http://", "https://")
+ static String basicAuthorization(String user, String password) {
+ String credentials = "${user}:${password ?: ''}"
+ return 'Basic ' + credentials.getBytes('UTF-8').encodeBase64().toString()
+ }
+
+ static URLConnection openConnection(String url) {
+ if (enableTls && url.startsWith('http://')) {
+ url = 'https://' + url.substring('http://'.length())
}
- def conn = new URL(url).openConnection()
+ return new URL(url).openConnection()
+ }
+
+ static Object GET(url, isJson = false, printText = true) {
+ return GET(url, isJson, printText, 'root', '')
+ }
+
+ static Object GET(url, isJson, printText, String user, String password) {
+ def conn = openConnection(url)
conn.setRequestMethod('GET')
- conn.setRequestProperty('Authorization', 'Basic cm9vdDo=') //token for root
+ conn.setRequestProperty('Authorization', basicAuthorization(user, password))
def code = conn.responseCode
def text = conn.content.text
if (printText) {
@@ -135,12 +151,16 @@ class Http {
}
static Object POST(url, data = null, isJson = false) {
+ return POST(url, data, isJson, 'root', '')
+ }
+
+ static Object POST(url, data, isJson, String user, String password) {
if (enableTls) {
url = url.replace("http://", "https://")
}
def conn = new URL(url).openConnection()
conn.setRequestMethod('POST')
- conn.setRequestProperty('Authorization', 'Basic cm9vdDo=') //token for root
+ conn.setRequestProperty('Authorization', basicAuthorization(user, password))
if (data) {
if (isJson) {
conn.setRequestProperty('Content-Type', 'application/json')
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/MultiClusterStateGuard.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/MultiClusterStateGuard.groovy
new file mode 100644
index 00000000000..2594c43d02b
--- /dev/null
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/MultiClusterStateGuard.groovy
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+
+/** Restores the dedicated multi-cluster test instance; never used for docker/upgrade tests. */
+class MultiClusterStateGuard {
+ Closure listClusters
+ Closure dropCluster
+ Closure addCluster
+ Closure readDefault
+ Closure setDefault
+ Closure awaitState
+ Closure restoreInstance = {}
+
+ /** Parse legacy IP:heartbeat:http:cloud-id entries with an optional brpc port. */
+ static List<Map> configuredBackends(String configuredBes) {
+ if (!configuredBes?.trim()) {
+ throw new IllegalArgumentException('Missing multiClusterBes configuration')
+ }
+ def nodes = []
+ configuredBes.split(',').eachWithIndex { entry, index ->
+ def parts = entry.trim().split(':', -1)
+ if (!(parts.length in [4, 5]) || parts.any { !it.trim() }) {
+ throw new IllegalArgumentException("Invalid multiClusterBes entry at index ${index}")
+ }
+ def ports = [parts[1], parts[2]] + (parts.length == 5 ? [parts[4]] : [])
+ if (ports.any { !it.isInteger() || it.toInteger() < 1 || it.toInteger() > 65535 }) {
+ throw new IllegalArgumentException("Invalid BE port at index ${index}")
+ }
+ nodes << [ip: parts[0], heartbeat_port: parts[1].toInteger(), http_port: parts[2].toInteger(),
+ cloud_unique_id: parts[3]]
+ }
+ // cloud_unique_id identifies a cloud deployment, not an individual BE process.
+ if (nodes.collect { [it.ip, it.heartbeat_port] }.toSet().size() != nodes.size()) {
+ throw new IllegalArgumentException('multiClusterBes contains duplicate BE endpoints')
+ }
+ return nodes
+ }
+
+ /** Shared fixture: the first two configured BEs each belong to one compute group. */
+ static List<Map> sharedBaseline(String configuredBes) {
+ def nodes = configuredBackends(configuredBes)
+ if (nodes.size() < 2) {
+ throw new IllegalArgumentException('The shared multi-cluster baseline requires at least two BEs')
+ }
+ return (0..1).collect { index ->
+ [type: 'COMPUTE', cluster_name: "regression_cluster_name${index}".toString(),
+ cluster_id: "regression_cluster_id${index}".toString(), cluster_status: 'NORMAL',
+ nodes: [nodes[index]]]
+ }
+ }
+
+ private static Object copy(Object value) {
+ return new JsonSlurper().parseText(JsonOutput.toJson(value))
+ }
+
+ // MS refreshes these timestamps when recreating a cluster or node.
+ static Object topology(Object value) {
+ if (value instanceof Map) {
+ def result = new TreeMap()
+ value.each { key, item ->
+ if (!(key in ['ctime', 'mtime'])) {
+ result[key] = topology(item)
+ }
+ }
+ return result
+ }
+ if (value instanceof List) {
+ return value.collect { topology(it) }.sort { a, b ->
+ JsonOutput.toJson(a) <=> JsonOutput.toJson(b)
+ }
+ }
+ return value
+ }
+
+ static boolean frontendMatches(List<Map> expected, List<Map> groups, List<Map> backends) {
+ if ((groups.collect { it.cluster } as Set) != (expected.collect { it.cluster_name } as Set)) {
+ return false
+ }
+ def expectedNodes = expected.collectMany { cluster ->
+ (cluster.nodes ?: []).collect { node ->
+ [cluster.cluster_id.toString(), (node.ip ?: node.host).toString(), node.heartbeat_port.toString(),
+ cluster.cluster_name.toString(), cluster.cluster_status.toString(), node.cloud_unique_id.toString()]
+ }
+ }.sort { a, b -> a.toString() <=> b.toString() }
+ // Backend.getCloudClusterStatus treats an absent status tag as NORMAL.
+ def actualNodes = backends.collect { backend ->
+ def tag = backend.Tag instanceof Map ? backend.Tag : new JsonSlurper().parseText(backend.Tag.toString())
+ [(tag.compute_group_id ?: tag.cloud_cluster_id)?.toString(), backend.Host.toString(),
+ backend.HeartbeatPort.toString(), (tag.compute_group_name ?: tag.cloud_cluster_name)?.toString(),
+ (tag.compute_group_status ?: tag.cloud_cluster_status ?: 'NORMAL').toString(), tag.cloud_unique_id?.toString()]
+ }.sort { a, b -> a.toString() <=> b.toString() }
+ return expectedNodes == actualNodes && backends.every {
+ it.Alive.toString().toBoolean() && !it.SystemDecommissioned?.toString()?.toBoolean()
+ }
+ }
+
+ private List<Map> computeClusters() {
+ List<Map> clusters = listClusters.call()
+ if (clusters.any { it.type == 'VIRTUAL' }) {
+ throw new IllegalStateException('Legacy multi-cluster state restoration does not support virtual clusters')
+ }
+ return clusters.findAll { it.type == 'COMPUTE' }
+ }
+
+ /** initializeBaseline keeps a successful load fixture, and rolls back a failed one. */
+ def run(boolean initializeBaseline, Closure body) {
+ List<Map> original = copy(computeClusters()) as List<Map>
+ if (original.any { it.cluster_status != 'NORMAL' ||
+ (it.nodes ?: []).any { n -> n.status && n.status != 'NODE_STATUS_RUNNING' } }) {
+ throw new IllegalStateException('Multi-cluster tests require normal clusters and running nodes')
+ }
+ String originalDefault = readDefault.call()
+ // A stale default cannot be restored with SET PROPERTY. Reject it before destructive tests.
+ if (originalDefault && !original.any { it.cluster_name == originalDefault }) {
+ throw new IllegalStateException("Initial default compute group does not exist: ${originalDefault}")
+ }
+ Throwable failure = null
+ try {
+ // Tests can replace the groups. New connections must not inherit a deleted group.
+ setDefault.call('')
+ awaitState.call(original, '')
+ return body.call()
+ } catch (Throwable t) {
+ failure = t
+ throw t
+ } finally {
+ def errors = []
+ def attempt = { Closure action ->
+ try {
+ action.call()
+ } catch (Throwable t) {
+ errors.add(t)
+ }
+ }
+ attempt { restoreInstance.call() }
+ List<Map> expected = original
+ String expectedDefault = originalDefault
+ attempt {
+ if (initializeBaseline && failure == null) {
+ // The load suite intentionally leaves its successful setup for subsequent suites.
+ expected = copy(computeClusters()) as List<Map>
+ expectedDefault = readDefault.call()
+ } else {
+ def expectedById = original.collectEntries { [(it.cluster_id): it] }
+ computeClusters().each { current ->
+ def saved = expectedById[current.cluster_id]
+ if (saved == null || topology(saved) != topology(current)) {
+ attempt { dropCluster.call(current) }
+ }
+ }
+ // Re-read after deletion; a failed drop must not prevent restoring other groups.
+ def currentById = computeClusters().collectEntries { [(it.cluster_id): it] }
+ original.each { saved ->
+ if (!currentById.containsKey(saved.cluster_id)) {
+ attempt { addCluster.call(copy(saved)) }
+ }
+ }
+ }
+ }
+ // Restore metadata and FE node maps before restoring the default property.
+ // Even if a wait fails, attempt the remaining cleanup and report every failure.
+ attempt { awaitState.call(expected, null) }
+ attempt { setDefault.call(expectedDefault) }
+ attempt { awaitState.call(expected, expectedDefault) }
+ if (!errors.isEmpty()) {
+ Throwable restoreFailure = new IllegalStateException('Failed to restore multi-cluster test state')
+ errors.each { restoreFailure.addSuppressed(it) }
+ if (failure != null) {
+ failure.addSuppressed(restoreFailure)
+ } else {
+ throw restoreFailure
+ }
+ }
+ }
+ }
+}
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/MySqlClient.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/MySqlClient.groovy
new file mode 100644
index 00000000000..e0de281b8f3
--- /dev/null
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/MySqlClient.groovy
@@ -0,0 +1,70 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+import java.nio.file.attribute.PosixFilePermissions
+
+class MySqlClient {
+ // MySQL option files interpret backslash escapes even inside quoted values.
+ private static String quoteOption(String value) {
+ String text = value ?: ''
+ if (text.indexOf(0) >= 0) {
+ throw new IllegalArgumentException('MySQL credentials must not contain NUL')
+ }
+ return '"' + text.replace('\\', '\\\\').replace('"', '\\"')
+ .replace('\n', '\\n').replace('\r', '\\r')
+ .replace('\t', '\\t').replace('\b', '\\b') + '"'
+ }
+
+ static Map execute(String user, String password, List<String> options, String sql,
+ File directory = null, String executable = 'mysql') {
+ def credentials = Files.createTempFile('doris-mysql-', '.cnf',
+ PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString('rw-------')))
+ Process process = null
+ try {
+ String config = "[client]\nuser=${quoteOption(user)}\npassword=${quoteOption(password)}\n"
+ Files.write(credentials, config.getBytes(StandardCharsets.UTF_8))
+ List<String> command = [executable, "--defaults-extra-file=${credentials}".toString()]
+ command.addAll(options.collect { it.toString() })
+ ProcessBuilder builder = new ProcessBuilder(command)
+ if (directory != null) {
+ builder.directory(directory)
+ }
+ process = builder.start()
+ StringBuilder stdout = new StringBuilder()
+ StringBuilder stderr = new StringBuilder()
+ Thread outputReader = process.consumeProcessOutputStream(stdout)
+ Thread errorReader = process.consumeProcessErrorStream(stderr)
+ process.outputStream.withWriter('UTF-8') { writer -> writer.write(sql) }
+ int exitCode = process.waitFor()
+ outputReader.join()
+ errorReader.join()
+ return [exitCode: exitCode, stdout: stdout.toString(), stderr: stderr.toString()]
+ } finally {
+ try {
+ if (process != null && process.isAlive()) {
+ process.destroyForcibly().waitFor()
+ }
+ } finally {
+ Files.deleteIfExists(credentials)
+ }
+ }
+ }
+}
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/RowsetMetaUtils.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/RowsetMetaUtils.groovy
new file mode 100644
index 00000000000..26b48e23f48
--- /dev/null
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/RowsetMetaUtils.groovy
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import java.nio.charset.StandardCharsets
+
+class RowsetMetaUtils {
+ /** Only active rowsets count; a high max version must not hide a gap in the version chain. */
+ static boolean coversVersion(Map status, long version) {
+ if (version < 0) {
+ throw new IllegalArgumentException("negative target version: ${version}")
+ }
+ if (!(status.rowsets instanceof List)) {
+ throw new IllegalArgumentException("compaction response is missing rowsets: ${status}")
+ }
+ def ranges = status.rowsets.collect { rowset ->
+ def match = rowset.toString() =~ /^\[(\d+)-(\d+)\](?:\s.*)?$/
+ if (!match.matches()) {
+ throw new IllegalArgumentException("invalid compaction rowset: ${rowset}")
+ }
+ long start = match.group(1).toLong()
+ long end = match.group(2).toLong()
+ if (start > end) {
+ throw new IllegalArgumentException("invalid rowset version range: ${rowset}")
+ }
+ [start, end]
+ }.sort { a, b -> a[0] <=> b[0] }
+ long covered = -1
+ for (def range : ranges) {
+ if (range[0] > covered + 1) {
+ return false
+ }
+ covered = Math.max(covered, range[1] as long)
+ if (covered >= version) {
+ return true
+ }
+ }
+ return false
+ }
+
+ /** MS protobuf JSON encodes bytes as base64. Preserve one character per key byte. */
+ static Map decodeKeyBounds(Map meta) {
+ Map result = new LinkedHashMap(meta)
+ if (meta.segments_key_bounds instanceof List) {
+ result.segments_key_bounds = meta.segments_key_bounds.collect { bounds ->
+ Map decoded = new LinkedHashMap(bounds)
+ ['min_key', 'max_key'].each { key ->
+ if (!(bounds[key] instanceof String)) {
+ throw new IllegalArgumentException("key bounds are missing ${key}")
+ }
+ decoded[key] = new String(Base64.decoder.decode(bounds[key]), StandardCharsets.ISO_8859_1)
+ }
+ decoded
+ }
+ }
+ // These optional protobuf flags default to false when omitted from MS JSON.
+ ['segments_key_bounds_truncated', 'segments_key_bounds_aggregated'].each { key ->
+ if (!result.containsKey(key)) {
+ result[key] = false
+ }
+ }
+ return result
+ }
+}
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/WarmupMetricsUtils.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/WarmupMetricsUtils.groovy
index 846ce4ac34b..e765c55a508 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/WarmupMetricsUtils.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/util/WarmupMetricsUtils.groovy
@@ -44,7 +44,7 @@ class WarmupMetricsUtils {
*/
static long getBrpcMetric(String ip, String port, String metricName) {
def url = "http://${ip}:${port}/brpc_metrics"
- def text = new URL(url).text
+ def text = readText(url)
def matcher = text =~ ~"${metricName}\\s+(\\d+)"
if (matcher.find()) {
return matcher[0][1] as long
@@ -53,7 +53,7 @@ class WarmupMetricsUtils {
}
static String getPrometheusMetrics(String ip, Object port) {
- return new URL("http://${ip}:${port}/metrics").text
+ return readText("http://${ip}:${port}/metrics")
}
static BigDecimal findPrometheusMetricValue(String metricsText, String metricName, Map labels) {
@@ -101,7 +101,7 @@ class WarmupMetricsUtils {
}
static void clearFileCache(String ip, String httpPort) {
- def response = new URL("http://${ip}:${httpPort}/api/file_cache?op=clear&sync=true").text
+ def response = readText("http://${ip}:${httpPort}/api/file_cache?op=clear&sync=true")
def json = new JsonSlurper().parseText(response)
if (json.status != "OK") {
throw new RuntimeException("Clear cache on ${ip}:${httpPort} failed: ${json.status}")
@@ -115,6 +115,96 @@ class WarmupMetricsUtils {
Thread.sleep(waitMs)
}
+ private static String backendIp(Object backend) {
+ return backend instanceof Map ? backend.ip.toString() : backend[1].toString()
+ }
+
+ private static String backendHttpPort(Object backend) {
+ return backend instanceof Map ? backend.httpPort.toString() : backend[4].toString()
+ }
+
+ private static String backendBrpcPort(Object backend) {
+ return backend instanceof Map ? backend.brpcPort.toString() : backend[5].toString()
+ }
+
+ private static String readText(String url) {
+ def connection = Http.openConnection(url)
+ connection.connectTimeout = 5000
+ connection.readTimeout = 60000
+ try {
+ return connection.inputStream.withCloseable { it.getText("UTF-8") }
+ } finally {
+ connection.disconnect()
+ }
+ }
+
+ static Map getBackendMetricValues(Collection backends, String metricName) {
+ Map values = [:]
+ for (be in backends) {
+ String key = "${backendIp(be)}:${backendBrpcPort(be)}"
+ values[key] = getBrpcMetric(backendIp(be), backendBrpcPort(be), metricName)
+ }
+ return values
+ }
+
+ static long getBackendMetricSum(Collection backends, String metricName) {
+ return getBackendMetricValues(backends, metricName).values().sum(0L) as long
+ }
+
+ static long waitForBackendMetricSum(Collection backends, String metricName,
+ Closure<Boolean> predicate, long timeoutMs = 60000,
+ String message = null) {
+ long deadline = System.currentTimeMillis() + timeoutMs
+ long value = 0L
+ while (System.currentTimeMillis() < deadline) {
+ value = getBackendMetricSum(backends, metricName)
+ if (predicate(value)) {
+ return value
+ }
+ logger.info("waiting ${metricName} sum, value=${value}")
+ Thread.sleep(1000)
+ }
+ throw new RuntimeException((message ?: "Metric ${metricName} sum did not satisfy predicate") +
+ " within ${timeoutMs}ms, last=${value}")
+ }
+
+ static long waitForBackendMetricSumStable(Collection backends, String metricName,
+ long stableMs = 10000, long timeoutMs = 60000) {
+ long deadline = System.currentTimeMillis() + timeoutMs
+ long lastValue = getBackendMetricSum(backends, metricName)
+ long stableSince = System.currentTimeMillis()
+ while (System.currentTimeMillis() < deadline) {
+ Thread.sleep(1000)
+ long currentValue = getBackendMetricSum(backends, metricName)
+ if (currentValue != lastValue) {
+ lastValue = currentValue
+ stableSince = System.currentTimeMillis()
+ } else if (System.currentTimeMillis() - stableSince >= stableMs) {
+ return currentValue
+ }
+ logger.info("waiting ${metricName} sum stable, value=${currentValue}")
+ }
+ throw new RuntimeException("Metric ${metricName} sum did not stay stable for ${stableMs}ms " +
+ "within ${timeoutMs}ms, last=${lastValue}")
+ }
+
+ static long waitForBrpcMetric(String ip, String port, String metricName,
+ Closure<Boolean> predicate, long timeoutMs = 60000,
+ String message = null) {
+ long deadline = System.currentTimeMillis() + timeoutMs
+ long value = 0L
+ while (System.currentTimeMillis() < deadline) {
+ value = getBrpcMetric(ip, port, metricName)
+ if (predicate(value)) {
+ return value
+ }
+ logger.info("waiting ${metricName} for ${ip}:${port}, value=${value}")
+ Thread.sleep(1000)
+ }
+ throw new RuntimeException((message ?: "Metric ${metricName} did not satisfy predicate") +
+ " within ${timeoutMs}ms, last=${value}")
+ }
+
static long sumProfileCounter(String profileText, String counterName) {
def matcher = profileText =~ ~"(?m)(?<![A-Za-z0-9_])${Pattern.quote(counterName)}\\s*:\\s*([0-9,]+)"
long sum = 0
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/ConfigArrowFlightSqlUrlTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/ConfigArrowFlightSqlUrlTest.groovy
new file mode 100644
index 00000000000..e04fcaba42d
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/ConfigArrowFlightSqlUrlTest.groovy
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression
+
+import org.junit.jupiter.api.Test
+
+import static org.junit.jupiter.api.Assertions.assertEquals
+import static org.junit.jupiter.api.Assertions.assertFalse
+import static org.junit.jupiter.api.Assertions.assertTrue
+
+class ConfigArrowFlightSqlUrlTest {
+ @Test
+ void plaintextFlightUrlRemainsAvailableWhenTlsIsDisabled() {
+ Config config = new Config()
+ assertEquals("jdbc:arrow-flight-sql://localhost:8070/?useServerPrepStmts=false&useEncryption=false",
+ config.getArrowFlightSqlJdbcUrl("localhost", "8070"))
+ }
+
+ @Test
+ void mtlsFlightUrlUsesCaAndClientPemFiles() {
+ Config config = new Config()
+ config.otherConfigs.put("enableTLS", "true")
+ config.otherConfigs.put("trustCACert", "/test/ca.crt")
+ config.otherConfigs.put("trustCert", "/test/client.crt")
+ config.otherConfigs.put("trustCAKey", "/test/client.key")
+
+ String url = config.getArrowFlightSqlJdbcUrl("localhost", "8070", "test_db")
+ assertTrue(url.startsWith("jdbc:arrow-flight-sql://localhost:8070/catalog=test_db?"))
+ assertTrue(url.contains("useEncryption=true"))
+ assertTrue(url.contains("tlsRootCerts=/test/ca.crt"))
+ assertTrue(url.contains("clientCertificate=/test/client.crt"))
+ assertTrue(url.contains("clientKey=/test/client.key"))
+ assertFalse(url.contains("disableCertificateVerification=true"))
+ }
+
+ @Test
+ void flightUrlHonorsDisabledCertificateVerification() {
+ Config config = new Config()
+ config.otherConfigs.put("enableTLS", "true")
+ config.otherConfigs.put("tlsVerifyMode", "none")
+
+ String url = config.getArrowFlightSqlJdbcUrl("localhost", "8070")
+ assertTrue(url.contains("useEncryption=true"))
+ assertTrue(url.contains("disableCertificateVerification=true"))
+ }
+}
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/ConfigRootPasswordTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/ConfigRootPasswordTest.groovy
new file mode 100644
index 00000000000..cd908606cba
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/ConfigRootPasswordTest.groovy
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression
+
+import org.junit.jupiter.api.Test
+
+import java.sql.Connection
+import java.sql.Driver
+import java.sql.DriverManager
+
+import static org.junit.jupiter.api.Assertions.*
+
+class ConfigRootPasswordTest {
+ @Test
+ void rootConnectionsOnlyReusePasswordsConfiguredForRoot() {
+ def calls = []
+ boolean closed = false
+ Connection connection = [close: { closed = true }] as Connection
+ Driver driver = [connect: { String url, Properties properties ->
+ if (url != 'jdbc:root-credentials:test') return null
+ calls.add(new HashMap(properties))
+ return connection
+ }] as Driver
+ DriverManager.registerDriver(driver)
+ try {
+ def cases = [
+ ['root', 'fake password \u5bc6\u78bc', 'fake password \u5bc6\u78bc'],
+ ['root', '', ''], ['root', null, ''], ['root', ' ', ' '],
+ ['test_user', 'other-user-password', ''], ['test_user', '', ''],
+ [null, 'other-user-password', '']
+ ]
+ cases.each { user, password, expected ->
+ Config config = new Config(jdbcUrl: 'jdbc:root-credentials:test',
+ jdbcUser: user, jdbcPassword: password)
+ assertEquals(expected, config.getRootPassword())
+ closed = false
+ config.getRootConnection().withCloseable { assertSame(connection, it) }
+ assertTrue(closed)
+ assertEquals('root', calls.last().user)
+ assertEquals(expected, calls.last().password)
+ }
+ } finally {
+ DriverManager.deregisterDriver(driver)
+ }
+ }
+}
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/suite/CompactionPluginCompatibilityTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/suite/CompactionPluginCompatibilityTest.groovy
new file mode 100644
index 00000000000..c8106d9e7bf
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/suite/CompactionPluginCompatibilityTest.groovy
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.suite
+
+import groovy.lang.Binding
+import groovy.lang.GroovyShell
+import groovy.lang.GroovySystem
+import groovy.json.JsonSlurper
+import org.apache.doris.regression.Config
+import org.junit.jupiter.api.AfterEach
+import org.junit.jupiter.api.Test
+import org.junit.jupiter.api.io.TempDir
+import org.slf4j.LoggerFactory
+
+import java.nio.file.Path
+
+import static org.junit.jupiter.api.Assertions.assertEquals
+import static org.junit.jupiter.api.Assertions.assertNotNull
+import static org.junit.jupiter.api.Assertions.assertThrows
+import static org.junit.jupiter.api.Assertions.assertTrue
+
+class CompactionPluginCompatibilityTest {
+ @TempDir
+ Path tempDir
+
+ @AfterEach
+ void removePluginMethods() {
+ GroovySystem.metaClassRegistry.removeMetaClass(Suite)
+ }
+
+ @Test
+ void tabletStatusAndCompactionStatusSupportDefaultAndBoundedProbes() {
+ File plugin = [new File("../plugins/plugin_compaction.groovy"),
+ new File("regression-test/plugins/plugin_compaction.groovy")]
+ .find { it.isFile() }
+ assertNotNull(plugin, "plugin_compaction.groovy must be available to this test")
+
+ def logger = LoggerFactory.getLogger(CompactionPluginCompatibilityTest)
+ new GroovyShell(getClass().classLoader, new Binding(logger: logger)).evaluate(plugin)
+
+ def requests = []
+ Suite.metaClass.curl = { String method, String url, String body, Integer timeoutSec,
+ String user, String password, Integer maxRetries ->
+ requests.add([method, url, body, timeoutSec, user, password, maxRetries])
+ return [0, "{}", ""]
+ }
+
+ File suiteDir = tempDir.resolve("suites/compaction").toFile()
+ assertTrue(suiteDir.mkdirs())
+ File suiteFile = new File(suiteDir, "compaction_plugin_compatibility.groovy")
+ assertTrue(suiteFile.createNewFile())
+ Config config = new Config()
+ config.suitePath = tempDir.resolve("suites").toString()
+ config.dataPath = tempDir.resolve("data").toString()
+ config.realDataPath = tempDir.resolve("real-data").toString()
+ config.defaultDb = "regression_test"
+ config.feHttpUser = "http_user"
+ config.feHttpPassword = "http_password"
+ ScriptContext scriptContext = new ScriptContext(
+ suiteFile, null, null, config, Collections.emptyList(), { true })
+ SuiteCluster cluster = new SuiteCluster("compaction_plugin_compatibility", config)
+ SuiteContext context = new SuiteContext(
+ suiteFile, "compaction_plugin_compatibility", "p0", scriptContext,
+ cluster, null, null, config)
+ Suite suite = new Suite("compaction_plugin_compatibility", "p0", context, cluster)
+ assertEquals([0, "{}", ""], suite.be_show_tablet_status("127.0.0.1", "8040", "42"))
+ assertEquals([0, "{}", ""], suite.be_show_tablet_status("127.0.0.1", "8040", "42", 5, 1))
+ assertEquals([0, "{}", ""], suite.be_get_compaction_status("127.0.0.1", "8040", "42"))
+ assertEquals([0, "{}", ""], suite.be_get_compaction_status("127.0.0.1", "8040", "42", 5, 1))
+
+ assertEquals([
+ ["GET", "http://127.0.0.1:8040/api/compaction/show?tablet_id=42",
+ null, 10, "http_user", "http_password", 10],
+ ["GET", "http://127.0.0.1:8040/api/compaction/show?tablet_id=42",
+ null, 5, "http_user", "http_password", 1],
+ ["GET", "http://127.0.0.1:8040/api/compaction/run_status?tablet_id=42",
+ null, 10, "http_user", "http_password", 10],
+ ["GET", "http://127.0.0.1:8040/api/compaction/run_status?tablet_id=42",
+ null, 5, "http_user", "http_password", 1]
+ ], requests)
+
+ def defaultCalls = []
+ Suite.metaClass.curl = { String method, String url, String body, Integer timeoutSec,
+ String user, String password, Integer maxRetries ->
+ defaultCalls.add([method, url, body, timeoutSec, user, password, maxRetries])
+ return [0, "{}", ""]
+ }
+ assertEquals([0, "{}", ""], suite.be_get_overall_compaction_status("127.0.0.1", "8040"))
+ assertEquals([0, "{}", ""], suite.be_run_base_compaction("127.0.0.1", "8040", "42"))
+ assertEquals([0, "{}", ""], suite.be_run_cumulative_compaction("127.0.0.1", "8040", "42"))
+ assertEquals([0, "{}", ""], suite.be_run_full_compaction("127.0.0.1", "8040", "42"))
+ assertEquals([0, "{}", ""], suite.be_run_full_compaction_by_table_id("127.0.0.1", "8040", "7"))
+ assertEquals([
+ ["GET", "http://127.0.0.1:8040/api/compaction/run_status",
+ null, 10, "http_user", "http_password", 10],
+ ["POST", "http://127.0.0.1:8040/api/compaction/run?tablet_id=42&compact_type=base",
+ null, 10, "http_user", "http_password", 10],
+ ["POST", "http://127.0.0.1:8040/api/compaction/run?tablet_id=42&compact_type=cumulative",
+ null, 10, "http_user", "http_password", 10],
+ ["POST", "http://127.0.0.1:8040/api/compaction/run?tablet_id=42&compact_type=full",
+ null, 10, "http_user", "http_password", 10],
+ ["POST", "http://127.0.0.1:8040/api/compaction/run?table_id=7&compact_type=full",
+ null, 10, "http_user", "http_password", 10]
+ ], defaultCalls)
+
+ // This reproduces the plugin-internal three-argument call that failed in TeamCity 212706.
+ Suite.metaClass.getBackendIpHttpPort = { Map hosts, Map ports ->
+ hosts["1"] = "127.0.0.1"
+ ports["1"] = "8040"
+ }
+ Suite.metaClass.sql_return_maparray = { String ignored -> [[BackendId: "1", TabletId: "42"]] }
+ Suite.metaClass.sql = { String ignored -> [[null, '"disable_auto_compaction" = "true"']] }
+ Suite.metaClass.curl = { String method, String url, String body, Integer timeoutSec,
+ String user, String password, Integer maxRetries ->
+ throw new IllegalStateException("tablet probe reached")
+ }
+ IllegalStateException probe = assertThrows(IllegalStateException) {
+ suite.trigger_and_wait_compaction("t", "full")
+ }
+ assertEquals("tablet probe reached", probe.message)
+
+ Suite.metaClass.parseJson = { String value -> new JsonSlurper().parseText(value) }
+ int showCount = 0
+ Suite.metaClass.curl = { String method, String url, String body = null, Integer timeoutSec = 10,
+ String user = "", String password = "", Integer maxRetries = 10 ->
+ if (url.contains("/api/compaction/show?")) {
+ showCount++
+ return [0, '{"last full success time":"' + (showCount == 1 ? "old" : "new")
+ + '","last full failure time":"old"}', ""]
+ }
+ if (url.contains("/api/compaction/run_status?")) {
+ return [0, '{"status":"success","run_status":false}', ""]
+ }
+ return [0, '{"status":"success"}', ""]
+ }
+ suite.trigger_and_wait_compaction("t", "full")
+ assertTrue(showCount >= 2)
+ }
+}
diff --git a/regression-test/suites/show_p0/test_show_data.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/suite/SuiteSshExecTest.groovy
similarity index 61%
copy from regression-test/suites/show_p0/test_show_data.groovy
copy to regression-test/framework/src/test/groovy/org/apache/doris/regression/suite/SuiteSshExecTest.groovy
index 20adc299d55..c7fe5e33f20 100644
--- a/regression-test/suites/show_p0/test_show_data.groovy
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/suite/SuiteSshExecTest.groovy
@@ -15,17 +15,17 @@
// specific language governing permissions and limitations
// under the License.
-suite("test_show_data") {
- def result = sql """show data;"""
- logger.info("show data result:${result}");
- assertTrue(result.size() > 0);
+package org.apache.doris.regression.suite
- def jdbcUrlWithoutDbStr = (context.config.jdbcUrl).split(context.config.defaultDb)
- logger.info("jdbcUrlWithoutDbStr:${jdbcUrlWithoutDbStr}");
- def result2 = connect(context.config.jdbcUser, context.config.jdbcPassword, jdbcUrlWithoutDbStr[0]) {
- sql """show data;"""
- }
+import org.junit.jupiter.api.Test
+
+import static org.junit.jupiter.api.Assertions.assertEquals
- assertTrue(result2.size() > 0);
- assertTrue(result2[0].size() == 6);
-}
\ No newline at end of file
+class SuiteSshExecTest {
+ @Test
+ void sshCommandAcceptsNewRegressionHosts() {
+ assertEquals(
+ "ssh -o StrictHostKeyChecking=no root@172.17.0.1 'mkdir -p /tmp/udf'",
+ Suite.buildSshCommand("root", "172.17.0.1", "mkdir -p /tmp/udf"))
+ }
+}
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/HttpAuthenticationTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/HttpAuthenticationTest.groovy
new file mode 100644
index 00000000000..a1542b24d13
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/HttpAuthenticationTest.groovy
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import com.sun.net.httpserver.HttpServer
+import groovy.json.JsonOutput
+import org.junit.jupiter.api.Test
+
+import java.util.concurrent.Executors
+import java.util.concurrent.TimeUnit
+
+import static org.junit.jupiter.api.Assertions.*
+
+class HttpAuthenticationTest {
+ @Test
+ void credentialsAreRequestScopedAndLegacyCallsStillUseEmptyRootPassword() {
+ def server = HttpServer.create(new InetSocketAddress('127.0.0.1', 0), 0)
+ def serverThreads = Executors.newCachedThreadPool()
+ def callers = Executors.newFixedThreadPool(6)
+ boolean originalTls = Http.enableTls
+ server.executor = serverThreads
+ server.createContext('/') { exchange ->
+ def response = [authorization: exchange.requestHeaders.getFirst('Authorization'),
+ body: exchange.requestBody.getText('UTF-8')]
+ byte[] bytes = JsonOutput.toJson(response).getBytes('UTF-8')
+ exchange.responseHeaders.set('Content-Type', 'application/json; charset=UTF-8')
+ exchange.sendResponseHeaders(200, bytes.length)
+ exchange.responseBody.withCloseable { it.write(bytes) }
+ }
+ try {
+ Http.enableTls = false
+ server.start()
+ String url = "http://127.0.0.1:${server.address.port}/"
+ def passwords = ['', null, 'test-password', '密碼 \\" :#; $()`\\\\\n']
+ def futures = (0..<24).collect { index ->
+ callers.submit({ ->
+ String user = "user${index}"
+ String password = passwords[index % passwords.size()]
+ def response = index % 2 == 0
+ ? Http.GET(url, true, false, user, password)
+ : Http.POST(url, [index: index], true, user, password)
+ String decoded = new String(Base64.decoder.decode(response.authorization.substring(6)), 'UTF-8')
+ assertEquals(user + ':' + (password ?: ''), decoded)
+ if (index % 2 != 0) assertEquals(JsonOutput.toJson([index: index]), response.body)
+ } as Runnable)
+ }
+ futures.each { it.get(10, TimeUnit.SECONDS) }
+ assertEquals('Basic cm9vdDo=', Http.GET(url, true, false).authorization)
+ assertEquals('Basic cm9vdDo=', Http.POST(url, null, true).authorization)
+ } finally {
+ Http.enableTls = originalTls
+ server.stop(0)
+ callers.shutdownNow()
+ serverThreads.shutdownNow()
+ }
+ }
+}
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/MultiClusterStateGuardTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/MultiClusterStateGuardTest.groovy
new file mode 100644
index 00000000000..1cdc9a17d06
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/MultiClusterStateGuardTest.groovy
@@ -0,0 +1,256 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import groovy.json.JsonOutput
+import groovy.json.JsonSlurper
+import org.junit.jupiter.api.Test
+
+import static org.junit.jupiter.api.Assertions.*
+
+class MultiClusterStateGuardTest {
+ private static def copy(value) {
+ new JsonSlurper().parseText(JsonOutput.toJson(value))
+ }
+
+ private static Map cluster(String id) {
+ [type: 'COMPUTE', cluster_id: id, cluster_name: "group_${id}".toString(),
+ cluster_status: 'NORMAL', public_endpoint: "public_${id}".toString(),
+ private_endpoint: "private_${id}".toString(), properties: [key: 'value'], ctime: 1,
+ nodes: [[cloud_unique_id: "be_${id}".toString(), ip: '127.0.0.1', heartbeat_port: 9050,
+ status: 'NODE_STATUS_RUNNING', ctime: 1],
+ [cloud_unique_id: "be2_${id}".toString(), ip: '127.0.0.2', heartbeat_port: 9051,
+ status: 'NODE_STATUS_RUNNING', ctime: 1]]]
+ }
+
+ private static Map fixture(List<Map> clusters = [cluster('a'), cluster('b')], String defaultGroup = 'group_a') {
+ def state = [clusters: copy(clusters), defaultGroup: defaultGroup, events: [],
+ failDrop: [], failAdd: [], failWait: false, instanceStatus: 'NORMAL']
+ state.guard = new MultiClusterStateGuard(
+ listClusters: { state.clusters },
+ restoreInstance: { state.events << 'instance'; state.instanceStatus = 'NORMAL' },
+ dropCluster: { c ->
+ state.events << "drop:${c.cluster_id}".toString()
+ if (c.cluster_id in state.failDrop) { throw new IOException('drop failed') }
+ state.clusters.removeAll { it.cluster_id == c.cluster_id }
+ },
+ addCluster: { c ->
+ state.events << "add:${c.cluster_id}".toString()
+ if (c.cluster_id in state.failAdd) { throw new IOException('add failed') }
+ c.ctime = 100
+ c.nodes.each { it.ctime = 100 }
+ state.clusters << c
+ },
+ readDefault: { state.defaultGroup },
+ setDefault: { saved -> state.events << 'default'; state.defaultGroup = saved },
+ awaitState: { expected, expectedDefault ->
+ state.events << "wait:${expectedDefault}".toString()
+ if (state.failWait) { throw new IOException('FE timeout') }
+ assertEquals(MultiClusterStateGuard.topology(expected), MultiClusterStateGuard.topology(state.clusters))
+ if (expectedDefault != null) { assertEquals(expectedDefault, state.defaultGroup) }
+ }
+ )
+ state
+ }
+
+ @Test
+ void sharedBaselineUsesFirstTwoConfiguredNodesWithoutSharingMutableSnapshots() {
+ String config = 'be0:9050:8040:id0,be1:9051:8041:id1,be2:9052:8042:id2'
+ def baseline = MultiClusterStateGuard.sharedBaseline(config)
+ assertEquals(['regression_cluster_name0', 'regression_cluster_name1'], baseline*.cluster_name)
+ assertEquals(['regression_cluster_id0', 'regression_cluster_id1'], baseline*.cluster_id)
+ assertEquals(['id0', 'id1'], baseline.collect { it.nodes[0].cloud_unique_id })
+ assertEquals(8041, baseline[1].nodes[0].http_port)
+ baseline[0].nodes.clear()
+ assertEquals(1, MultiClusterStateGuard.sharedBaseline(config)[0].nodes.size())
+ }
+
+ @Test
+ void acceptsOptionalBrpcPortAndSharedCloudId() {
+ def nodes = MultiClusterStateGuard.configuredBackends('be0:9050:8040:shared:8060,be1:9050:8040:shared')
+ assertEquals(['shared', 'shared'], nodes*.cloud_unique_id)
+ assertEquals(['be0', 'be1'], nodes*.ip)
+ assertEquals(2, MultiClusterStateGuard.sharedBaseline('be0:9050:8040:shared:8060,be1:9050:8040:shared:8060').size())
+ }
+
+ @Test
+ void initializerReplacesExistingFixtureAndRollsBackFailedReplacement() {
+ def state = fixture()
+ state.guard.run(true) {
+ assertEquals('', state.defaultGroup)
+ state.clusters = [cluster('canonical')]
+ state.defaultGroup = 'group_canonical'
+ }
+ assertEquals(['canonical'], state.clusters*.cluster_id)
+ assertEquals('group_canonical', state.defaultGroup)
+ state = fixture()
+ def original = copy(state.clusters)
+ assertThrows(IOException) {
+ state.guard.run(true) {
+ state.clusters = [cluster('canonical')]
+ state.defaultGroup = 'group_canonical'
+ throw new IOException('load failed')
+ }
+ }
+ assertEquals(MultiClusterStateGuard.topology(original), MultiClusterStateGuard.topology(state.clusters))
+ assertEquals('group_a', state.defaultGroup)
+ }
+
+ @Test
+ void sharedBaselineRejectsMissingMalformedOrDuplicateNodes() {
+ [null, '', 'be0:9050:8040:id0', 'be0:9050:8040:id0,invalid',
+ 'be0:0:8040:id0,be1:9050:8040:id1', 'be0:9050:8040:id0:invalid,be1:9050:8040:id0',
+ 'be0:9050:8040:id0,be0:9050:8041:id1'].each { config ->
+ assertThrows(IllegalArgumentException) { MultiClusterStateGuard.sharedBaseline(config) }
+ }
+ }
+
+ @Test
+ void successRestoresFullMetadataAndDefaultAfterTopology() {
+ def state = fixture()
+ def original = copy(state.clusters)
+ assertEquals(42, state.guard.run(false) {
+ state.clusters[0].nodes.remove(0)
+ state.clusters[0].properties.key = 'changed'
+ state.clusters << cluster('temporary')
+ state.defaultGroup = 'group_temporary'
+ state.instanceStatus = 'OVERDUE'
+ 42
+ })
+ assertEquals(MultiClusterStateGuard.topology(original), MultiClusterStateGuard.topology(state.clusters))
+ assertEquals('group_a', state.defaultGroup)
+ assertEquals('NORMAL', state.instanceStatus)
+ assertEquals(['default', 'wait:', 'instance', 'drop:a', 'drop:temporary', 'add:a', 'wait:null', 'default', 'wait:group_a'], state.events)
+ }
+
+ @Test
+ void failureAfterDeletingGroupsRestoresSnapshotAndRethrowsOriginal() {
+ def state = fixture()
+ def original = copy(state.clusters)
+ def failure = new AssertionError('original failure')
+ assertSame(failure, assertThrows(AssertionError) {
+ state.guard.run(false) {
+ state.clusters.clear()
+ state.clusters << cluster('temporary')
+ state.defaultGroup = 'group_temporary'
+ throw failure
+ }
+ })
+ assertEquals(MultiClusterStateGuard.topology(original), MultiClusterStateGuard.topology(state.clusters))
+ assertEquals('group_a', state.defaultGroup)
+ assertEquals(0, failure.suppressed.length)
+ }
+
+ @Test
+ void unchangedTopologyIsNotRecreated() {
+ def state = fixture()
+ state.guard.run(false) {
+ state.clusters.reverse(true)
+ state.clusters[0].ctime = 123
+ state.clusters[0].nodes.reverse(true)
+ }
+ assertEquals(['default', 'wait:', 'instance', 'wait:null', 'default', 'wait:group_a'], state.events)
+ }
+
+ @Test
+ void cleanupFailuresAreSuppressedAndOtherGroupsAreStillRestored() {
+ def state = fixture()
+ def failure = new AssertionError('query failed')
+ state.failDrop = ['temporary']
+ state.failAdd = ['a']
+ assertSame(failure, assertThrows(AssertionError) {
+ state.guard.run(false) {
+ state.failWait = true
+ state.clusters = [cluster('temporary')]
+ throw failure
+ }
+ })
+ assertTrue(state.events.contains('add:b'))
+ assertTrue(state.events.contains('default'))
+ assertEquals(1, failure.suppressed.length)
+ assertEquals(4, failure.suppressed[0].suppressed.length)
+ }
+
+ @Test
+ void cleanupFailureFailsAnOtherwiseSuccessfulSuite() {
+ def state = fixture()
+ def failure = assertThrows(IllegalStateException) { state.guard.run(false) { state.failWait = true; 42 } }
+ assertEquals(2, failure.suppressed.length)
+ assertTrue(state.events.contains('default'))
+ }
+
+ @Test
+ void bootstrapKeepsSuccessfulInitialSetupButRollsBackFailure() {
+ def state = fixture([], '')
+ state.guard.run(true) { state.clusters << cluster('a'); state.defaultGroup = 'group_a' }
+ assertEquals(['a'], state.clusters*.cluster_id)
+ assertEquals('group_a', state.defaultGroup)
+
+ state = fixture([], '')
+ assertThrows(IOException) {
+ state.guard.run(true) { state.clusters << cluster('a'); throw new IOException('load failed') }
+ }
+ assertTrue(state.clusters.isEmpty())
+ assertEquals('', state.defaultGroup)
+
+ state = fixture([], '')
+ state.guard.run(false) { state.clusters << cluster('a') }
+ assertTrue(state.clusters.isEmpty())
+ }
+
+ @Test
+ void rejectsInvalidInitialStateBeforeRunningBody() {
+ def suspended = cluster('a'); suspended.cluster_status = 'SUSPENDED'
+ def decommissioning = cluster('a'); decommissioning.nodes[0].status = 'NODE_STATUS_DECOMMISSIONING'
+ [fixture([cluster('a')], 'missing'), fixture([suspended]), fixture([decommissioning]),
+ fixture([[type: 'VIRTUAL']])].each { state ->
+ assertThrows(IllegalStateException) { state.guard.run(false) { fail('body must not run') } }
+ assertTrue(state.events.isEmpty())
+ }
+ }
+
+ @Test
+ void feMustSeeExactNodeMappingAndAllHeartbeats() {
+ def expected = [cluster('a')]
+ def groups = [[cluster: 'group_a']]
+ def nodes = [[Host: '127.0.0.1', HeartbeatPort: 9050, Alive: 'true', Tag: JsonOutput.toJson([cloud_cluster_id: 'a', cloud_cluster_name: 'group_a',
+ cloud_cluster_status: 'NORMAL', cloud_unique_id: 'be_a'])],
+ [Host: '127.0.0.2', HeartbeatPort: 9051, Alive: true, Tag: [compute_group_id: 'a',
+ compute_group_name: 'group_a', compute_group_status: 'NORMAL', cloud_unique_id: 'be2_a']]]
+ assertTrue(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, [], nodes))
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, groups, nodes.take(1)))
+ nodes[1].Tag.remove('compute_group_status')
+ assertTrue(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ nodes[1].Tag.compute_group_status = 'NORMAL'
+ nodes[1].Alive = false
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ nodes[1].Alive = true
+ nodes[1].HeartbeatPort = 9052
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ nodes[1].HeartbeatPort = 9051
+ nodes[1].Tag.compute_group_status = 'SUSPENDED'
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ nodes[1].Tag.compute_group_status = 'NORMAL'
+ nodes[1].SystemDecommissioned = true
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ nodes[1].SystemDecommissioned = false
+ nodes[1].Tag.compute_group_id = 'wrong'
+ assertFalse(MultiClusterStateGuard.frontendMatches(expected, groups, nodes))
+ }
+}
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/MySqlClientTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/MySqlClientTest.groovy
new file mode 100644
index 00000000000..1444dc54ec7
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/MySqlClientTest.groovy
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import org.junit.jupiter.api.Test
+import org.junit.jupiter.api.io.TempDir
+
+import java.nio.file.Files
+import java.nio.file.Path
+
+import static org.junit.jupiter.api.Assertions.*
+
+class MySqlClientTest {
+ @TempDir
+ Path tempDir
+
+ private File client(int exitCode) {
+ File script = tempDir.resolve('fake mysql client').toFile()
+ script.text = """#!/bin/sh
+file=\${1#--defaults-extra-file=}
+printf '%s\\n' "\$file"
+ls -l "\$file"
+printf '%s\\n' 'CONFIG'
+cat "\$file"
+printf '%s\\n' 'ARGS'
+printf '%s\\n' "\$@"
+printf '%s\\n' 'SQL'
+cat
+printf '%s\\n' 'DIR'
+pwd
+printf '%s' 'test diagnostic' >&2
+exit ${exitCode}
+"""
+ assertTrue(script.setExecutable(true))
+ return script
+ }
+
+ @Test
+ void passesOptionsAndSqlLiterallyAndCleansPrivateCredentials() {
+ File executable = client(0)
+ File directory = Files.createDirectory(tempDir.resolve('working directory')).toFile()
+ String sql = 'select "$HOME", "`id`", "\\n";\n'
+ ['', null, 'fake secret "\\\\#;\n\t密碼'].each { password ->
+ def result = MySqlClient.execute('root', password,
+ ['--ssl-mode=VERIFY_CA', '--ssl-ca=/path with spaces/ca.pem'], sql, directory, executable.path)
+ assertEquals(0, result.exitCode)
+ assertEquals('test diagnostic\n', result.stderr)
+ def lines = result.stdout.readLines()
+ assertFalse(new File(lines[0]).exists())
+ assertTrue(lines[1].startsWith('-rw-------'))
+ String args = result.stdout.split('ARGS\n', 2)[1].split('SQL\n', 2)[0]
+ assertEquals(["--defaults-extra-file=${lines[0]}".toString(), '--ssl-mode=VERIFY_CA',
+ '--ssl-ca=/path with spaces/ca.pem'], args.readLines())
+ assertTrue(result.stdout.contains('SQL\n' + sql))
+ assertTrue(result.stdout.endsWith('DIR\n' + directory.canonicalPath + '\n'))
+ String config = result.stdout.split('CONFIG\n', 2)[1].split('ARGS\n', 2)[0]
+ assertTrue(config.startsWith('[client]\nuser="root"\npassword="'))
+ if (!password) assertTrue(config.endsWith('password=""\n'))
+ else assertEquals(3, config.readLines().size())
+ }
+ }
+
+ @Test
+ void preservesNonRootUserAndFailureExitCodeAndStillCleansFile() {
+ def result = MySqlClient.execute('load_user', 'fake-secret', [], 'select 1;', null, client(7).path)
+ assertEquals(7, result.exitCode)
+ assertTrue(result.stdout.contains('user="load_user"'))
+ assertFalse(new File(result.stdout.readLines()[0]).exists())
+ }
+
+ @Test
+ void removesCredentialsWhenProcessCannotStart() {
+ def directory = new File(System.getProperty('java.io.tmpdir'))
+ def credentialFiles = { directory.listFiles().findAll { it.name.startsWith('doris-mysql-') }.collect { it.name }.toSet() }
+ def before = credentialFiles()
+ assertThrows(IOException) {
+ MySqlClient.execute('root', 'fake-secret', [], '', null, tempDir.resolve('missing-client').toString())
+ }
+ assertEquals(before, credentialFiles())
+ }
+}
diff --git a/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/RowsetMetaUtilsTest.groovy b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/RowsetMetaUtilsTest.groovy
new file mode 100644
index 00000000000..baa3984591d
--- /dev/null
+++ b/regression-test/framework/src/test/groovy/org/apache/doris/regression/util/RowsetMetaUtilsTest.groovy
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.regression.util
+
+import org.junit.jupiter.api.Test
+
+import java.nio.charset.StandardCharsets
+
+import static org.junit.jupiter.api.Assertions.*
+
+class RowsetMetaUtilsTest {
+ @Test
+ void mergedAndUnsortedRowsetsCoverCommittedVersion() {
+ assertTrue(RowsetMetaUtils.coversVersion(
+ [rowsets: ['[32-61] 3 DATA', '[0-1] 0 DATA', '[2-31] 2 DATA']], 61))
+ }
+
+ @Test
+ void maximumVersionCannotHideGapOrMissingInitialRowset() {
+ assertFalse(RowsetMetaUtils.coversVersion([rowsets: ['[0-1]', '[3-100]']], 11))
+ assertFalse(RowsetMetaUtils.coversVersion([rowsets: ['[2-100]']], 11))
+ assertFalse(RowsetMetaUtils.coversVersion([rowsets: ['[0-1]']], 2))
+ assertFalse(RowsetMetaUtils.coversVersion([rowsets: []], 2))
+ }
+
+ @Test
+ void gapAfterTargetDoesNotInvalidateTargetCoverage() {
+ assertTrue(RowsetMetaUtils.coversVersion([rowsets: ['[0-1]', '[2-6]', '[8-9]']], 6))
+ }
+
+ @Test
+ void malformedMetadataFailsInsteadOfTimingOut() {
+ assertThrows(IllegalArgumentException) { RowsetMetaUtils.coversVersion([:], 2) }
+ assertThrows(IllegalArgumentException) {
+ RowsetMetaUtils.coversVersion([rowsets: ['not a rowset']], 2)
+ }
+ assertThrows(IllegalArgumentException) {
+ RowsetMetaUtils.coversVersion([rowsets: ['[5-2]']], 2)
+ }
+ assertThrows(IllegalArgumentException) { RowsetMetaUtils.coversVersion([rowsets: []], -1) }
+ }
+
+ @Test
+ void keyBoundsPreserveAllBytesAndDoNotMutateInput() {
+ byte[] bytes = [0, 2, 127, -128, -1] as byte[]
+ String encoded = Base64.encoder.encodeToString(bytes)
+ def input = [segments_key_bounds: [[min_key: encoded, max_key: encoded]],
+ segments_key_bounds_truncated: true]
+ def decoded = RowsetMetaUtils.decodeKeyBounds(input)
+ assertArrayEquals(bytes, decoded.segments_key_bounds[0].min_key.getBytes(StandardCharsets.ISO_8859_1))
+ assertEquals(encoded, input.segments_key_bounds[0].min_key)
+ assertTrue(decoded.segments_key_bounds_truncated)
+ assertFalse(decoded.segments_key_bounds_aggregated)
+ assertFalse(input.containsKey('segments_key_bounds_aggregated'))
+ }
+
+ @Test
+ void missingOrInvalidKeyBoundsAreRejected() {
+ assertThrows(IllegalArgumentException) {
+ RowsetMetaUtils.decodeKeyBounds([segments_key_bounds: [[max_key: 'AA==']]])
+ }
+ assertThrows(IllegalArgumentException) {
+ RowsetMetaUtils.decodeKeyBounds([segments_key_bounds: [[min_key: '***', max_key: 'AA==']]])
+ }
+ }
+}
diff --git a/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy b/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
index 66024db5649..9e514f98073 100644
--- a/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
+++ b/regression-test/pipeline/nonConcurrent/conf/regression-conf.groovy
@@ -74,7 +74,7 @@ excludeDirectories = "000_the_start_sentinel_do_not_touch," + // keep this line
"variant_github_events_new_p2," +
"hdfs_vault_p2," +
"nereids_p0/hbo," +
- "cloud_p0/multi_cluster," +
+ "cloud_p0," +
"zzz_the_end_sentinel_do_not_touch"// keep this line as the last line
// for test csv with header
diff --git a/regression-test/plugins/cloud_multi_cluster_state.groovy b/regression-test/plugins/cloud_multi_cluster_state.groovy
new file mode 100644
index 00000000000..977410fc03e
--- /dev/null
+++ b/regression-test/plugins/cloud_multi_cluster_state.groovy
@@ -0,0 +1,136 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+import groovy.json.JsonOutput
+import org.apache.doris.regression.Config
+import org.apache.doris.regression.suite.Suite
+import org.apache.doris.regression.util.MultiClusterStateGuard
+
+// Shared by the load fixture and cases that reuse or deliberately recreate its groups.
+Suite.metaClass.multiClusterBaseline = { ->
+ MultiClusterStateGuard.sharedBaseline(delegate.context.config.multiClusterBes)
+}
+
+Suite.metaClass.checkMultiClusterBaseline = { List<Map> baseline ->
+ def suite = delegate
+ suite.assertTrue(MultiClusterStateGuard.frontendMatches(baseline,
+ suite.sql_return_maparray('SHOW CLUSTERS'), suite.sql_return_maparray('SHOW BACKENDS')),
+ 'Expected shared regression_cluster_name0/1 groups with their configured BEs alive; ' +
+ 'run the multi_cluster_s3_load/load fixture first and check prior suite cleanup')
+}
+
+// Only for serial legacy suites on the dedicated multiClusterInstance.
+// Docker/VCG/upgrade suites have their own cluster lifecycle.
+Suite.metaClass.withRestoredMultiClusterState = { boolean initializeBaseline, Closure action ->
+ def suite = delegate
+ def config = suite.context.config
+ def nodes = MultiClusterStateGuard.configuredBackends(config.multiClusterBes)
+ if (!config.multiClusterInstance) {
+ throw new IllegalStateException('Missing dedicated multi-cluster test instance/BE configuration')
+ }
+ def request = { String operation, Map payload ->
+ def result
+ suite.httpTest {
+ printResponse false
+ endpoint config.metaServiceHttpAddress
+ uri "/MetaService/http/${operation}?token=${config.metaServiceToken}" +
+ "&instance_id=${java.net.URLEncoder.encode(config.multiClusterInstance, 'UTF-8')}"
+ body JsonOutput.toJson([instance_id: config.multiClusterInstance] + payload)
+ check { code, response ->
+ result = suite.parseJson(response)
+ suite.assertTrue(code == 200 && result.code == 'OK',
+ "Multi-cluster ${operation} failed: ${result.code}")
+ }
+ }
+ result
+ }
+ def readInstanceStatus = { request('get_instance', [:]).result.status }
+ if (readInstanceStatus() != 'NORMAL') {
+ throw new IllegalStateException('Multi-cluster tests require a normal dedicated instance')
+ }
+ def fetchClusters = { request('get_cluster', [cloud_unique_id: nodes[0].cloud_unique_id]).result.cluster }
+ def frontends = suite.sql_return_maparray('SHOW FRONTENDS')
+ if (frontends.isEmpty() || frontends.any { !it.Alive.toString().toBoolean() }) {
+ throw new IllegalStateException('All configured frontends must be alive before changing cluster state')
+ }
+ def readDefault = {
+ def rows = suite.sql_return_maparray('SHOW PROPERTY')
+ def row = rows.find { it.Key == 'default_cloud_cluster' }
+ if (row == null) {
+ throw new IllegalStateException('SHOW PROPERTY did not return default_cloud_cluster')
+ }
+ row.Value?.toString() ?: ''
+ }
+ def guard = new MultiClusterStateGuard(
+ listClusters: fetchClusters,
+ restoreInstance: {
+ if (readInstanceStatus() != 'NORMAL') {
+ request('set_instance_status', [op: 'SET_NORMAL'])
+ }
+ },
+ dropCluster: { saved ->
+ request('drop_cluster', [cluster: [type: 'COMPUTE',
+ cluster_name: saved.cluster_name, cluster_id: saved.cluster_id, nodes: []]])
+ },
+ addCluster: { saved -> request('add_cluster', [cluster: saved]) },
+ readDefault: readDefault,
+ setDefault: { String saved ->
+ def literal = saved.replace('\\', '\\\\').replace("'", "''")
+ suite.sql("SET PROPERTY 'default_cloud_cluster' = '${literal}'")
+ },
+ awaitState: { List<Map> expected, String expectedDefault ->
+ long deadline = System.nanoTime() + 120_000_000_000L
+ Throwable lastError = null
+ while (System.nanoTime() < deadline) {
+ try {
+ if (readInstanceStatus() != 'NORMAL') {
+ throw new IllegalStateException('MS instance status has not returned to normal')
+ }
+ def actual = fetchClusters().findAll { it.type == 'COMPUTE' }
+ if (MultiClusterStateGuard.topology(actual) != MultiClusterStateGuard.topology(expected)) {
+ throw new IllegalStateException('MS compute groups do not match the saved topology')
+ }
+ for (def frontend : frontends) {
+ boolean tls = config.otherConfigs.get('enableTLS')?.toString()?.toBoolean() ?: false
+ def url = tls ? Config.buildUrlWithDb(frontend.Host as String, frontend.QueryPort as int,
+ suite.context.dbName, config.otherConfigs.get('keyStorePath')?.toString(),
+ config.otherConfigs.get('keyStorePassword')?.toString(),
+ config.otherConfigs.get('trustStorePath')?.toString(),
+ config.otherConfigs.get('trustStorePassword')?.toString()) :
+ Config.buildUrlWithDb(frontend.Host as String, frontend.QueryPort as int, suite.context.dbName)
+ url += '&connectTimeout=5000&socketTimeout=5000'
+ suite.connect(config.jdbcUser, config.jdbcPassword, url) {
+ def visibleGroups = suite.sql_return_maparray('SHOW CLUSTERS')
+ def visibleNodes = suite.sql_return_maparray('SHOW BACKENDS')
+ if (!MultiClusterStateGuard.frontendMatches(expected, visibleGroups, visibleNodes)) {
+ throw new IllegalStateException("FE ${frontend.Host} has not restored groups/nodes/heartbeats")
+ }
+ if (expectedDefault != null && readDefault() != expectedDefault) {
+ throw new IllegalStateException("FE ${frontend.Host} has not restored the user default group")
+ }
+ }
+ }
+ return
+ } catch (Exception e) {
+ lastError = e
+ }
+ Thread.sleep(500)
+ }
+ throw new IllegalStateException('Timed out restoring multi-cluster state on MS and all frontends', lastError)
+ }
+ )
+ guard.run(initializeBaseline, action)
+}
diff --git a/regression-test/plugins/plugin_compaction.groovy b/regression-test/plugins/plugin_compaction.groovy
index 8cf6b1f0f87..b3e600ad958 100644
--- a/regression-test/plugins/plugin_compaction.groovy
+++ b/regression-test/plugins/plugin_compaction.groovy
@@ -19,45 +19,57 @@ import org.apache.doris.regression.suite.Suite
import java.util.concurrent.TimeUnit
import org.awaitility.Awaitility;
-Suite.metaClass.be_get_compaction_status{ String ip, String port, String tablet_id /* param */->
- return curl("GET", String.format("http://%s:%s/api/compaction/run_status?tablet_id=%s", ip, port, tablet_id))
+Suite.metaClass.be_get_compaction_status{ String ip, String port, String tablet_id,
+ Integer timeout_sec = 10, Integer max_retries = 10 /* param */->
+ return delegate.curl("GET", String.format("http://%s:%s/api/compaction/run_status?tablet_id=%s", ip, port, tablet_id),
+ null, timeout_sec, delegate.context.config.feHttpUser,
+ delegate.context.config.feHttpPassword, max_retries)
}
Suite.metaClass.be_get_overall_compaction_status{ String ip, String port /* param */->
- return curl("GET", String.format("http://%s:%s/api/compaction/run_status", ip, port))
+ return delegate.curl("GET", String.format("http://%s:%s/api/compaction/run_status", ip, port),
+ null, 10, delegate.context.config.feHttpUser, delegate.context.config.feHttpPassword, 10)
}
-Suite.metaClass.be_show_tablet_status{ String ip, String port, String tablet_id /* param */->
- return curl("GET", String.format("http://%s:%s/api/compaction/show?tablet_id=%s", ip, port, tablet_id))
+Suite.metaClass.be_show_tablet_status{ String ip, String port, String tablet_id,
+ Integer timeout_sec = 10, Integer max_retries = 10 /* param */->
+ return delegate.curl("GET", String.format("http://%s:%s/api/compaction/show?tablet_id=%s", ip, port, tablet_id),
+ null, timeout_sec, delegate.context.config.feHttpUser,
+ delegate.context.config.feHttpPassword, max_retries)
}
Suite.metaClass._be_run_compaction = { String ip, String port, String tablet_id, String compact_type ->
- return curl("POST", String.format("http://%s:%s/api/compaction/run?tablet_id=%s&compact_type=%s",
- ip, port, tablet_id, compact_type))
+ return delegate.curl("POST", String.format("http://%s:%s/api/compaction/run?tablet_id=%s&compact_type=%s",
+ ip, port, tablet_id, compact_type), null, 10, delegate.context.config.feHttpUser,
+ delegate.context.config.feHttpPassword, 10)
}
Suite.metaClass.be_run_base_compaction = { String ip, String port, String tablet_id /* param */->
- return _be_run_compaction(ip, port, tablet_id, "base")
+ return delegate._be_run_compaction(ip, port, tablet_id, "base")
}
logger.info("Added 'be_run_base_compaction' function to Suite")
Suite.metaClass.be_run_cumulative_compaction = { String ip, String port, String tablet_id /* param */->
- return _be_run_compaction(ip, port, tablet_id, "cumulative")
+ return delegate._be_run_compaction(ip, port, tablet_id, "cumulative")
}
logger.info("Added 'be_run_cumulative_compaction' function to Suite")
Suite.metaClass.be_run_full_compaction = { String ip, String port, String tablet_id /* param */->
- return _be_run_compaction(ip, port, tablet_id, "full")
+ return delegate._be_run_compaction(ip, port, tablet_id, "full")
}
Suite.metaClass.be_run_full_compaction_by_table_id = { String ip, String port, String table_id /* param */->
- return curl("POST", String.format("http://%s:%s/api/compaction/run?table_id=%s&compact_type=full", ip, port, table_id))
+ return delegate.curl("POST", String.format("http://%s:%s/api/compaction/run?table_id=%s&compact_type=full",
+ ip, port, table_id), null, 10, delegate.context.config.feHttpUser,
+ delegate.context.config.feHttpPassword, 10)
}
logger.info("Added 'be_run_full_compaction' function to Suite")
-Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compaction_type, int timeout_seconds=300, String[] ignored_errors=[] ->
+Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compaction_type, int timeout_seconds=300,
+ String[] ignored_errors=[], Collection tablet_ids=[],
+ String[] retryable_errors=[] ->
if (!(compaction_type in ["cumulative", "base", "full"])) {
throw new IllegalArgumentException("invalid compaction type: ${compaction_type}, supported types: cumulative, base, full")
}
@@ -66,6 +78,13 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
def backendId_to_backendHttpPort = [:]
getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
def tablets = sql_return_maparray """show tablets from ${table_name}"""
+ if (!tablet_ids.isEmpty()) {
+ def requestedTabletIds = tablet_ids.collect { "${it}" }.toSet()
+ tablets = tablets.findAll { requestedTabletIds.contains("${it.TabletId}") }
+ def foundTabletIds = tablets.collect { "${it.TabletId}" }.toSet()
+ assert foundTabletIds == requestedTabletIds:
+ "Unable to find all requested tablets for ${table_name}, requested: ${requestedTabletIds}, found: ${foundTabletIds}"
+ }
def exit_code, stdout, stderr
def auto_compaction_disabled = sql("show create table ${table_name}")[0][1].contains('"disable_auto_compaction" = "true"')
@@ -76,7 +95,7 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
for (tablet in tablets) {
def be_host = backendId_to_backendIP["${tablet.BackendId}"]
def be_port = backendId_to_backendHttpPort["${tablet.BackendId}"]
- (exit_code, stdout, stderr) = be_show_tablet_status(be_host, be_port, tablet.TabletId)
+ (exit_code, stdout, stderr) = delegate.be_show_tablet_status(be_host, be_port, tablet.TabletId)
assert exit_code == 0: "get tablet status failed, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}"
def tabletStatus = parseJson(stdout.trim())
@@ -87,23 +106,38 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
for (tablet in tablets) {
def be_host = backendId_to_backendIP["${tablet.BackendId}"]
def be_port = backendId_to_backendHttpPort["${tablet.BackendId}"]
- switch (compaction_type) {
- case "cumulative":
- (exit_code, stdout, stderr) = be_run_cumulative_compaction(be_host, be_port, tablet.TabletId)
- break
- case "base":
- (exit_code, stdout, stderr) = be_run_base_compaction(be_host, be_port, tablet.TabletId)
- break
- case "full":
- (exit_code, stdout, stderr) = be_run_full_compaction(be_host, be_port, tablet.TabletId)
- break
- }
- assert exit_code == 0: "trigger compaction failed, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}"
- def trigger_status = parseJson(stdout.trim())
- if (trigger_status.status.toLowerCase() != "success") {
+ long triggerDeadline = System.currentTimeMillis() + timeout_seconds * 1000L
+ while (true) {
+ switch (compaction_type) {
+ case "cumulative":
+ (exit_code, stdout, stderr) = delegate.be_run_cumulative_compaction(be_host, be_port, tablet.TabletId)
+ break
+ case "base":
+ (exit_code, stdout, stderr) = delegate.be_run_base_compaction(be_host, be_port, tablet.TabletId)
+ break
+ case "full":
+ (exit_code, stdout, stderr) = delegate.be_run_full_compaction(be_host, be_port, tablet.TabletId)
+ break
+ }
+ assert exit_code == 0: "trigger compaction failed, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}"
+ def trigger_status = parseJson(stdout.trim())
def status_lower = trigger_status.status.toLowerCase()
- if (status_lower == "already_exist") {
- triggered_tablets.add(tablet) // compaction already in queue, treat it as successfully triggered
+ if (status_lower == "success" || status_lower == "already_exist") {
+ triggered_tablets.add(tablet)
+ break
+ } else if (retryable_errors.any { error -> status_lower.contains(error.toLowerCase()) }) {
+ assert System.currentTimeMillis() < triggerDeadline:
+ "retry trigger compaction timeout, be host: ${be_host}, tablet id: ${tablet.TabletId}, status: ${trigger_status.status}"
+ logger.info("retry transient compaction trigger failure, be host: ${be_host}, tablet id: ${tablet.TabletId}, status: ${trigger_status.status}")
+ // The transient trigger failure updates the tablet's failure timestamp.
+ // Refresh the baseline so it cannot be mistaken for completion of the
+ // later successful asynchronous trigger.
+ (exit_code, stdout, stderr) = delegate.be_show_tablet_status(be_host, be_port, tablet.TabletId)
+ assert exit_code == 0:
+ "refresh tablet status failed, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}"
+ be_tablet_compaction_status.put("${be_host}-${tablet.TabletId}", parseJson(stdout.trim()))
+ sleep(1000)
+ continue
} else if (!auto_compaction_disabled) {
// ignore the error if auto compaction enabled
} else if (status_lower.contains("e-2000") || status_lower.contains("e-2010")
@@ -119,8 +153,7 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
} else {
throw new Exception("trigger compaction failed, be host: ${be_host}, tablet id: ${tablet.TabletId}, status: ${trigger_status.status}")
}
- } else {
- triggered_tablets.add(tablet)
+ break
}
}
@@ -136,24 +169,34 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
return null
}
}
- Awaitility.await().atMost(timeout_seconds, TimeUnit.SECONDS).pollInterval(1, TimeUnit.SECONDS).until(() -> {
+ // Parallel suites own their worker failures; do not capture uncaught exceptions from other suites.
+ Awaitility.await().dontCatchUncaughtExceptions().atMost(timeout_seconds, TimeUnit.SECONDS).pollInterval(1, TimeUnit.SECONDS).until(() -> {
for (tablet in triggered_tablets) {
def be_host = backendId_to_backendIP["${tablet.BackendId}"]
def be_port = backendId_to_backendHttpPort["${tablet.BackendId}"]
- (exit_code, stdout, stderr) = be_get_compaction_status(be_host, be_port, tablet.TabletId)
- assert exit_code == 0: "get compaction status failed, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}"
+ // Awaitility owns the retry loop. Keep each HTTP probe bounded so an
+ // inner curl retry cannot consume the entire compaction timeout.
+ (exit_code, stdout, stderr) = delegate.be_get_compaction_status(be_host, be_port, tablet.TabletId, 5, 1)
+ if (exit_code != 0) {
+ logger.warn("get compaction status failed, will retry, be host: ${be_host}, tablet id: ${tablet.TabletId}, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}")
+ return false
+ }
def compactionStatus = parseJson(stdout.trim())
assert compactionStatus.status.toLowerCase() == "success": "compaction failed, be host: ${be_host}, tablet id: ${tablet.TabletId}, status: ${compactionStatus.status}"
// running is true means compaction is still running
running = compactionStatus.run_status
if (!is_time_series_compaction) {
- (exit_code, stdout, stderr) = be_show_tablet_status(be_host, be_port, tablet.TabletId)
- assert exit_code == 0: "get tablet status failed, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}"
+ (exit_code, stdout, stderr) = delegate.be_show_tablet_status(be_host, be_port, tablet.TabletId, 5, 1)
+ if (exit_code != 0) {
+ logger.warn("get tablet status failed, will retry, be host: ${be_host}, tablet id: ${tablet.TabletId}, exit code: ${exit_code}, stdout: ${stdout}, stderr: ${stderr}")
+ return false
+ }
def tabletStatus = parseJson(stdout.trim())
def oldStatus = be_tablet_compaction_status.get("${be_host}-${tablet.TabletId}")
- // last compaction success/failure time isn't updated, indicates compaction is not started(so we treat it as running and wait)
+ // The HTTP trigger is asynchronous in cloud mode. run_status may already be
+ // false at the first poll, so also require a completion marker for this run.
def handedOffToBaseCompactionAfterDeleteVersion = false
def completedByBaseCompactionAfterDeleteVersion = false
if (compaction_type == "cumulative") {
@@ -163,9 +206,6 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
def baseSuccessTimeChanged = oldStatus["last base success time"] != tabletStatus["last base success time"]
def cumulativeSuccessTimeChanged =
oldStatus["last cumulative success time"] != tabletStatus["last cumulative success time"]
- // E-2010 advances the cumulative point and lets base compaction handle delete-version rowsets.
- // In some timing windows, base success is already visible in the cached old status while
- // cumulative success advances later, so accept either success signal but not failure time alone.
handedOffToBaseCompactionAfterDeleteVersion = lastCumulativeStatus.contains("e-2010") &&
oldCumulativePoint != null && newCumulativePoint != null &&
newCumulativePoint > oldCumulativePoint
@@ -173,19 +213,21 @@ Suite.metaClass.trigger_and_wait_compaction = { String table_name, String compac
handedOffToBaseCompactionAfterDeleteVersion &&
(baseSuccessTimeChanged || cumulativeSuccessTimeChanged)
}
- def success_time_unchanged = (oldStatus["last ${compaction_type} success time"] == tabletStatus["last ${compaction_type} success time"])
- def failure_time_unchanged = (oldStatus["last ${compaction_type} failure time"] == tabletStatus["last ${compaction_type} failure time"])
- def currentCompactionTimestampChanged = !success_time_unchanged || !failure_time_unchanged
+ def successTimeUnchanged = oldStatus["last ${compaction_type} success time"] ==
+ tabletStatus["last ${compaction_type} success time"]
+ def failureTimeUnchanged = oldStatus["last ${compaction_type} failure time"] ==
+ tabletStatus["last ${compaction_type} failure time"]
+ def completionTimestampChanged = !successTimeUnchanged || !failureTimeUnchanged
def compactionFinished = completedByBaseCompactionAfterDeleteVersion ||
- (!handedOffToBaseCompactionAfterDeleteVersion && currentCompactionTimestampChanged)
+ (!handedOffToBaseCompactionAfterDeleteVersion && completionTimestampChanged)
running = running || !compactionFinished
if (running) {
logger.info("compaction is still running, be host: ${be_host}, tablet id: ${tablet.TabletId}, run status: ${compactionStatus.run_status}, old status: ${oldStatus}, new status: ${tabletStatus}")
return false
}
} else {
- // time series compaction sometimes doesn't update compaction success time
- // so we solely check run_status for it
+ // Time series compaction sometimes doesn't update compaction success time,
+ // so solely check run_status for it.
if (running) {
logger.info("compaction is still running, be host: ${be_host}, tablet id: ${tablet.TabletId}")
return false
diff --git a/regression-test/plugins/plugin_curl_requester.groovy b/regression-test/plugins/plugin_curl_requester.groovy
index a0a813fb800..e16945616db 100644
--- a/regression-test/plugins/plugin_curl_requester.groovy
+++ b/regression-test/plugins/plugin_curl_requester.groovy
@@ -146,7 +146,7 @@ Suite.metaClass.http_client = { String method, String url /* param */ ->
logger.info("Added 'http_client' function to Suite")
-Suite.metaClass.curl = { String method, String url, String body = null, Integer timeoutSec = 10, String user = "", String pwd = ""->
+Suite.metaClass.curl = { String method, String url, String body = null, Integer timeoutSec = 10, String user = "", String pwd = "", Integer maxRetries = 10 ->
Suite suite = delegate as Suite
if (method != "GET" && method != "POST") {
throw new Exception(String.format("invalid curl method: %s", method))
@@ -155,7 +155,6 @@ Suite.metaClass.curl = { String method, String url, String body = null, Integer
throw new Exception("invalid curl url, blank")
}
- Integer maxRetries = 10; // Maximum number of retries
Integer retryCount = 0; // Current retry count
Integer sleepTime = 5000; // Sleep time in milliseconds
diff --git a/regression-test/plugins/plugin_index_change.groovy b/regression-test/plugins/plugin_index_change.groovy
index eb47798f274..ad7edd1a12c 100644
--- a/regression-test/plugins/plugin_index_change.groovy
+++ b/regression-test/plugins/plugin_index_change.groovy
@@ -16,6 +16,7 @@
// under the License.
import org.apache.doris.regression.suite.Suite
+import org.apache.doris.regression.util.Http
import groovy.json.JsonSlurper
import java.util.regex.Matcher;
@@ -75,7 +76,7 @@ Suite.metaClass.wait_for_last_schema_change_finish = {table_name, OpTimeout ->
Suite.metaClass.http_get = { url ->
def dst = 'http://' + context.config.feHttpAddress
- def conn = new URL(dst + url).openConnection()
+ def conn = Http.openConnection(dst + url)
conn.setRequestMethod("GET")
def encoding = Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
(context.config.feHttpPassword == null ? "" : context.config.feHttpPassword)).getBytes("UTF-8"))
@@ -137,4 +138,4 @@ Suite.metaClass.check_bf_index_filter_rows = { sql, expectedRowsBfFiltered ->
log.info("filter number:{}", number)
assertEquals(expectedRowsBfFiltered, number)
}
-}
\ No newline at end of file
+}
diff --git a/regression-test/suites/account_p0/test_auth_show.groovy b/regression-test/suites/account_p0/test_auth_show.groovy
index d9d4f3fa7f1..80050aa06c8 100644
--- a/regression-test/suites/account_p0/test_auth_show.groovy
+++ b/regression-test/suites/account_p0/test_auth_show.groovy
@@ -45,7 +45,7 @@ suite("test_auth_show", "account") {
// With select priv for table, should be able to see db
sql """GRANT SELECT_PRIV ON ${dbName}.${tableName} TO ${user}"""
- def result1 = connect(user, "${pwd}", url) {
+ def result1 = connectToDoris(user, "${pwd}", url) {
sql """show databases like '${dbName}'"""
}
assertEquals(result1.size(), 1)
@@ -53,7 +53,7 @@ suite("test_auth_show", "account") {
// With show_view priv for table, should be able to see db
sql """GRANT SHOW_VIEW_PRIV ON ${dbName}.${tableName} TO ${user}"""
- def result2 = connect(user, "${pwd}", url) {
+ def result2 = connectToDoris(user, "${pwd}", url) {
sql """show databases like '${dbName}'"""
}
assertEquals(result2.size(), 1)
diff --git a/regression-test/suites/account_p0/test_grant_priv.groovy b/regression-test/suites/account_p0/test_grant_priv.groovy
index 21d26dd30d8..f23134bb214 100644
--- a/regression-test/suites/account_p0/test_grant_priv.groovy
+++ b/regression-test/suites/account_p0/test_grant_priv.groovy
@@ -40,7 +40,7 @@ suite("test_grant_priv") {
// test only have select_priv, can not grant to other user
sql """grant select_priv on ${dbName}.* to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
try {
sql """grant select_priv on ${dbName}.* to ${user2}"""
Assert.fail("can not grant to other user");
@@ -51,7 +51,7 @@ suite("test_grant_priv") {
// test both have select_priv and grant_priv , can grant to other user
sql """grant grant_priv on ${dbName}.* to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
try {
sql """grant select_priv on ${dbName}.* to ${user2}"""
} catch (Exception e) {
@@ -96,7 +96,7 @@ suite("test_grant_priv") {
sql """grant select_priv on *.*.* to role ${role2}"""
sql """grant '${role1}' to ${user1}"""
// test only have role1 can not grant
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
test {
sql """grant select_priv on *.*.* to ${user2}"""
exception "denied"
@@ -105,7 +105,7 @@ suite("test_grant_priv") {
sql """revoke '${role1}' from ${user1}"""
sql """grant '${role2}' to ${user1}"""
// test only have role2 can not grant
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
test {
sql """grant select_priv on *.*.* to ${user2}"""
exception "denied"
@@ -113,7 +113,7 @@ suite("test_grant_priv") {
}
// test both have role1 and role2 can grant to other
sql """grant '${role1}' to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
sql """grant select_priv on *.*.* to ${user2}"""
}
sql """drop user if exists ${user1}"""
diff --git a/regression-test/suites/account_p0/test_grant_priv_resource.groovy b/regression-test/suites/account_p0/test_grant_priv_resource.groovy
index c8eea500a07..a9c10da6ef7 100644
--- a/regression-test/suites/account_p0/test_grant_priv_resource.groovy
+++ b/regression-test/suites/account_p0/test_grant_priv_resource.groovy
@@ -33,7 +33,7 @@ suite("test_grant_priv_resource") {
// test only have USAGE_PRIV, can not grant to other user
sql """grant USAGE_PRIV on RESOURCE ${resource1} to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
try {
sql """grant USAGE_PRIV on RESOURCE ${resource1} to ${user2}"""
Assert.fail("can not grant to other user");
@@ -44,7 +44,7 @@ suite("test_grant_priv_resource") {
// test both have USAGE_PRIV and grant_priv , can grant to other user
sql """grant grant_priv on RESOURCE * to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
try {
sql """grant USAGE_PRIV on RESOURCE ${resource1} to ${user2}"""
} catch (Exception e) {
diff --git a/regression-test/suites/account_p0/test_grant_priv_workload.groovy b/regression-test/suites/account_p0/test_grant_priv_workload.groovy
index 8fa9f901a3e..f0f831a55cb 100644
--- a/regression-test/suites/account_p0/test_grant_priv_workload.groovy
+++ b/regression-test/suites/account_p0/test_grant_priv_workload.groovy
@@ -45,7 +45,7 @@ suite("test_grant_priv_workload") {
// test only have USAGE_PRIV, can not grant to other user
sql """grant USAGE_PRIV on WORKLOAD GROUP ${workload1} to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
try {
sql """grant USAGE_PRIV on WORKLOAD GROUP ${workload1} to ${user2}"""
Assert.fail("can not grant to other user");
@@ -56,7 +56,7 @@ suite("test_grant_priv_workload") {
// test both have USAGE_PRIV and grant_priv , can grant to other user
sql """grant grant_priv on WORKLOAD GROUP ${workload1} to ${user1}"""
- connect(user1, "${pwd}", url) {
+ connectToDoris(user1, "${pwd}", url) {
try {
sql """grant USAGE_PRIV on WORKLOAD GROUP ${workload1} to ${user2}"""
} catch (Exception e) {
diff --git a/regression-test/suites/account_p0/test_nereids_authentication.groovy b/regression-test/suites/account_p0/test_nereids_authentication.groovy
index 4b8c644495c..70648ee27b4 100644
--- a/regression-test/suites/account_p0/test_nereids_authentication.groovy
+++ b/regression-test/suites/account_p0/test_nereids_authentication.groovy
@@ -54,26 +54,26 @@ suite("test_nereids_authentication", "query") {
def tokens = context.config.jdbcUrl.split('/')
def url=tokens[0] + "//" + tokens[2] + "/" + dbName + "?"
- def result = connect(user, 'Doris_123456', url) {
+ def result = connectToDoris(user, 'Doris_123456', url) {
sql "SELECT * FROM ${tableName1}"
}
assertEquals(result.size(), 0)
- connect(user, 'Doris_123456', url) {
+ connectToDoris(user, 'Doris_123456', url) {
test {
sql "SELECT * FROM ${tableName2}"
exception "denied"
}
}
- connect(user, 'Doris_123456', url) {
+ connectToDoris(user, 'Doris_123456', url) {
test {
sql "SELECT count(*) FROM ${tableName2}"
exception "denied"
}
}
- connect(user, 'Doris_123456', url) {
+ connectToDoris(user, 'Doris_123456', url) {
test {
sql "SELECT * FROM ${tableName1}, ${tableName2} WHERE ${tableName1}.`key` = ${tableName2}.`key`"
exception "denied"
@@ -81,14 +81,14 @@ suite("test_nereids_authentication", "query") {
}
sql "GRANT SELECT_PRIV ON internal.${dbName}.${tableName2} TO ${user}"
- connect(user, 'Doris_123456', url) {
+ connectToDoris(user, 'Doris_123456', url) {
sql "SELECT * FROM ${tableName2}"
}
assertEquals(result.size(), 0)
- connect(user, 'Doris_123456', url) {
+ connectToDoris(user, 'Doris_123456', url) {
sql "SELECT count(*) FROM ${tableName2}"
}
- connect(user, 'Doris_123456', url) {
+ connectToDoris(user, 'Doris_123456', url) {
sql "SELECT * FROM ${tableName1}, ${tableName2} WHERE ${tableName1}.`key` = ${tableName2}.`key`"
}
assertEquals(result.size(), 0)
diff --git a/regression-test/suites/account_p0/test_nereids_row_policy.groovy b/regression-test/suites/account_p0/test_nereids_row_policy.groovy
index 4463a41c90b..0ed3e8dc9b5 100644
--- a/regression-test/suites/account_p0/test_nereids_row_policy.groovy
+++ b/regression-test/suites/account_p0/test_nereids_row_policy.groovy
@@ -32,10 +32,10 @@ suite("test_nereids_row_policy") {
}
def assertQueryResult = { size ->
- def result = connect(user, '123abc!@#', url) {
+ def result = connectToDoris(user, '123abc!@#', url) {
sql "SELECT * FROM ${tableName}"
}
- connect(user, '123abc!@#', url) {
+ connectToDoris(user, '123abc!@#', url) {
test {
sql "SELECT * FROM ${viewName}"
exception "does not have privilege for"
diff --git a/regression-test/suites/account_p0/test_set_password.groovy b/regression-test/suites/account_p0/test_set_password.groovy
index 65f08d5f40e..8ef1bbc2aa3 100644
--- a/regression-test/suites/account_p0/test_set_password.groovy
+++ b/regression-test/suites/account_p0/test_set_password.groovy
@@ -38,7 +38,7 @@ suite("test_set_password", "account") {
def tokens = context.config.jdbcUrl.split('/')
def url = tokens[0] + "//" + tokens[2] + "/" + "information_schema" + "?"
- connect(user, password1, url) {
+ connectToDoris(user, password1, url) {
def result = sql "SELECT 1"
assertEquals(1, result[0][0])
}
@@ -48,14 +48,14 @@ suite("test_set_password", "account") {
sql "SET PASSWORD FOR '${user}'@'%' = PASSWORD('${password2}')"
// verify login with new password
- connect(user, password2, url) {
+ connectToDoris(user, password2, url) {
def result = sql "SELECT 1"
assertEquals(1, result[0][0])
}
// verify old password no longer works
try {
- connect(user, password1, url) {
+ connectToDoris(user, password1, url) {
sql "SELECT 1"
}
assertTrue(false, "Old password should not work after SET PASSWORD")
diff --git a/regression-test/suites/account_p0/test_view_row_policy.groovy b/regression-test/suites/account_p0/test_view_row_policy.groovy
index 6a7b3d02109..45e5b70fcfa 100644
--- a/regression-test/suites/account_p0/test_view_row_policy.groovy
+++ b/regression-test/suites/account_p0/test_view_row_policy.groovy
@@ -65,7 +65,7 @@ suite("test_view_row_policy") {
}
// no policy
- connect(user, "${pwd}", url) {
+ connectToDoris(user, "${pwd}", url) {
order_qt_no_policy "SELECT * FROM ${viewName}"
}
@@ -74,7 +74,7 @@ suite("test_view_row_policy") {
CREATE ROW POLICY IF NOT EXISTS ${tablePolcyName} ON ${dbName}.${tableName}
AS RESTRICTIVE TO ${user} USING (k=1)
"""
- connect(user, "${pwd}", url) {
+ connectToDoris(user, "${pwd}", url) {
order_qt_table_policy "SELECT * FROM ${viewName}"
}
@@ -83,7 +83,7 @@ suite("test_view_row_policy") {
CREATE ROW POLICY IF NOT EXISTS ${viewPolcyName} ON ${dbName}.${viewName}
AS RESTRICTIVE TO ${user} USING (v=2)
"""
- connect(user, "${pwd}", url) {
+ connectToDoris(user, "${pwd}", url) {
order_qt_view_policy "SELECT * FROM ${viewName}"
}
diff --git a/regression-test/suites/ann_index_p0/ann_topn_small_candidate_fallback.groovy b/regression-test/suites/ann_index_p0/ann_topn_small_candidate_fallback.groovy
index b4f0e29085b..92e9fa56d0e 100644
--- a/regression-test/suites/ann_index_p0/ann_topn_small_candidate_fallback.groovy
+++ b/regression-test/suites/ann_index_p0/ann_topn_small_candidate_fallback.groovy
@@ -16,27 +16,16 @@
// under the License.
import groovy.json.JsonSlurper
+import org.apache.doris.regression.util.Http
def getProfileList = {
- def dst = "http://" + context.config.feHttpAddress
- def conn = new URL(dst + "/rest/v1/query_profile").openConnection()
- conn.setRequestMethod("GET")
- def encoding = Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
- (context.config.feHttpPassword == null ? "" : context.config.feHttpPassword))
- .getBytes("UTF-8"))
- conn.setRequestProperty("Authorization", "Basic ${encoding}")
- return conn.getInputStream().getText()
+ return Http.GET("http://${context.config.feHttpAddress}/rest/v1/query_profile", false, false,
+ context.config.feHttpUser, context.config.feHttpPassword)
}
def getProfile = { id ->
- def dst = "http://" + context.config.feHttpAddress
- def conn = new URL(dst + "/api/profile/text/?query_id=$id").openConnection()
- conn.setRequestMethod("GET")
- def encoding = Base64.getEncoder().encodeToString((context.config.feHttpUser + ":" +
- (context.config.feHttpPassword == null ? "" : context.config.feHttpPassword))
- .getBytes("UTF-8"))
- conn.setRequestProperty("Authorization", "Basic ${encoding}")
- return conn.getInputStream().getText()
+ return Http.GET("http://${context.config.feHttpAddress}/api/profile/text/?query_id=$id", false, false,
+ context.config.feHttpUser, context.config.feHttpPassword)
}
def extractCounterValue = { String profileText, String counterName ->
diff --git a/regression-test/suites/arrow_flight_sql_p0/test_auth_remote_ip.groovy b/regression-test/suites/arrow_flight_sql_p0/test_auth_remote_ip.groovy
index 1d9f02260da..707dff1e1bf 100644
--- a/regression-test/suites/arrow_flight_sql_p0/test_auth_remote_ip.groovy
+++ b/regression-test/suites/arrow_flight_sql_p0/test_auth_remote_ip.groovy
@@ -57,8 +57,7 @@ suite("test_auth_remote_ip", "arrow_flight_sql") {
Class.forName("org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver")
String arrowFlightSqlHost = context.config.otherConfigs.get("extArrowFlightSqlHost")
String arrowFlightSqlPort = context.config.otherConfigs.get("extArrowFlightSqlPort")
- String arrowFlightSqlUrl = "jdbc:arrow-flight-sql://${arrowFlightSqlHost}:${arrowFlightSqlPort}" +
- "/?useServerPrepStmts=false&useSSL=false&useEncryption=false"
+ String arrowFlightSqlUrl = context.config.getArrowFlightSqlJdbcUrl(arrowFlightSqlHost, arrowFlightSqlPort)
Connection conn = DriverManager.getConnection(arrowFlightSqlUrl, user, password)
try {
diff --git a/regression-test/suites/audit/test_audit_log_queue_time.groovy b/regression-test/suites/audit/test_audit_log_queue_time.groovy
index e40d1688c42..e4cc20e9c52 100644
--- a/regression-test/suites/audit/test_audit_log_queue_time.groovy
+++ b/regression-test/suites/audit/test_audit_log_queue_time.groovy
@@ -17,6 +17,7 @@
suite("test_audit_log_queue_time", "nonConcurrent") {
// Check admin privilege
+ def originalAuditPlugin = sql("show global variables like 'enable_audit_plugin'")[0][1]
try {
sql "set global enable_audit_plugin = true"
} catch (Exception e) {
@@ -27,103 +28,148 @@ suite("test_audit_log_queue_time", "nonConcurrent") {
def tableName = "audit_queue_time_test"
def wgName = "test_queue_time_wg"
- def testMarker = UUID.randomUUID().toString().substring(0, 8)
+ def maxConcurrency = 1
+ def queueTimeoutMs = 30000
+ def guaranteedQueueTimeMs = 1000
+ def blockerSleepTime = 10
+ def markerPrefix = UUID.randomUUID().toString().substring(0, 8)
+ def blockerMarker = "${markerPrefix}_blocker"
+ def queuedMarker = "${markerPrefix}_queued"
+ def threads = []
+ def queryErrors = Collections.synchronizedList([])
- // Cleanup environment
- sql "drop table if exists ${tableName}"
- sql "drop workload group if exists ${wgName}"
+ def getQueueState = {
+ def row = sql("show workload groups").find { it[1].toString() == wgName }
+ if (row == null) {
+ return null
+ }
+ return [
+ running: row[row.size() - 2] as int,
+ waiting: row[row.size() - 1] as int
+ ]
+ }
- // Create test table
- sql """
- CREATE TABLE `${tableName}` (
- `id` bigint,
- `name` varchar(32)
- ) ENGINE=OLAP
- DUPLICATE KEY(`id`)
- DISTRIBUTED BY HASH(`id`) BUCKETS 1
- PROPERTIES ("replication_allocation" = "tag.location.default: 1")
- """
+ def waitForQueueState = { int expectedRunning, int expectedWaiting ->
+ def state = null
+ for (int i = 0; i < 100; i++) {
+ state = getQueueState()
+ if (state != null
+ && state.running == expectedRunning
+ && state.waiting == expectedWaiting) {
+ return
+ }
+ sleep(100)
+ }
+ throw new RuntimeException("workload group ${wgName} did not reach "
+ + "running=${expectedRunning}, waiting=${expectedWaiting}; last state=${state}")
+ }
+
+ try {
+ // Cleanup environment
+ sql "drop table if exists ${tableName}"
+ sql "drop workload group if exists ${wgName}"
+
+ // Create test table
+ sql """
+ CREATE TABLE `${tableName}` (
+ `id` bigint,
+ `name` varchar(32)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+
+ sql "insert into ${tableName} values (1, 'test')"
+
+ // max_concurrency=1 ensures that the second query enters the queue.
+ sql """
+ create workload group ${wgName}
+ properties (
+ 'max_concurrency' = '${maxConcurrency}',
+ 'max_queue_size' = '10',
+ 'queue_timeout' = '${queueTimeoutMs}'
+ )
+ """
+
+ // Wait for workload group to take effect.
+ Thread.sleep(5000)
+
+ // Truncate audit_log for easier testing.
+ sql "truncate table __internal_schema.audit_log"
+
+ // Occupy the only running slot before submitting the query whose queue time is audited.
+ threads << Thread.start {
+ try {
+ sql "set workload_group=${wgName}"
+ sql """
+ select sleep(${blockerSleepTime}), '${blockerMarker}' as marker
+ from ${tableName} limit 1
+ """
+ } catch (Throwable t) {
+ queryErrors.add(t)
+ }
+ }
- sql "insert into ${tableName} values (1, 'test')"
+ waitForQueueState(maxConcurrency, 0)
- def maxConcurrency = 1
- // Create workload group: max_concurrency=1 ensures queries queue up
- sql """
- create workload group ${wgName}
- properties (
- 'max_concurrency' = '${maxConcurrency}',
- 'max_queue_size' = '10',
- 'queue_timeout' = '30000'
- )
- """
-
- // Wait for workload group to take effect
- Thread.sleep(5000)
-
- // Truncate audit_log for easier testing
- sql "truncate table __internal_schema.audit_log"
-
- // Submit concurrent queries with marker for later lookup
- def sqlSleepTime = 5
- def queuedSqlCnt = 1
- def threads = []
- for (int i = 0; i < maxConcurrency + queuedSqlCnt; i++) {
- def idx = i
threads << Thread.start {
try {
sql "set workload_group=${wgName}"
- // Use sleep function to simulate long query, ensuring subsequent queries need to queue
sql """
- select sleep(${sqlSleepTime}), '${testMarker}_${idx}' as marker
+ select id, '${queuedMarker}' as marker
from ${tableName} limit 1
"""
- } catch (Exception e) {
- log.warn("Query ${idx} failed: ${e.getMessage()}")
+ } catch (Throwable t) {
+ queryErrors.add(t)
}
}
- }
- // Wait for all queries to complete
- threads.each { it.join() }
-
- // Wait for audit log to flush
- Thread.sleep(5000)
- sql "call flush_audit_log()"
- Thread.sleep(5000)
-
- // Verify queue_time_ms column exists
- def schemaResult = sql "desc internal.__internal_schema.audit_log"
- def hasQueueTimeMs = schemaResult.any { it[0] == "queue_time_ms" }
- assertTrue(hasQueueTimeMs)
-
- // check result
- def retry = 10
- def query = """
- select query_id, queue_time_ms, stmt
- from __internal_schema.audit_log
- where stmt like '%${testMarker}%'
- and queue_time_ms > 0
- order by time
- """
- def auditResult = sql "${query}"
-
- while (auditResult.isEmpty()) {
- if (retry-- < 0) {
- throw new RuntimeException("It has retried a few but still failed, you need to check it")
+ waitForQueueState(maxConcurrency, 1)
+ sleep(guaranteedQueueTimeMs)
+ waitForQueueState(maxConcurrency, 1)
+
+ threads.each { it.join() }
+ assertTrue(queryErrors.isEmpty(), "query failures: ${queryErrors}")
+
+ // Verify queue_time_ms column exists.
+ def schemaResult = sql "desc internal.__internal_schema.audit_log"
+ def hasQueueTimeMs = schemaResult.any { it[0] == "queue_time_ms" }
+ assertTrue(hasQueueTimeMs)
+
+ // Match only the queued query. Excluding audit_log statements prevents the lookup
+ // queries themselves from matching the marker during retries.
+ def query = """
+ select query_id, queue_time_ms, stmt
+ from __internal_schema.audit_log
+ where stmt like '%${queuedMarker}%'
+ and stmt not like '%__internal_schema.audit_log%'
+ order by time
+ """
+ def auditResult = []
+ for (int retry = 0; retry < 10 && auditResult.isEmpty(); retry++) {
+ sql "call flush_audit_log()"
+ sleep(1000)
+ auditResult = sql "${query}"
}
- sql "call flush_audit_log()"
- sleep(3000)
- auditResult = sql "${query}"
- }
- auditResult.each { row ->
- assertTrue(row[1] >= sqlSleepTime * 1000)
+ assertFalse(auditResult.isEmpty(), "queued query was not found in audit log")
+ logger.info("Queued query audit result: ${auditResult}")
+ auditResult.each { row ->
+ def queueTimeMs = row[1] as long
+ assertTrue(queueTimeMs >= guaranteedQueueTimeMs,
+ "queue_time_ms ${queueTimeMs} is less than guaranteed wait ${guaranteedQueueTimeMs}")
+ assertTrue(queueTimeMs < queueTimeoutMs,
+ "queue_time_ms ${queueTimeMs} reached queue timeout ${queueTimeoutMs}")
+ }
+ } finally {
+ threads.each { thread ->
+ if (thread.isAlive()) {
+ thread.join(blockerSleepTime * 1000 + 5000)
+ }
+ }
+ sql "drop table if exists ${tableName}"
+ sql "drop workload group if exists ${wgName}"
+ sql "set global enable_audit_plugin = ${originalAuditPlugin}"
}
-
- assertTrue(auditResult.size() >= queuedSqlCnt)
-
- // Cleanup
- sql "drop table if exists ${tableName}"
- sql "drop workload group if exists ${wgName}"
- sql "set global enable_audit_plugin = false"
}
diff --git a/regression-test/suites/auth_call/test_dml_select_udf_auth.groovy b/regression-test/suites/auth_call/test_dml_select_udf_auth.groovy
index f09d28b167c..1845304d3b3 100644
--- a/regression-test/suites/auth_call/test_dml_select_udf_auth.groovy
+++ b/regression-test/suites/auth_call/test_dml_select_udf_auth.groovy
@@ -44,6 +44,7 @@ suite("test_dml_select_udf_auth","p0,auth_call") {
def jarPath = """${context.file.parent}/../javaudf_p0/jars/java-udf-case-jar-with-dependencies.jar"""
scp_udf_file_to_all_be(jarPath)
+ scp_udf_file_to_all_fe(jarPath)
log.info("Jar path: ${jarPath}".toString())
sql """ CREATE FUNCTION ${dbName}.${udfName}(string) RETURNS int PROPERTIES (
diff --git a/regression-test/suites/auth_call/test_dml_stream_load_be_auth_docker.groovy b/regression-test/suites/auth_call/test_dml_stream_load_be_auth_docker.groovy
index 46d06e5df81..10d0388ea0c 100644
--- a/regression-test/suites/auth_call/test_dml_stream_load_be_auth_docker.groovy
+++ b/regression-test/suites/auth_call/test_dml_stream_load_be_auth_docker.groovy
@@ -16,6 +16,27 @@
// under the License.
import org.apache.doris.regression.suite.ClusterOptions
+import java.util.concurrent.TimeUnit
+
+// Drain both pipes while curl runs. Reading process.text after waitForOrKill can read an
+// already closed stream and also risks blocking when the child fills a pipe.
+def executeCurl = { List command ->
+ def process = command.execute()
+ def stdout = new ByteArrayOutputStream()
+ def stderr = new ByteArrayOutputStream()
+ def stdoutThread = process.consumeProcessOutputStream(stdout)
+ def stderrThread = process.consumeProcessErrorStream(stderr)
+ boolean completed = process.waitFor(7200, TimeUnit.SECONDS)
+ if (!completed) {
+ process.destroyForcibly()
+ process.waitFor()
+ }
+ stdoutThread.join()
+ stderrThread.join()
+ assert completed : "curl timed out: ${stderr.toString('UTF-8')}"
+ assert process.exitValue() == 0 : "curl failed: ${stderr.toString('UTF-8')}"
+ return [stdout.toString('UTF-8').trim(), stderr.toString('UTF-8').trim()]
+}
suite("test_dml_stream_load_be_auth_off_docker", "docker,auth_call") {
def options = new ClusterOptions()
@@ -40,7 +61,7 @@ suite("test_dml_stream_load_be_auth_off_docker", "docker,auth_call") {
def parseCurlResult = { String out, String err ->
def lines = out.readLines()
- assertTrue("curl output should include http code, output=${out}, err=${err}", !lines.isEmpty())
+ assertTrue(!lines.isEmpty(), "curl output should include http code, output=${out}, err=${err}")
def httpCode = lines.last() as int
def body = lines.size() == 1 ? "" : lines[0..-2].join("\n")
return [httpCode, body, err]
@@ -63,10 +84,7 @@ suite("test_dml_stream_load_be_auth_off_docker", "docker,auth_call") {
"http://${endpoint}/api/${dbName}/${table}/_stream_load"
])
logger.info("stream load target: ${endpoint}, table: ${table}, label: ${label}, user: ${authUser}")
- def process = command.execute()
- process.waitForOrKill(7200000)
- def out = process.text.trim()
- def err = process.errorStream.text.trim()
+ def (out, err) = executeCurl(command)
logger.info("stream load out: ${out}, err: ${err}")
return parseCurlResult(out, err)
}
@@ -82,10 +100,7 @@ suite("test_dml_stream_load_be_auth_off_docker", "docker,auth_call") {
]
logger.info("stream load 2pc target: ${beHttpAddress}, table: ${table}, "
+ "txn operation: ${txnOperation}, user: ${authUser}")
- def process = command.execute()
- process.waitForOrKill(7200000)
- def out = process.text.trim()
- def err = process.errorStream.text.trim()
+ def (out, err) = executeCurl(command)
logger.info("stream load 2pc out: ${out}, err: ${err}")
return parseCurlResult(out, err)
}
@@ -93,7 +108,7 @@ suite("test_dml_stream_load_be_auth_off_docker", "docker,auth_call") {
def assertLoadFailed = { def result ->
assertEquals(200, result[0])
def json = parseJson(result[1])
- assertTrue("stream load should fail, body=${result[1]}", json.Status != "Success")
+ assertTrue(json.Status != "Success", "stream load should fail, body=${result[1]}")
}
def assertLoadSuccess = { def result ->
@@ -192,7 +207,7 @@ suite("test_dml_stream_load_be_auth_on_docker", "docker,auth_call") {
def parseCurlResult = { String out, String err ->
def lines = out.readLines()
- assertTrue("curl output should include http code, output=${out}, err=${err}", !lines.isEmpty())
+ assertTrue(!lines.isEmpty(), "curl output should include http code, output=${out}, err=${err}")
def httpCode = lines.last() as int
def body = lines.size() == 1 ? "" : lines[0..-2].join("\n")
return [httpCode, body, err]
@@ -215,10 +230,7 @@ suite("test_dml_stream_load_be_auth_on_docker", "docker,auth_call") {
"http://${endpoint}/api/${dbName}/${table}/_stream_load"
])
logger.info("stream load target: ${endpoint}, table: ${table}, label: ${label}, user: ${authUser}")
- def process = command.execute()
- process.waitForOrKill(7200000)
- def out = process.text.trim()
- def err = process.errorStream.text.trim()
+ def (out, err) = executeCurl(command)
logger.info("stream load out: ${out}, err: ${err}")
return parseCurlResult(out, err)
}
@@ -234,10 +246,7 @@ suite("test_dml_stream_load_be_auth_on_docker", "docker,auth_call") {
]
logger.info("stream load 2pc target: ${beHttpAddress}, table: ${table}, "
+ "txn operation: ${txnOperation}, user: ${authUser}")
- def process = command.execute()
- process.waitForOrKill(7200000)
- def out = process.text.trim()
- def err = process.errorStream.text.trim()
+ def (out, err) = executeCurl(command)
logger.info("stream load 2pc out: ${out}, err: ${err}")
return parseCurlResult(out, err)
}
@@ -245,7 +254,7 @@ suite("test_dml_stream_load_be_auth_on_docker", "docker,auth_call") {
def assertLoadFailed = { def result ->
assertEquals(200, result[0])
def json = parseJson(result[1])
- assertTrue("stream load should fail, body=${result[1]}", json.Status != "Success")
+ assertTrue(json.Status != "Success", "stream load should fail, body=${result[1]}")
}
def assertLoadSuccess = { def result ->
diff --git a/regression-test/suites/auth_p0/test_role_mapping_system_table.groovy b/regression-test/suites/auth_p0/test_role_mapping_system_table.groovy
index 284b533a352..95680a8df28 100644
--- a/regression-test/suites/auth_p0/test_role_mapping_system_table.groovy
+++ b/regression-test/suites/auth_p0/test_role_mapping_system_table.groovy
@@ -101,7 +101,7 @@ suite("test_role_mapping_system_table", "p0,auth") {
assertTrue(result[0][6] != null && result[0][6].length() > 0)
assertEquals(result[0][5], result[0][7])
- connect(user, "${pwd}", jdbcUrlWithoutSchema) {
+ connectToDoris(user, "${pwd}", jdbcUrlWithoutSchema) {
test {
sql """
SELECT NAME
diff --git a/regression-test/suites/cloud_p0/auth/test_set_default_cluster.groovy b/regression-test/suites/cloud_p0/auth/test_set_default_cluster.groovy
index ddcf6cd28f4..5193c3e5af6 100644
--- a/regression-test/suites/cloud_p0/auth/test_set_default_cluster.groovy
+++ b/regression-test/suites/cloud_p0/auth/test_set_default_cluster.groovy
@@ -42,8 +42,10 @@ suite("test_default_cluster", "docker") {
def user1 = "default_user1"
// admin role
def user2 = "default_user2"
- // domain user
- def user3 = "default_user3@'175.%'"
+ // The Docker client address depends on the network allocated to this run.
+ def clientHost = (sql "SELECT USER()")[0][0].toString().split('@').last().replace("'", "")
+ assertTrue(clientHost ==~ /[0-9A-Za-z.:_-]+/, "Unexpected Docker client host: ${clientHost}")
+ def user3 = "default_user3@'${clientHost}'"
sql """CREATE USER $user1 IDENTIFIED BY 'Cloud123456' DEFAULT ROLE 'admin'"""
sql """CREATE USER $user2 IDENTIFIED BY 'Cloud123456'"""
diff --git a/regression-test/suites/cloud_p0/balance/test_cross_cg_bvar_metrics.groovy b/regression-test/suites/cloud_p0/balance/test_cross_cg_bvar_metrics.groovy
index b21b347e84e..fdc7a2b4202 100644
--- a/regression-test/suites/cloud_p0/balance/test_cross_cg_bvar_metrics.groovy
+++ b/regression-test/suites/cloud_p0/balance/test_cross_cg_bvar_metrics.groovy
@@ -232,22 +232,26 @@ suite('test_cross_cg_bvar_metrics', 'docker') {
assert beA2 != null : "Could not identify BE_A2"
logger.info("T_bvar2: BE_A2 host={} brpcPort={}", beA2.Host, beA2.BrpcPort)
- // Wait until BE_A2 has both accepted tablets and registered same-CG candidates.
+ // Wait until this table has at least one tablet routed to BE_A2. TabletNum and the
+ // process-wide mapping gauge may become positive before FE updates this table's route.
+ def routedTabletIds = []
awaitUntil(120) {
- def latest = sql_return_maparray('show backends').find {
+ def routedTablets = sql_return_maparray("SHOW TABLETS FROM ${tbl}").findAll {
it.BackendId.toLong() == beA2.BackendId.toLong()
}
- latest != null && latest.TabletNum.toInteger() > 0
+ def candidateReady =
+ getBrpcMetric(beA2.Host, beA2.BrpcPort, "balance_tablet_be_mapping_size") > 0
+ routedTabletIds = routedTablets.collect { it.TabletId.toLong() }
+ logger.info("T_bvar2 readiness: BE_A2 routed tablets={}, candidateReady={}",
+ routedTabletIds, candidateReady)
+ !routedTabletIds.isEmpty() && candidateReady
}
- awaitUntil(120) {
- getBrpcMetric(beA2.Host, beA2.BrpcPort, "balance_tablet_be_mapping_size") > 0
- }
-
- // Give the rebalance warmup workflow a short window to submit and finish the skipped
- // warmup tasks before query traffic starts.
- sleep(2000)
sql "use @compute_cluster"
+ // Retries must reach BE_A2 after its tablet route becomes active. Otherwise the first
+ // result can be served repeatedly from FE SQL cache without exercising same-CG reads.
+ sql "SET enable_sql_cache = false"
+ sql "SET enable_query_cache = false"
def t2_sameBefore_A1 = getBrpcMetric(beA.Host, beA.BrpcPort, "peer_same_compute_group_read")
def t2_sameBefore_A2 = getBrpcMetric(beA2.Host, beA2.BrpcPort, "peer_same_compute_group_read")
@@ -255,7 +259,8 @@ suite('test_cross_cg_bvar_metrics', 'docker') {
// Run query iteratively; full-table scans should cover tablets rebalanced to BE_A2.
// Those scans see: (a) same-CG candidate (BE_A1) registered, (b) empty local cache
// after warmup timeout -> same-CG winner race with peer win -> peer_same_compute_group_read++.
- sql "SELECT * FROM ${tbl} ORDER BY k1"
+ def result = sql "SELECT * FROM ${tbl} ORDER BY k1"
+ assertEquals(6, result.size(), "T_bvar2 query should return all rows")
def sameAfterA1 = getBrpcMetric(beA.Host, beA.BrpcPort, "peer_same_compute_group_read")
def sameAfterA2 = getBrpcMetric(beA2.Host, beA2.BrpcPort, "peer_same_compute_group_read")
((sameAfterA1 - t2_sameBefore_A1) + (sameAfterA2 - t2_sameBefore_A2)) > 0
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/sync_insert.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/sync_insert.groovy
index a80dab3105a..c653462c57d 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/sync_insert.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/sync_insert.groovy
@@ -15,148 +15,228 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.WarmupMetricsUtils
-suite("sync_insert") {
- List<String> ipList = new ArrayList<>();
- List<String> hbPortList = new ArrayList<>()
- List<String> httpPortList = new ArrayList<>()
- List<String> brpcPortList = new ArrayList<>()
- List<String> beUniqueIdList = new ArrayList<>()
-
- String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
- int num = 0
- for(String values : bes) {
- if (num++ == 2) break;
- println("the value is " + values);
- String[] beInfo = values.split(':');
- ipList.add(beInfo[0]);
- hbPortList.add(beInfo[1]);
- httpPortList.add(beInfo[2]);
- beUniqueIdList.add(beInfo[3]);
- brpcPortList.add(beInfo[4]);
+suite("sync_insert", "nonConcurrent") {
+ String sourceCluster = "regression_cluster_name0"
+ String targetCluster = "regression_cluster_name1"
+ String tableName = "test_dup_tab_basic_int_tab_nullable"
+ def values = [
+ [9,10,11,12], [9,10,11,12], [21,null,23,null], [1,2,3,4],
+ [1,2,3,4], [13,14,15,16], [13,21,22,16], [13,14,15,16],
+ [13,21,22,16], [17,18,19,20], [17,18,19,20], [null,21,null,23],
+ [22,null,24,25], [26,27,null,29], [5,6,7,8], [5,6,7,8]
+ ]
+ def backends = sql_return_maparray("SHOW BACKENDS")
+ def clusterBackends = [sourceCluster, targetCluster].collectEntries { clusterName ->
+ def members = backends.findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collect { be ->
+ [id: be.BackendId as Long, ip: be.Host.toString(),
+ httpPort: be.HttpPort.toString(), brpcPort: be.BrpcPort.toString()]
+ }
+ assertTrue(!members.isEmpty(), "No alive backends in ${clusterName}")
+ [(clusterName): members]
}
-
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
- println("the brpc port is " + brpcPortList);
-
- def clearFileCache = { ip, port ->
- httpTest {
- endpoint ""
- uri ip + ":" + port + """/api/file_cache?op=clear&sync=true"""
- op "get"
- body ""
+ logger.info("Sync insert backends: ${clusterBackends}")
+ def tabletIds = []
+ def primaryBackends = [:]
+ def waitUntil = { String description, Closure condition ->
+ long deadline = System.currentTimeMillis() + 120000L
+ while (System.currentTimeMillis() < deadline) {
+ if (condition()) {
+ return
+ }
+ sleep(1000)
}
+ assertTrue(false, "Timeout after 120000ms: ${description}; see last state in log")
}
-
- def getMetricsMethod = { ip, port, check_func ->
+ def getGlobalBytes = {
+ [source: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[sourceCluster], "file_cache_cache_size"),
+ target: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[targetCluster], "file_cache_cache_size")]
+ }
+ def getCache = { String clusterName ->
+ sql "use @${clusterName}"
+ def beIds = clusterBackends[clusterName].collect { it.id }
+ def rows = sql """select tablet_id, be_id, lower(type), sum(size), count(*)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${beIds.join(',')})
+ group by tablet_id, be_id, lower(type)"""
+ rows.groupBy { "${it[0]}:${it[1]}".toString() }.collectEntries { key, entries ->
+ [(key): [normalBytes: entries.findAll { it[2]?.toString() == "normal" }.sum(0L) { it[3] as Long },
+ otherBlocks: entries.findAll { it[2]?.toString() != "normal" }.sum(0L) { it[4] as Long },
+ blocks: entries.sum(0L) { it[4] as Long }]]
+ }
+ }
+ def getHeader = { be, long tabletId ->
+ def meta = null
httpTest {
- endpoint ip + ":" + port
- uri "/brpc_metrics"
+ endpoint "${be.ip}:${be.httpPort}"
+ uri "/api/meta/header/${tabletId}"
op "get"
- check check_func
+ check { code, body ->
+ if ((code as int) == 200) {
+ meta = parseJson(body.toString())
+ }
+ }
}
+ meta
}
- def table1 = "test_dup_tab_basic_int_tab_nullable"
- sql """ drop table if exists ${table1} """
+ try {
+ sql "use @${sourceCluster}"
+ String dbName = sql("select database()")[0][0].toString()
+ sql "DROP TABLE IF EXISTS ${tableName} FORCE"
+ sql """CREATE TABLE ${tableName} (
+ siteid INT NULL, citycode INT NULL, userid INT NULL, pv INT NULL
+ ) ENGINE=OLAP DUPLICATE KEY(siteid)
+ DISTRIBUTED BY HASH(siteid) BUCKETS 1
+ PROPERTIES("disable_auto_compaction"="true", "file_cache_ttl_seconds"="0")"""
+ tabletIds = sql("show tablets from ${tableName}").collect { it[0] as Long }.unique()
+ assertTrue(tabletIds.size() == 1, "Expected one tablet: ${tabletIds}")
+ def initialPartitions = sql_return_maparray("show partitions from ${tableName}")
+ assertTrue(initialPartitions.size() == 1, "Expected one partition: ${initialPartitions}")
+ long initialVersion = initialPartitions[0].VisibleVersion as Long
- // clear all file cache on all BEs of all clusters
- clearFileCache.call(ipList[0], httpPortList[0]);
- clearFileCache.call(ipList[1], httpPortList[1]);
-
- sql """
-CREATE TABLE IF NOT EXISTS `${table1}` (
- `siteid` int(11) NULL COMMENT "",
- `citycode` int(11) NULL COMMENT "",
- `userid` int(11) NULL COMMENT "",
- `pv` int(11) NULL COMMENT ""
-) ENGINE=OLAP
-DUPLICATE KEY(`siteid`)
-COMMENT "OLAP"
-DISTRIBUTED BY HASH(`siteid`) BUCKETS 1
-"""
-
- long insertedSizeBytes = 795
- connect('root') {
- sql "use @regression_cluster_name0"
- sql "use regression_test_cloud_p0_cache_multi_cluster_read_write"
- sql """ set enable_multi_cluster_sync_load = true """
- // all the inserted values will result in insertedSizeBytes bytes of size
- sql """insert into test_dup_tab_basic_int_tab_nullable values
- (9,10,11,12),
- (9,10,11,12),
- (21,null,23,null),
- (1,2,3,4),
- (1,2,3,4),
- (13,14,15,16),
- (13,21,22,16),
- (13,14,15,16),
- (13,21,22,16),
- (17,18,19,20),
- (17,18,19,20),
- (null,21,null,23),
- (22,null,24,25),
- (26,27,null,29),
- (5,6,7,8),
- (5,6,7,8)
- """
- sql """ set enable_multi_cluster_sync_load = false """
- }
+ waitUntil("source and target primary tablet routes") {
+ [sourceCluster, targetCluster].each { clusterName ->
+ sql "use @${clusterName}"
+ def members = clusterBackends[clusterName].collectEntries { [(it.id): it] }
+ def routes = [:]
+ sql_return_maparray("show tablets from ${tableName}").each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (id in tabletIds && members.containsKey(primary) && (tablet.BackendId as Long) == primary) {
+ routes[id] = members[primary]
+ }
+ }
+ primaryBackends[clusterName] = routes
+ }
+ logger.info("Sync insert primary routes: ${primaryBackends}")
+ [sourceCluster, targetCluster].every { primaryBackends[it].keySet() == tabletIds.toSet() }
+ }
+ def beforeClear = getGlobalBytes()
+ clusterBackends.values().flatten().each { be ->
+ WarmupMetricsUtils.clearFileCache(be.ip, be.httpPort)
+ }
+ def afterClear = getGlobalBytes()
+ logger.info("Sync insert clear: before=${beforeClear}, after=${afterClear}, " +
+ "source_minus_target_after_clear=${afterClear.source - afterClear.target}, " +
+ "source_change_after_clear=${afterClear.source - beforeClear.source}, " +
+ "target_change_after_clear=${afterClear.target - beforeClear.target}")
+ [sourceCluster, targetCluster].each { clusterName ->
+ def cache = getCache(clusterName)
+ assertTrue(cache.isEmpty(), "New test tablet must be cold in ${clusterName}: ${cache}")
+ }
- sleep(30000) // wait for download from remote
+ connect('root') {
+ sql "use @${sourceCluster}"
+ sql "use `${dbName.replace('`', '``')}`"
+ String previousSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(previousSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${previousSyncLoad}")
+ try {
+ sql "set enable_multi_cluster_sync_load = true"
+ def tuples = values.collect { row ->
+ "(" + row.collect { it == null ? "NULL" : it.toString() }.join(',') + ")"
+ }.join(',')
+ sql "INSERT INTO ${tableName} VALUES ${tuples}"
+ } finally {
+ sql "set enable_multi_cluster_sync_load = ${previousSyncLoad}"
+ }
+ }
- // get insert size of first cluster, may contain system tables
- long srcClusterSize = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("file_cache_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ sql "use @${sourceCluster}"
+ long loadedVersion = -1L
+ waitUntil("INSERT visible version") {
+ def partitions = sql_return_maparray("show partitions from ${tableName}")
+ logger.info("Sync insert visible partition state: ${partitions}")
+ if (partitions.size() == 1) {
+ loadedVersion = partitions[0].VisibleVersion as Long
+ }
+ loadedVersion > initialVersion
+ }
+ def expectedBytes = [:]
+ waitUntil("source rowset metadata for the inserted rows") {
+ def ready = [:]
+ tabletIds.each { id ->
+ try {
+ def meta = getHeader(primaryBackends[sourceCluster][id], id)
+ if (meta == null || (meta.tablet_id as Long) != id) {
+ return
}
- def i = line.indexOf(' ')
- srcClusterSize = line.substring(i).toLong()
- flag = true
- break
+ // Cloud /api/meta/header omits rs_metas. This metadata scanner reads the
+ // source BE's rowset map without scanning data or filling target cache.
+ def rowsets = sql("""select rowset_id, start_version, end_version, rowset_num_rows,
+ data_disk_size, index_disk_size, num_segments
+ from information_schema.rowsets
+ where tablet_id = ${id} and backend_id = ${primaryBackends[sourceCluster][id].id}
+ and start_version > ${initialVersion} and end_version <= ${loadedVersion}""")
+ .collect { row ->
+ [rowset_id_v2: row[0], start_version: row[1] as Long, end_version: row[2] as Long,
+ num_rows: row[3] as Long, data_disk_size: row[4] as Long,
+ index_disk_size: row[5] as Long, num_segments: row[6] as Long]
+ }
+ logger.info("Inserted rowsets for ${id}: " + rowsets.collect {
+ [rowset: it.rowset_id_v2, start: it.start_version, end: it.end_version,
+ rows: it.num_rows, dataBytes: it.data_disk_size, indexBytes: it.index_disk_size,
+ segments: it.num_segments]
+ })
+ if (!rowsets.isEmpty() && rowsets.max { it.end_version as Long }.end_version as Long == loadedVersion &&
+ rowsets.sum(0L) { (it.num_rows ?: 0L) as Long } == values.size()) {
+ // This table has no inverted indexes. data_disk_size sums actual segment file sizes,
+ // including packed slices, without depending on a particular encoding or file format.
+ long bytes = rowsets.sum(0L) { (it.data_disk_size ?: 0L) as Long }
+ assertTrue(rowsets.every { ((it.index_disk_size ?: 0L) as Long) == 0L },
+ "Unexpected external index files for ${id}: ${rowsets}")
+ if (bytes > 0L) {
+ ready[id] = bytes
+ }
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for inserted rowset metadata on ${id}: ${e.message}")
}
}
- assertTrue(flag)
- }
+ if (ready.keySet() == tabletIds.toSet()) {
+ expectedBytes = ready
+ return true
+ }
+ false
+ }
+ logger.info("Sync insert actual segment bytes: ${expectedBytes}")
- // get synced insert size of first cluster
- long dstClusterSize = 0
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("file_cache_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- dstClusterSize = line.substring(i).toLong()
- flag = true
- break
+ waitUntil("source and target NORMAL cache to match actual inserted segment bytes") {
+ def source = getCache(sourceCluster)
+ def target = getCache(targetCluster)
+ logger.info("Sync insert cache: expected=${expectedBytes}, source=${source}, target=${target}, " +
+ "primary=${primaryBackends}")
+ // Inspect one complete copy on each cluster's primary BE. Do not add replicated copies:
+ // syncLoadForTablets can send the same tablet to multiple BEs in a target cluster.
+ [sourceCluster, targetCluster].every { clusterName ->
+ def cache = clusterName == sourceCluster ? source : target
+ tabletIds.every { id ->
+ String key = "${id}:${primaryBackends[clusterName][id].id}".toString()
+ def copy = cache[key]
+ copy != null && copy.blocks > 0L && copy.otherBlocks == 0L &&
+ copy.normalBytes == expectedBytes[id]
}
}
- assertTrue(flag)
+ }
+
+ // Query only after proving automatic download, so reads cannot mask a failed sync load.
+ connect('root') {
+ sql "use @${targetCluster}"
+ sql "use `${dbName.replace('`', '``')}`"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ def actual = sql("select siteid, citycode, userid, pv from ${tableName}")
+ .collect { row -> row.collect { it == null ? null : it as Long } }
+ def expected = values.collect { row -> row.collect { it == null ? null : it as Long } }
+ assertTrue(actual.countBy { it } == expected.countBy { it },
+ "Target data differs, including duplicate/NULL rows: actual=${actual}, expected=${expected}")
+ }
+ } finally {
+ sql "DROP TABLE IF EXISTS ${tableName} FORCE"
}
- // FIXME(gavin): this is a strong assertion, make it weaker and robuster
- assertEquals(insertedSizeBytes, dstClusterSize)
- org.junit.Assert.assertTrue("insertedSizeBytes ${insertedSizeBytes} <= dstClusterSize ${dstClusterSize}", insertedSizeBytes <= dstClusterSize)
- org.junit.Assert.assertTrue("insertedSizeBytes ${insertedSizeBytes} <= srcClusterSize ${dstClusterSize}", insertedSizeBytes <= srcClusterSize)
- org.junit.Assert.assertTrue("dstClusterSize ${insertedSizeBytes} <= srcClusterSize ${dstClusterSize}", dstClusterSize <= srcClusterSize)
- sql "drop table if exists ${table1}"
}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/test_multi_stale_rowset.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/test_multi_stale_rowset.groovy
index b2bc06ffd81..0b7b2e24430 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/test_multi_stale_rowset.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/read_write/test_multi_stale_rowset.groovy
@@ -17,7 +17,16 @@
import org.codehaus.groovy.runtime.IOGroovyMethods
-suite("test_multi_stale_rowset") {
+suite("test_multi_stale_rowset", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
+ }
+
def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="1200", "disable_auto_compaction"="true") """
List<String> ipList = new ArrayList<>();
List<String> hbPortList = new ArrayList<>()
@@ -138,6 +147,10 @@ suite("test_multi_stale_rowset") {
load_customer_once()
sleep(30000);
def backendIdToAfterLoadCacheSize = getCurCacheSize()
+ assertTrue(backendIdToAfterLoadCacheSize.get(beUniqueIdList[0]) > 0,
+ "Source backend should cache rowsets generated by load")
+ assertTrue(backendIdToAfterLoadCacheSize.get(beUniqueIdList[1]) > 0,
+ "Target backend should cache rowsets synchronized by multi-cluster sync load")
sql "use @regression_cluster_name0"
String[][] tablets = sql """ show tablets from ${tableName}; """
@@ -147,50 +160,17 @@ suite("test_multi_stale_rowset") {
sleep(90000);
def backendIdToAfterCompactionCacheSize = getCurCacheSize()
- assertTrue(backendIdToAfterCompactionCacheSize.get(beUniqueIdList[1]) < backendIdToAfterLoadCacheSize.get(beUniqueIdList[1]));
-
+ logger.info("File cache size after load: ${backendIdToAfterLoadCacheSize}, " +
+ "after compaction: ${backendIdToAfterCompactionCacheSize}")
+ assertTrue(backendIdToAfterCompactionCacheSize.get(beUniqueIdList[0]) > 0,
+ "Source backend should cache the compaction output rowsets")
+ assertTrue(backendIdToAfterCompactionCacheSize.get(beUniqueIdList[1]) <
+ backendIdToAfterLoadCacheSize.get(beUniqueIdList[1]),
+ "Target backend should recycle the stale rowset cache")
+
+ // Multi-cluster sync load only synchronizes rowsets generated by load. It does not
+ // synchronize subsequent compaction output rowsets, so the source and target cache
+ // sizes are not expected to remain equal after stale rowset compaction.
sql "use @regression_cluster_name1"
-
- long ttl_cache_size = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- flag = true
- break
- }
- }
- assertTrue(flag)
- }
-
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
- }
- }
- assertTrue(flag)
- }
sql new File("""${context.file.parent}/ddl/${table}_delete.sql""").text
- sql """ set enable_multi_cluster_sync_load=false """
}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_batch.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_batch.groovy
index 21a3773b262..9244e7f9e24 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_batch.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_batch.groovy
@@ -15,177 +15,280 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.WarmupMetricsUtils
-suite("test_warm_up_cluster_batch") {
- def getJobState = { jobId ->
- def jobStateResult = sql """ SHOW WARM UP JOB WHERE ID = ${jobId} """
- return jobStateResult[0]
+suite("test_warm_up_cluster_batch", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
}
- def table = "customer"
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
- List<String> ipList = new ArrayList<>();
- List<String> hbPortList = new ArrayList<>()
- List<String> httpPortList = new ArrayList<>()
- List<String> brpcPortList = new ArrayList<>()
- List<String> beUniqueIdList = new ArrayList<>()
-
- String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
- int num = 0
- for(String values : bes) {
- if (num++ == 2) break;
- println("the value is " + values);
- String[] beInfo = values.split(':');
- ipList.add(beInfo[0]);
- hbPortList.add(beInfo[1]);
- httpPortList.add(beInfo[2]);
- beUniqueIdList.add(beInfo[3]);
- brpcPortList.add(beInfo[4]);
- }
-
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
- println("the brpc port is " + brpcPortList);
-
- sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
- sql new File("""${context.file.parent}/../ddl/supplier_delete.sql""").text
- // create table if not exists
- sql new File("""${context.file.parent}/../ddl/${table}.sql""").text
- sql new File("""${context.file.parent}/../ddl/supplier.sql""").text
-
- sql """ TRUNCATE TABLE __internal_schema.cloud_cache_hotspot; """
- sleep(30000)
-
- def s3BucketName = getS3BucketName()
- def s3WithProperties = """WITH S3 (
- |"AWS_ACCESS_KEY" = "${getS3AK()}",
- |"AWS_SECRET_KEY" = "${getS3SK()}",
- |"AWS_ENDPOINT" = "${getS3Endpoint()}",
- |"AWS_REGION" = "${getS3Region()}",
- |"provider" = "${getS3Provider()}")
- |PROPERTIES(
- |"exec_mem_limit" = "8589934592",
- |"load_parallelism" = "3")""".stripMargin()
-
-
-
- def clearFileCache = { ip, port ->
- httpTest {
- endpoint ""
- uri ip + ":" + port + """/api/file_cache?op=clear&sync=true"""
- op "get"
- body ""
- }
- }
-
- def getMetricsMethod = { ip, port, check_func ->
- httpTest {
- endpoint ip + ":" + port
- uri "/brpc_metrics"
- op "get"
- check check_func
+ String sourceCluster = "regression_cluster_name0"
+ String targetCluster = "regression_cluster_name1"
+ def tableNames = ["customer", "supplier"]
+ def dataQueries = [
+ customer: "select sum(C_CUSTKEY), sum(length(C_COMMENT)) from customer",
+ supplier: "select sum(s_suppkey), sum(length(s_address)) from supplier"
+ ]
+ def backends = sql_return_maparray("SHOW BACKENDS")
+ def clusterBackends = [sourceCluster, targetCluster].collectEntries { clusterName ->
+ def members = backends.findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collect { be ->
+ [id: be.BackendId as Long, ip: be.Host.toString(),
+ httpPort: be.HttpPort.toString(), brpcPort: be.BrpcPort.toString()]
}
+ assertTrue(!members.isEmpty(), "No alive backends in ${clusterName}")
+ [(clusterName): members]
}
+ logger.info("Batch warmup backends: ${clusterBackends}")
+ assertTrue(getFeConfig("enable_fetch_cluster_cache_hotspot").equalsIgnoreCase("true"),
+ "Cluster warmup requires the FE cache hotspot collector to be enabled at startup")
+ def hotspotInterval = getFeConfig("fetch_cluster_cache_hotspot_interval_ms")
- clearFileCache.call(ipList[0], httpPortList[0]);
- clearFileCache.call(ipList[1], httpPortList[1]);
-
- def load_customer_once = {
- def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
- def loadLabel = table + "_" + uniqueID
- // load data from cos
- def loadSql = new File("""${context.file.parent}/../ddl/${table}_load.sql""").text.replaceAll("\\\$\\{s3BucketName\\}", s3BucketName)
- loadSql = loadSql.replaceAll("\\\$\\{loadLabel\\}", loadLabel) + s3WithProperties
- sql loadSql
-
- // check load state
- while (true) {
- def stateResult = sql "show load where Label = '${loadLabel}'"
- def loadState = stateResult[stateResult.size() - 1][2].toString()
- if ("CANCELLED".equalsIgnoreCase(loadState)) {
- throw new IllegalStateException("load ${loadLabel} failed.")
- } else if ("FINISHED".equalsIgnoreCase(loadState)) {
- break
+ def waitUntil = { String description, long timeoutMs, Closure condition ->
+ long deadline = System.currentTimeMillis() + timeoutMs
+ while (System.currentTimeMillis() < deadline) {
+ if (condition()) {
+ return
}
- sleep(5000)
- }
- }
-
- sql "use @regression_cluster_name0"
- def waitJobDone = { jobId ->
- int retryTime = 120
- int i = 0
- for (; i < retryTime; i++) {
sleep(1000)
- def statuses = getJobState(jobId[0][0])
- if (statuses.any { it != null && it.equals("CANCELLED") }) {
- assertTrue(false);
- }
- if (statuses.any { it != null && it.equals("FINISHED") }) {
- break;
- }
- }
- if (i == retryTime) {
- sql "cancel warm up job where id = ${jobId[0][0]}"
- assertTrue(false);
}
+ assertTrue(false, "Timeout after ${timeoutMs}ms: ${description}")
+ }
+ def pendingLoads = [] as Set
+ def warmupJobs = [] as Set
+ def tableInfo = [:]
+ def getGlobalTtlBytes = {
+ [source: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[sourceCluster], "ttl_cache_size"),
+ target: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[targetCluster], "ttl_cache_size")]
}
- for (int k = 0; k < 10; k++) {
- load_customer_once()
- for (int i = 0; i < 10; i++) {
- sql "select count(*) from customer"
+ def getNormalCache = { String clusterName ->
+ sql "use @${clusterName}"
+ def beIds = clusterBackends[clusterName].collect { it.id }.join(',')
+ tableInfo.collectEntries { tableName, info ->
+ def rows = sql """select tablet_id, sum(size)
+ from information_schema.file_cache_info
+ where tablet_id in (${info.tabletIds.join(',')})
+ and be_id in (${beIds}) and lower(type) = 'normal'
+ group by tablet_id"""
+ def bytesByTablet = rows.collectEntries { [(it[0] as Long): it[1] as Long] }
+ [(tableName): [bytes: bytesByTablet.values().sum(0L) as Long,
+ complete: info.tabletIds.every { (bytesByTablet[it] ?: 0L) > 0L }]]
}
- if (k % 2 == 1) {
- sleep(40000)
- def jobId_ = sql "WARM UP CLUSTER regression_cluster_name1 WITH CLUSTER regression_cluster_name0"
- waitJobDone(jobId_)
- sleep(30000)
- long ttl_cache_size = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ }
+ def getHotspotCounts = {
+ sql "use @${sourceCluster}"
+ def rows = sql """select table_id, partition_id, sum(query_per_day)
+ from __internal_schema.cloud_cache_hotspot
+ where cluster_name = '${sourceCluster}'
+ and table_id in (${tableInfo.values().collect { it.id }.join(',')})
+ and insert_day = current_date()
+ group by table_id, partition_id"""
+ rows.collectEntries { [("${it[0]}:${it[1]}".toString()): it[2] as Long] }
+ }
+ def waitForTargetTablets = {
+ sql "use @${targetCluster}"
+ def targetBes = clusterBackends[targetCluster].collectEntries { [(it.id): it] }
+ def missing = []
+ waitUntil("target primary routing and tablet metadata; see missing tablet log", 120000L) {
+ missing = []
+ tableInfo.each { tableName, info ->
+ def ready = [] as Set
+ def tablets = sql_return_maparray("show tablets from ${tableName}")
+ tablets.each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (info.tabletIds.contains(id) && targetBes.containsKey(primary) &&
+ (tablet.BackendId as Long) == primary) {
+ def be = targetBes[primary]
+ try {
+ // Load metadata only; querying the target table could mask a failed warmup.
+ httpTest {
+ endpoint "${be.ip}:${be.httpPort}"
+ uri "/api/meta/header/${id}"
+ op "get"
+ check { code, body ->
+ if ((code as int) == 200) {
+ def meta = parseJson(body.toString())
+ // Cloud header JSON omits rowsets; data readiness is
+ // established by the source scan and cache baseline.
+ if ((meta.tablet_id as Long) == id &&
+ (meta.ttl_seconds as Long) == 0L &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null) {
+ ready.add(id)
+ }
+ }
+ }
}
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- flag = true
- break
+ } catch (Exception e) {
+ logger.info("Waiting for target tablet ${id} on BE ${primary}: ${e.message}")
}
}
- assertTrue(flag)
+ }
+ missing.addAll(info.tabletIds - ready)
}
+ logger.info("Target tablet readiness: missing=${missing}")
+ missing.isEmpty()
+ }
+ }
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
+ // This case must exercise hotspot-based selection, not the all-partitions bypass.
+ setFeConfigTemporary([cloud_warm_up_force_all_partitions: false]) {
+ try {
+ sql "use @${sourceCluster}"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ // Only disabling SQL/query cache does not disable COUNT(*) constant rewriting.
+ // The SUM expressions above force scans of actual data columns on the source BE.
+ tableNames.each { tableName ->
+ sql "DROP TABLE IF EXISTS ${tableName} FORCE"
+ sql (new File("${context.file.parent}/../ddl/${tableName}.sql").text + """
+ PROPERTIES("disable_auto_compaction"="true", "file_cache_ttl_seconds"="0")""")
+ def tabletIds = sql("show tablets from ${tableName}").collect { it[0] as Long }.unique()
+ def partitionIds = sql("show partitions from ${tableName}").collect { it[0] as Long }.unique()
+ long tableId = getTableId(tableName)
+ assertTrue(tableId > 0L && !tabletIds.isEmpty() && !partitionIds.isEmpty(),
+ "Missing table/partition/tablet IDs for ${tableName}")
+ tableInfo[tableName] = [id: tableId, tabletIds: tabletIds, partitionIds: partitionIds]
+ }
+ logger.info("Batch warmup tables: ${tableInfo}")
+ sql "TRUNCATE TABLE __internal_schema.cloud_cache_hotspot"
+
+ def ttlBeforeClear = getGlobalTtlBytes()
+ clusterBackends.values().flatten().each { be ->
+ WarmupMetricsUtils.clearFileCache(be.ip, be.httpPort)
+ }
+ // Held blocks can survive sync=true. Keep their baseline for diagnosis only.
+ def ttlAfterClear = getGlobalTtlBytes()
+ logger.info("TTL cache clear: before=${ttlBeforeClear}, after=${ttlAfterClear}, " +
+ "source_minus_target_after_clear=${ttlAfterClear.source - ttlAfterClear.target}, " +
+ "source_change_after_clear=${ttlAfterClear.source - ttlBeforeClear.source}, " +
+ "target_change_after_clear=${ttlAfterClear.target - ttlBeforeClear.target}")
+
+ def s3BucketName = getS3BucketName()
+ def s3Properties = """WITH S3 (
+ "AWS_ACCESS_KEY" = "${getS3AK()}",
+ "AWS_SECRET_KEY" = "${getS3SK()}",
+ "AWS_ENDPOINT" = "${getS3Endpoint()}",
+ "AWS_REGION" = "${getS3Region()}",
+ "provider" = "${getS3Provider()}")
+ PROPERTIES("exec_mem_limit"="8589934592", "load_parallelism"="3")"""
+ def loadOnce = { String tableName ->
+ String label = "${tableName}_${UUID.randomUUID().toString().replace('-', '')}"
+ String loadSql = new File("${context.file.parent}/../ddl/${tableName}_load.sql").text
+ .replace('${s3BucketName}', s3BucketName).replace('${loadLabel}', label)
+ sql (loadSql + s3Properties)
+ pendingLoads.add(label)
+ waitUntil("load ${label} to finish", 300000L) {
+ def rows = sql "show load where Label = '${label}'"
+ if (rows.isEmpty()) {
+ return false
+ }
+ String state = rows.last()[2].toString()
+ assertTrue(!state.equalsIgnoreCase("CANCELLED"), "Load ${label} cancelled: ${rows.last()}")
+ if (state.equalsIgnoreCase("FINISHED")) {
+ pendingLoads.remove(label)
+ return true
+ }
+ false
+ }
+ }
+ def previousBytes = tableNames.collectEntries { [(it): 0L] }
+ for (int round = 0; round < 10; round++) {
+ sql "use @${sourceCluster}"
+ tableNames.each { loadOnce(it) }
+ if (round % 2 == 0) {
+ continue
+ }
+ int batch = (round + 1) / 2
+ def hotspotBeforeQueries = getHotspotCounts()
+ // One real scan per table is enough to advance the hotspot counters for this batch.
+ tableNames.each { tableName ->
+ def result = sql(dataQueries[tableName])
+ assertTrue(!result.isEmpty() && result[0].every { it != null },
+ "Data query should read nonempty ${tableName}")
+ }
+ // Require hotspot counters for these new table/partition IDs to advance beyond
+ // the pre-query snapshot, without assuming identical FE and BE timestamps.
+ waitUntil("batch ${batch} hotspot refresh (configured interval=${hotspotInterval}ms); " +
+ "run with a short FE startup hotspot interval", 180000L) {
+ def counts = getHotspotCounts()
+ logger.info("Batch ${batch} hotspot counts: before=${hotspotBeforeQueries}, current=${counts}")
+ tableInfo.values().every { info ->
+ info.partitionIds.every { partitionId ->
+ String key = "${info.id}:${partitionId}"
+ (counts[key] ?: 0L) > (hotspotBeforeQueries[key] ?: 0L)
}
}
- assertTrue(flag)
+ }
+ waitForTargetTablets()
+ def sourceCache = [:]
+ waitUntil("batch ${batch} source NORMAL cache to include new data", 120000L) {
+ sourceCache = getNormalCache(sourceCluster)
+ logger.info("Batch ${batch} source NORMAL cache=${sourceCache}, previous=${previousBytes}")
+ tableNames.every { sourceCache[it].complete && sourceCache[it].bytes > previousBytes[it] }
+ }
+ def job = sql "WARM UP CLUSTER ${targetCluster} WITH CLUSTER ${sourceCluster}"
+ def jobId = job[0][0]
+ warmupJobs.add(jobId)
+ waitUntil("warmup job ${jobId} to finish", 300000L) {
+ def rows = sql "SHOW WARM UP JOB WHERE ID = ${jobId}"
+ if (rows.isEmpty()) {
+ return false
+ }
+ def states = rows[0].collect { it?.toString() }
+ assertTrue(!states.contains("CANCELLED"), "Warmup ${jobId} cancelled: ${rows[0]}")
+ states.contains("FINISHED")
+ }
+ def targetCache = [:]
+ waitUntil("batch ${batch}: per-table NORMAL cache bytes must be positive and equal", 180000L) {
+ sourceCache = getNormalCache(sourceCluster)
+ targetCache = getNormalCache(targetCluster)
+ logger.info("Batch ${batch} NORMAL cache: source=${sourceCache}, target=${targetCache}")
+ tableNames.every { tableName ->
+ sourceCache[tableName].complete && targetCache[tableName].complete &&
+ sourceCache[tableName].bytes > previousBytes[tableName] &&
+ sourceCache[tableName].bytes == targetCache[tableName].bytes
+ }
+ }
+ previousBytes = sourceCache.collectEntries { name, cache -> [(name): cache.bytes] }
+ def currentTtl = getGlobalTtlBytes()
+ logger.info("Batch ${batch} global TTL=${currentTtl}, baseline_after_clear=${ttlAfterClear}, " +
+ "source_minus_target=${currentTtl.source - currentTtl.target}")
+ }
+ } finally {
+ warmupJobs.each { jobId ->
+ try {
+ def rows = sql "SHOW WARM UP JOB WHERE ID = ${jobId}"
+ if (!rows.isEmpty() && rows[0].any { it?.toString() in ["PENDING", "RUNNING"] }) {
+ sql "CANCEL WARM UP JOB WHERE ID = ${jobId}"
+ }
+ } catch (Exception e) {
+ logger.warn("Failed to clean up warmup ${jobId}: ${e.message}")
+ }
+ }
+ sql "use @${sourceCluster}"
+ pendingLoads.each { label ->
+ try {
+ sql "CANCEL LOAD WHERE LABEL = '${label}'"
+ } catch (Exception e) {
+ logger.warn("Failed to clean up load ${label}: ${e.message}")
+ }
+ }
+ try {
+ sql "DROP TABLE IF EXISTS customer FORCE"
+ } finally {
+ sql "DROP TABLE IF EXISTS supplier FORCE"
}
- sql "use @regression_cluster_name0"
}
}
- sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
- sql new File("""${context.file.parent}/../ddl/supplier_delete.sql""").text
-}
+}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_event_cancel_expired.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_event_cancel_expired.groovy
index 6114892039a..5cfef65570a 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_event_cancel_expired.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_cluster_event_cancel_expired.groovy
@@ -16,6 +16,7 @@
// under the License.
import org.apache.doris.regression.suite.ClusterOptions
+import org.apache.doris.regression.util.NodeType
import groovy.json.JsonSlurper
// Covers a two-part bug on the event-driven warm-up path:
@@ -43,7 +44,11 @@ suite('test_warm_up_cluster_event_cancel_expired', 'docker') {
'file_cache_enter_disk_resource_limit_mode_percent=99',
'enable_evict_file_cache_in_advance=false',
'file_cache_background_monitor_interval_ms=1000',
+ // Expire cached replica locations before the post-cancel INSERTs so
+ // source BE must ask FE again and consume the CANCELLED status.
+ 'warmup_tablet_replica_info_cache_ttl_sec=5',
]
+ options.enableDebugPoints()
options.cloudMode = true
def clearFileCache = { ip, port ->
@@ -77,18 +82,22 @@ suite('test_warm_up_cluster_event_cancel_expired', 'docker') {
}
}
- def getSkippedRowsetSum = { cluster ->
+ def getMetricSum = { cluster, metric ->
def backends = sql """SHOW BACKENDS"""
def cluster_bes = backends.findAll {
it[19].contains("""\"compute_group_name\" : \"${cluster}\"""")
}
long sum = 0
for (be in cluster_bes) {
- sum += getBrpcMetrics(be[1], be[5], "file_cache_event_driven_warm_up_skipped_rowset_num")
+ sum += getBrpcMetrics(be[1], be[5], metric)
}
return sum
}
+ def getSkippedRowsetSum = { cluster ->
+ return getMetricSum(cluster, "file_cache_event_driven_warm_up_skipped_rowset_num")
+ }
+
docker(options) {
def clusterSrc = "warmup_source"
def clusterDst = "warmup_target"
@@ -124,27 +133,55 @@ suite('test_warm_up_cluster_event_cancel_expired', 'docker') {
}
sleep(5000)
- // 3. Cancel the job. After the fix, BE sees TStatus.CANCELLED on
- // the next get_replica_info and drops the job from
- // _tablet_replica_cache; before the fix (CANCELED typo) it
- // would keep the entry.
- sql """CANCEL WARM UP JOB WHERE ID = ${jobId}"""
+ // 3. Ignore the proactive CLEAR_JOB on source BE so this case
+ // exercises the passive cleanup path: after replica-info cache
+ // expiry, get_replica_info asks FE and receives TStatus.CANCELLED.
+ def sourceBes = sql("""SHOW BACKENDS""").findAll {
+ it[19].contains("""\"compute_group_name\" : \"${clusterSrc}\"""")
+ }
+ assertFalse(sourceBes.isEmpty(), "source cluster should have at least one BE")
+ def activeJobsBeforeCancel = getMetricSum(clusterSrc, "file_cache_warm_up_job_num")
+ assertTrue(activeJobsBeforeCancel > 0, "source BE should hold the event-driven job")
+ try {
+ for (be in sourceBes) {
+ GetDebugPoint().enableDebugPoint(be[1].toString(), be[4] as int, NodeType.BE,
+ "CloudWarmUpManager.set_event.ignore_all")
+ }
+ sql """CANCEL WARM UP JOB WHERE ID = ${jobId}"""
+ } finally {
+ for (be in sourceBes) {
+ GetDebugPoint().disableDebugPoint(be[1].toString(), be[4] as int, NodeType.BE,
+ "CloudWarmUpManager.set_event.ignore_all")
+ }
+ }
def st = sql """SHOW WARM UP JOB WHERE ID = ${jobId}"""
assertEquals("CANCELLED", st[0][3])
+ def activeJobsAfterCancel = getMetricSum(clusterSrc, "file_cache_warm_up_job_num")
+ assertEquals(activeJobsBeforeCancel, activeJobsAfterCancel,
+ "ignored CLEAR_JOB should leave the source BE cache intact")
+ sleep(6000)
- // 4. One more batch so BE actually sees the CANCELLED status
- // and (with the fix) purges its cache entry.
+ // 4. Waited past the 5s replica-info TTL after cancellation.
+ // The next batch makes source BE observe CANCELLED and purge the job.
for (int i = 0; i < 20; i++) {
sql """INSERT INTO t_exp VALUES (${100 + i}, 'y')"""
}
- sleep(5000)
+ def activeJobsAfterPassiveCleanup = getMetricSum(clusterSrc, "file_cache_warm_up_job_num")
+ for (int i = 0; i < 20 && activeJobsAfterPassiveCleanup >= activeJobsAfterCancel; i++) {
+ sleep(500)
+ activeJobsAfterPassiveCleanup = getMetricSum(clusterSrc, "file_cache_warm_up_job_num")
+ }
+ assertTrue(activeJobsAfterPassiveCleanup < activeJobsAfterCancel,
+ "source BE should remove the job after receiving TStatus.CANCELLED, " +
+ "before=${activeJobsAfterCancel} after=${activeJobsAfterPassiveCleanup}")
// 5. Baseline for the skipped-rowset counter. After BE has
// cleaned its cache, subsequent warm_up_rowset calls return
// early (empty replicas -> "skipping rowset") and bump this
- // counter on every commit. If the typo is unfixed the counter
- // stays flat because BE keeps calling FE.
- def skippedBaseline = getSkippedRowsetSum(clusterDst)
+ // counter on every commit.
+ // warm_up_rowset is invoked by the BE committing source rowsets, so
+ // this metric belongs to the source cluster rather than the target.
+ def skippedBaseline = getSkippedRowsetSum(clusterSrc)
logger.info("skipped_rowset baseline=${skippedBaseline}")
// 6. Wait past history_cloud_warm_up_job_keep_max_second plus one
@@ -160,13 +197,16 @@ suite('test_warm_up_cluster_event_cancel_expired', 'docker') {
logger.info("job ${jobId} removed from FE after ${i}s")
break
}
- } catch (Exception e) {
- if (!e.getMessage().contains("cloud warm up with job ${jobId} does not exist")) {
- throw e
+ } catch (Throwable t) {
+ def message = (t.getMessage() ?: t.toString()).toLowerCase()
+ if (message.contains("warm up") && message.contains("does not exist")) {
+ // FE may report an expired job as "does not exist" instead of an empty result.
+ // Both states mean the cancelled job has been removed from FE.
+ logger.info("job ${jobId} removed from FE after ${i}s: ${t.getMessage()}")
+ removed = true
+ break
}
- removed = true
- logger.info("job ${jobId} removed from FE after ${i}s: ${e.getMessage()}")
- break
+ throw t
}
sleep(1000)
}
@@ -197,11 +237,15 @@ suite('test_warm_up_cluster_event_cancel_expired', 'docker') {
"post-removal inserts should not be blocked by NPE sleeps, " +
"took ${elapsedMs}ms")
- // 8. On the fixed path every commit short-circuits through
- // g_file_cache_event_driven_warm_up_skipped_rowset_num.
- // We expect it to grow; on the buggy path it would be flat
- // since BE never stopped pursuing FE replicas.
- def skippedAfter = getSkippedRowsetSum(clusterDst)
+ // 8. After passive cleanup, source commits short-circuit through
+ // g_file_cache_event_driven_warm_up_skipped_rowset_num. The
+ // warm-up tasks are asynchronous, so allow a short observation
+ // window before checking that source-side traffic was counted.
+ def skippedAfter = getSkippedRowsetSum(clusterSrc)
+ for (int i = 0; i < 20 && skippedAfter <= skippedBaseline; i++) {
+ sleep(500)
+ skippedAfter = getSkippedRowsetSum(clusterSrc)
+ }
logger.info("skipped_rowset after=${skippedAfter}")
assertTrue(skippedAfter > skippedBaseline,
"BE should skip warm_up_rowset for tablets after cancel+expire, " +
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_compute_group.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_compute_group.groovy
index 19ca1958e2e..66e5b8c5019 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_compute_group.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/cluster/test_warm_up_compute_group.groovy
@@ -15,9 +15,21 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.WarmupMetricsUtils
+
+suite("test_warm_up_compute_group", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
+ }
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
-suite("test_warm_up_compute_group") {
def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="12000") """
def getJobState = { jobId ->
def jobStateResult = sql """ SHOW WARM UP JOB WHERE ID = ${jobId} """
@@ -25,31 +37,24 @@ suite("test_warm_up_compute_group") {
}
def table = "customer"
- List<String> ipList = new ArrayList<>();
- List<String> hbPortList = new ArrayList<>()
- List<String> httpPortList = new ArrayList<>()
- List<String> brpcPortList = new ArrayList<>()
- List<String> beUniqueIdList = new ArrayList<>()
-
- String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
- int num = 0
- for(String values : bes) {
- if (num++ == 2) break;
- println("the value is " + values);
- String[] beInfo = values.split(':');
- ipList.add(beInfo[0]);
- hbPortList.add(beInfo[1]);
- httpPortList.add(beInfo[2]);
- beUniqueIdList.add(beInfo[3]);
- brpcPortList.add(beInfo[4]);
+ String sourceCluster = "regression_cluster_name0"
+ String targetCluster = "regression_cluster_name1"
+ def backends = sql_return_maparray("SHOW BACKENDS")
+ def clusterBackends = [sourceCluster, targetCluster].collectEntries { clusterName ->
+ def members = backends.findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collect { be ->
+ [id: be.BackendId as Long, ip: be.Host.toString(),
+ httpPort: be.HttpPort.toString(), brpcPort: be.BrpcPort.toString()]
+ }
+ assertTrue(!members.isEmpty(), "No alive backends in ${clusterName}")
+ [(clusterName): members]
}
-
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
- println("the brpc port is " + brpcPortList);
+ logger.info("Warmup source=${sourceCluster}, target=${targetCluster}, backends=${clusterBackends}")
+ sql "use @${sourceCluster}"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
sql new File("""${context.file.parent}/../ddl/supplier_delete.sql""").text
@@ -73,26 +78,46 @@ suite("test_warm_up_compute_group") {
- def clearFileCache = { ip, port ->
- httpTest {
- endpoint ""
- uri ip + ":" + port + """/api/file_cache?op=clear&sync=true"""
- op "get"
- body ""
- }
+ // Capture the IDs before DROP so the final check can still address this case's tablets.
+ def tableTabletIds = [table, "supplier"].collectEntries { tableName ->
+ def ids = sql("show tablets from ${tableName}").collect { it[0] as Long }.unique()
+ assertTrue(!ids.isEmpty(), "No tablets found for ${tableName}")
+ [(tableName): ids]
}
+ logger.info("Case table tablets: ${tableTabletIds}")
- def getMetricsMethod = { ip, port, check_func ->
- httpTest {
- endpoint ip + ":" + port
- uri "/brpc_metrics"
- op "get"
- check check_func
+ def getGlobalTtlBytes = {
+ [source: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[sourceCluster], "ttl_cache_size"),
+ target: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[targetCluster], "ttl_cache_size")]
+ }
+ def clearCacheAndLogTtl = { List<String> clustersToClear, String phase ->
+ def before = getGlobalTtlBytes()
+ clustersToClear.each { clusterName ->
+ clusterBackends[clusterName].each { be ->
+ WarmupMetricsUtils.clearFileCache(be.ip, be.httpPort)
+ }
}
+ // sync=true can leave referenced blocks pending recycle. Record the residual baseline;
+ // it is diagnostic data, not an assertion that either BE's entire cache must be empty.
+ def after = getGlobalTtlBytes()
+ logger.info("TTL cache clear phase=${phase}, bytes_before=${before}, bytes_after=${after}, " +
+ "source_minus_target_after_clear=${after.source - after.target}, " +
+ "source_change_after_clear=${after.source - before.source}, " +
+ "target_change_after_clear=${after.target - before.target}")
+ return after
}
-
- clearFileCache.call(ipList[0], httpPortList[0]);
- clearFileCache.call(ipList[1], httpPortList[1]);
+ def getTableTtlBytes = { String clusterName ->
+ sql "use @${clusterName}"
+ def backendIds = clusterBackends[clusterName].collect { it.id }.join(',')
+ tableTabletIds.collectEntries { tableName, ids ->
+ def rows = sql """select coalesce(sum(size), 0)
+ from information_schema.file_cache_info
+ where tablet_id in (${ids.join(',')})
+ and be_id in (${backendIds}) and lower(type) = 'ttl'"""
+ [(tableName): rows[0][0] as Long]
+ }
+ }
+ def initialTtlAfterClear = clearCacheAndLogTtl([sourceCluster, targetCluster], "before_load")
def load_customer_once = {
def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
@@ -172,47 +197,33 @@ suite("test_warm_up_compute_group") {
}
waitJobDone(jobId_)
- sleep(30000)
- long ttl_cache_size = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- flag = true
- break
- }
- }
- assertTrue(flag)
- }
-
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
- }
- }
- assertTrue(flag)
+ // Cache metadata is published asynchronously. Compare each table independently so that
+ // missing bytes in one table cannot be hidden by extra bytes in the other table.
+ long cacheDeadline = System.currentTimeMillis() + 120000L
+ def sourceTableBytes = [:]
+ def targetTableBytes = [:]
+ boolean cacheMatches = false
+ while (System.currentTimeMillis() < cacheDeadline) {
+ sourceTableBytes = getTableTtlBytes(sourceCluster)
+ targetTableBytes = getTableTtlBytes(targetCluster)
+ logger.info("Case table TTL cache bytes: source=${sourceTableBytes}, target=${targetTableBytes}")
+ cacheMatches = tableTabletIds.keySet().every { tableName ->
+ sourceTableBytes[tableName] > 0L &&
+ sourceTableBytes[tableName] == targetTableBytes[tableName]
+ }
+ if (cacheMatches) {
+ break
+ }
+ sleep(1000)
}
+ def ttlAfterWarmup = getGlobalTtlBytes()
+ logger.info("Global TTL cache bytes after warmup=${ttlAfterWarmup}, " +
+ "source_minus_target=${ttlAfterWarmup.source - ttlAfterWarmup.target}, " +
+ "baseline_after_clear=${initialTtlAfterClear}, " +
+ "source_change_since_clear=${ttlAfterWarmup.source - initialTtlAfterClear.source}, " +
+ "target_change_since_clear=${ttlAfterWarmup.target - initialTtlAfterClear.target}")
+ assertTrue(cacheMatches, "Case table TTL cache bytes must be positive and match on both clusters: " +
+ "source=${sourceTableBytes}, target=${targetTableBytes}, tablets=${tableTabletIds}")
try {
sql "WARM UP COMPUTE GROUP regression_cluster_name1 WITH COMPUTE GROUP regression_cluster_name2"
@@ -238,27 +249,21 @@ suite("test_warm_up_compute_group") {
sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
sql new File("""${context.file.parent}/../ddl/supplier_delete.sql""").text
- clearFileCache.call(ipList[1], httpPortList[1]);
+ clearCacheAndLogTtl([targetCluster], "after_drop")
jobId_ = sql "WARM UP COMPUTE GROUP regression_cluster_name1 WITH COMPUTE GROUP regression_cluster_name0"
waitJobDone(jobId_)
- sleep(40000)
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def j = line.indexOf(' ')
- assertEquals(0, line.substring(j).toLong())
- flag = true
- break
- }
- }
- assertTrue(flag)
+ long droppedCacheDeadline = System.currentTimeMillis() + 120000L
+ def droppedTableBytes = [:]
+ boolean droppedCacheCleared = false
+ while (System.currentTimeMillis() < droppedCacheDeadline) {
+ droppedTableBytes = getTableTtlBytes(targetCluster)
+ droppedCacheCleared = droppedTableBytes.values().every { it == 0L }
+ if (droppedCacheCleared) {
+ break
+ }
+ logger.info("Waiting for dropped table TTL cache to clear on ${targetCluster}: ${droppedTableBytes}")
+ sleep(1000)
}
+ assertTrue(droppedCacheCleared,
+ "Dropped tables must have no TTL cache on ${targetCluster}: ${droppedTableBytes}")
}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_partition.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_partition.groovy
index fce44d37e36..c73c818a05c 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_partition.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_partition.groovy
@@ -15,184 +15,271 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.WarmupMetricsUtils
-suite("test_warm_up_partition") {
- def custoBeConfig = [
- enable_evict_file_cache_in_advance : false,
- file_cache_enter_disk_resource_limit_mode_percent : 99
- ]
- setBeConfigTemporary(custoBeConfig) {
- def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="12000") """
- def getJobState = { jobId ->
- def jobStateResult = sql """ SHOW WARM UP JOB WHERE ID = ${jobId} """
- return jobStateResult[0]
+suite("test_warm_up_partition", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
}
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
- List<String> ipList = new ArrayList<>();
- List<String> hbPortList = new ArrayList<>()
- List<String> httpPortList = new ArrayList<>()
- List<String> brpcPortList = new ArrayList<>()
- List<String> beUniqueIdList = new ArrayList<>()
-
- String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
- int num = 0
- for(String values : bes) {
- if (num++ == 2) break;
- println("the value is " + values);
- String[] beInfo = values.split(':');
- ipList.add(beInfo[0]);
- hbPortList.add(beInfo[1]);
- httpPortList.add(beInfo[2]);
- beUniqueIdList.add(beInfo[3]);
- brpcPortList.add(beInfo[4]);
+ String sourceCluster = "regression_cluster_name0"
+ String targetCluster = "regression_cluster_name1"
+ String tableName = "customer"
+ def partitionNames = ["p1", "p2", "p3"]
+ long ttlSeconds = 12000L
+ def partitionInfo = [:]
+ def warmupJobs = [] as Set
+ def pendingLoads = [] as Set
+ def backends = sql_return_maparray("SHOW BACKENDS")
+ def clusterBackends = [sourceCluster, targetCluster].collectEntries { clusterName ->
+ def members = backends.findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collect { be ->
+ [id: be.BackendId as Long, ip: be.Host.toString(),
+ httpPort: be.HttpPort.toString(), brpcPort: be.BrpcPort.toString()]
+ }
+ assertTrue(!members.isEmpty(), "No alive backends in ${clusterName}")
+ [(clusterName): members]
}
+ logger.info("Partition warmup backends: ${clusterBackends}")
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
- println("the brpc port is " + brpcPortList);
-
- def s3BucketName = getS3BucketName()
- def s3WithProperties = """WITH S3 (
- |"AWS_ACCESS_KEY" = "${getS3AK()}",
- |"AWS_SECRET_KEY" = "${getS3SK()}",
- |"AWS_ENDPOINT" = "${getS3Endpoint()}",
- |"AWS_REGION" = "${getS3Region()}",
- |"provider" = "${getS3Provider()}")
- |PROPERTIES(
- |"exec_mem_limit" = "8589934592",
- |"load_parallelism" = "3")""".stripMargin()
-
-
-
- sql "use @regression_cluster_name0"
-
- def table = "customer"
- sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
- // create table if not exists
- sql (new File("""${context.file.parent}/../ddl/customer_with_partition.sql""").text + ttlProperties)
- sleep(10000)
-
- def load_customer_once = {
- def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
- def loadLabel = table + "_" + uniqueID
- // load data from cos
- def loadSql = new File("""${context.file.parent}/../ddl/${table}_load.sql""").text.replaceAll("\\\$\\{s3BucketName\\}", s3BucketName)
- loadSql = loadSql.replaceAll("\\\$\\{loadLabel\\}", loadLabel) + s3WithProperties
- sql loadSql
-
- // check load state
- while (true) {
- def stateResult = sql "show load where Label = '${loadLabel}'"
- def loadState = stateResult[stateResult.size() - 1][2].toString()
- if ("CANCELLED".equalsIgnoreCase(loadState)) {
- throw new IllegalStateException("load ${loadLabel} failed.")
- } else if ("FINISHED".equalsIgnoreCase(loadState)) {
- break
+ def waitUntil = { String description, long timeoutMs, Closure condition ->
+ long deadline = System.currentTimeMillis() + timeoutMs
+ while (System.currentTimeMillis() < deadline) {
+ if (condition()) {
+ return
}
- sleep(5000)
+ sleep(1000)
}
+ assertTrue(false, "Timeout after ${timeoutMs}ms: ${description}; see last state in log")
}
-
- def clearFileCache = { ip, port ->
- httpTest {
- endpoint ""
- uri ip + ":" + port + """/api/file_cache?op=clear&sync=true"""
- op "get"
- body ""
+ def getGlobalTtlBytes = {
+ [source: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[sourceCluster], "ttl_cache_size"),
+ target: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[targetCluster], "ttl_cache_size")]
+ }
+ def getPartitionCache = { String clusterName ->
+ sql "use @${clusterName}"
+ def tabletIds = partitionInfo.values().collectMany { it.tabletIds }
+ def beIds = clusterBackends[clusterName].collect { it.id }
+ // Read all cache types: a NORMAL block in an untouched partition is also unexpected.
+ def rows = sql """select tablet_id, lower(type), sum(size), count(*)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${beIds.join(',')})
+ group by tablet_id, lower(type)"""
+ partitionInfo.collectEntries { name, info ->
+ def partitionRows = rows.findAll { (it[0] as Long) in info.tabletIds }
+ def ttlRows = partitionRows.findAll { it[1]?.toString() == "ttl" }
+ def ttlByTablet = ttlRows.collectEntries { [(it[0] as Long): it[2] as Long] }
+ [(name): [ttlBytes: ttlByTablet.values().sum(0L) as Long, ttlByTablet: ttlByTablet,
+ complete: info.tabletIds.every { (ttlByTablet[it] ?: 0L) > 0L },
+ blocks: partitionRows.sum(0L) { it[3] as Long },
+ nonTtlBlocks: partitionRows.findAll { it[1]?.toString() != "ttl" }
+ .sum(0L) { it[3] as Long }]]
}
}
-
- def getMetricsMethod = { ip, port, check_func ->
- httpTest {
- endpoint ip + ":" + port
- uri "/brpc_metrics"
- op "get"
- check check_func
+ def waitForTargetTablets = {
+ sql "use @${targetCluster}"
+ def targetBes = clusterBackends[targetCluster].collectEntries { [(it.id): it] }
+ waitUntil("target primary routing and tablet metadata", 120000L) {
+ def missing = [:]
+ partitionInfo.each { name, info ->
+ def ready = [] as Set
+ def tablets = sql_return_maparray("show tablets from ${tableName} partition ${name}")
+ tablets.each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (info.tabletIds.contains(id) && targetBes.containsKey(primary) &&
+ (tablet.BackendId as Long) == primary) {
+ def be = targetBes[primary]
+ try {
+ // Fetch metadata only; a target data query could populate the cache itself.
+ httpTest {
+ endpoint "${be.ip}:${be.httpPort}"
+ uri "/api/meta/header/${id}"
+ op "get"
+ check { code, body ->
+ if ((code as int) == 200) {
+ def meta = parseJson(body.toString())
+ // Cloud header JSON omits rowsets. The source baseline and
+ // per-partition warm-up assertions verify the loaded data.
+ if ((meta.tablet_id as Long) == id &&
+ (meta.ttl_seconds as Long) == ttlSeconds &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null) {
+ ready.add(id)
+ }
+ }
+ }
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for ${name} tablet ${id} on BE ${primary}: ${e.message}")
+ }
+ }
+ }
+ missing[name] = [tablets: info.tabletIds - ready, loadedSourceVersion: info.version]
+ }
+ logger.info("Target tablet readiness: ${missing}")
+ missing.values().every { it.tablets.isEmpty() }
}
}
- clearFileCache.call(ipList[0], httpPortList[0]);
- clearFileCache.call(ipList[1], httpPortList[1]);
-
- load_customer_once()
+ setBeConfigTemporary([
+ enable_evict_file_cache_in_advance: false,
+ file_cache_enter_disk_resource_limit_mode_percent: 99
+ ]) {
+ try {
+ sql "use @${sourceCluster}"
+ sql "DROP TABLE IF EXISTS ${tableName} FORCE"
+ sql (new File("${context.file.parent}/../ddl/customer_with_partition.sql").text + """
+ PROPERTIES("file_cache_ttl_seconds"="${ttlSeconds}", "disable_auto_compaction"="true")""")
+ partitionNames.each { name ->
+ def ids = sql("show tablets from ${tableName} partition ${name}")
+ .collect { it[0] as Long }.unique()
+ assertTrue(!ids.isEmpty(), "Missing tablets for ${name}")
+ partitionInfo[name] = [tabletIds: ids]
+ }
- def jobId_ = sql "warm up cluster regression_cluster_name1 with table customer partition p1;"
- def waitJobDone = { jobId ->
- int retryTime = 120
- int i = 0
- for (; i < retryTime; i++) {
- sleep(1000)
- def statuses = getJobState(jobId[0][0])
- if (statuses.any { it != null && it.equals("CANCELLED") }) {
- assertTrue(false);
+ def ttlBeforeClear = getGlobalTtlBytes()
+ clusterBackends.values().flatten().each { be ->
+ WarmupMetricsUtils.clearFileCache(be.ip, be.httpPort)
}
- if (statuses.any { it != null && it.equals("FINISHED") }) {
- break;
+ // sync=true can leave held blocks from other tables. Keep global totals for diagnosis only.
+ def ttlAfterClear = getGlobalTtlBytes()
+ logger.info("TTL cache clear: before=${ttlBeforeClear}, after=${ttlAfterClear}, " +
+ "source_minus_target_after_clear=${ttlAfterClear.source - ttlAfterClear.target}, " +
+ "source_change_after_clear=${ttlAfterClear.source - ttlBeforeClear.source}, " +
+ "target_change_after_clear=${ttlAfterClear.target - ttlBeforeClear.target}")
+
+ String loadLabel = "customer_${UUID.randomUUID().toString().replace('-', '')}"
+ String loadSql = new File("${context.file.parent}/../ddl/customer_load.sql").text
+ .replace('${s3BucketName}', getS3BucketName()).replace('${loadLabel}', loadLabel)
+ sql (loadSql + """WITH S3 (
+ "AWS_ACCESS_KEY" = "${getS3AK()}", "AWS_SECRET_KEY" = "${getS3SK()}",
+ "AWS_ENDPOINT" = "${getS3Endpoint()}", "AWS_REGION" = "${getS3Region()}",
+ "provider" = "${getS3Provider()}")
+ PROPERTIES("exec_mem_limit"="8589934592", "load_parallelism"="3")""")
+ pendingLoads.add(loadLabel)
+ waitUntil("load ${loadLabel} to finish", 300000L) {
+ def rows = sql "show load where Label = '${loadLabel}'"
+ logger.info("Partition warmup load state: ${rows}")
+ if (rows.isEmpty()) {
+ return false
+ }
+ String state = rows.last()[2].toString()
+ assertTrue(!state.equalsIgnoreCase("CANCELLED"), "Load ${loadLabel} cancelled: ${rows.last()}")
+ if (state.equalsIgnoreCase("FINISHED")) {
+ pendingLoads.remove(loadLabel)
+ return true
+ }
+ false
}
- }
- if (i == retryTime) {
- sql "cancel warm up job where id = ${jobId[0][0]}"
- assertTrue(false);
- }
- }
- waitJobDone(jobId_)
- jobId_ = sql "warm up cluster regression_cluster_name1 with table customer partition p2;"
- waitJobDone(jobId_)
- jobId_ = sql "warm up cluster regression_cluster_name1 with table customer partition p3;"
- waitJobDone(jobId_)
+ def partitions = sql_return_maparray("show partitions from ${tableName}")
+ partitionNames.each { name ->
+ def partition = partitions.find { it.PartitionName.toString() == name }
+ assertTrue(partition != null && (partition.VisibleVersion as Long) > 1L,
+ "Missing loaded version for ${name}: ${partitions}")
+ partitionInfo[name].version = partition.VisibleVersion as Long
+ }
+ logger.info("Loaded partitions: ${partitionInfo}")
+ waitForTargetTablets()
- sleep(30000)
- long ttl_cache_size = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- flag = true
- break
+ def sourceBaseline = [:]
+ long stableSince = System.currentTimeMillis()
+ waitUntil("source partitions to have stable nonempty TTL cache", 120000L) {
+ def current = getPartitionCache(sourceCluster)
+ logger.info("Source partition cache before warmup: ${current}")
+ if (current != sourceBaseline) {
+ sourceBaseline = current
+ stableSince = System.currentTimeMillis()
}
+ current.values().every { it.complete && it.nonTtlBlocks == 0L } &&
+ System.currentTimeMillis() - stableSince >= 3000L
}
- assertTrue(flag)
- }
+ def targetBefore = getPartitionCache(targetCluster)
+ assertTrue(targetBefore.values().every { it.blocks == 0L },
+ "Target partitions must be cold before warmup: ${targetBefore}")
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ def warmedPartitions = [] as Set
+ partitionNames.each { name ->
+ // Explicit partition warmup does not require hotspot queries or hotspot collection.
+ def job = sql "WARM UP CLUSTER ${targetCluster} WITH TABLE ${tableName} PARTITION ${name}"
+ assertTrue(!job.isEmpty(), "No warmup job returned for ${name}")
+ def jobId = job[0][0]
+ warmupJobs.add(jobId)
+ waitUntil("warmup ${name} job ${jobId} to finish", 120000L) {
+ def rows = sql "SHOW WARM UP JOB WHERE ID = ${jobId}"
+ logger.info("Partition ${name} warmup job ${jobId}: ${rows}")
+ if (rows.isEmpty()) {
+ return false
+ }
+ def states = rows[0].collect { it?.toString() }
+ assertTrue(!states.contains("CANCELLED"), "Warmup ${name} job ${jobId} cancelled: ${rows[0]}")
+ states.contains("FINISHED")
+ }
+ warmedPartitions.add(name)
+ waitUntil("after ${name}: partition TTL cache must match source and untouched partitions stay cold",
+ 120000L) {
+ def source = getPartitionCache(sourceCluster)
+ def target = getPartitionCache(targetCluster)
+ logger.info("After ${name} job ${jobId}: warmed=${warmedPartitions}, " +
+ "source=${source}, target=${target}, baseline=${sourceBaseline}")
+ // Fail immediately on scope/type violations; waiting must not hide unintended warmup.
+ partitionNames.each { partitionName ->
+ if (!warmedPartitions.contains(partitionName)) {
+ assertTrue(target[partitionName].blocks == 0L,
+ "Untouched partition ${partitionName} has cache after ${name}: ${target}")
+ } else {
+ assertTrue(target[partitionName].nonTtlBlocks == 0L,
+ "Expected only TTL cache for ${partitionName}: ${target[partitionName]}")
+ }
+ }
+ partitionNames.every { partitionName ->
+ def src = source[partitionName]
+ def dst = target[partitionName]
+ src.complete && src.nonTtlBlocks == 0L &&
+ src.ttlByTablet == sourceBaseline[partitionName].ttlByTablet &&
+ (!warmedPartitions.contains(partitionName) ||
+ (dst.complete && dst.ttlBytes == src.ttlBytes && dst.ttlByTablet == src.ttlByTablet))
}
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
}
}
- assertTrue(flag)
- }
-
- try {
- sql "warm up cluster regression_cluster_name1 with table customer partition p5;"
- assertTrue(false)
- } catch (Exception e) {
- assertTrue(true)
- }
- sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
+ sql "use @${targetCluster}"
+ test {
+ sql "WARM UP CLUSTER ${targetCluster} WITH TABLE ${tableName} PARTITION p5"
+ exception "The partition p5 doesn't exist"
+ }
+ } finally {
+ warmupJobs.each { jobId ->
+ try {
+ def rows = sql "SHOW WARM UP JOB WHERE ID = ${jobId}"
+ if (!rows.isEmpty() && rows[0].any { it?.toString() in ["PENDING", "RUNNING"] }) {
+ sql "CANCEL WARM UP JOB WHERE ID = ${jobId}"
+ }
+ } catch (Exception e) {
+ logger.warn("Failed to clean up warmup ${jobId}: ${e.message}")
+ }
+ }
+ // Keep DROP in finally so even a cleanup SQL failure does not leave the test table behind.
+ try {
+ sql "use @${sourceCluster}"
+ pendingLoads.each { label ->
+ try {
+ sql "CANCEL LOAD WHERE LABEL = '${label}'"
+ } catch (Exception e) {
+ logger.warn("Failed to clean up load ${label}: ${e.message}")
+ }
+ }
+ } finally {
+ sql "DROP TABLE IF EXISTS ${tableName} FORCE"
+ }
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_same_table_multi_times.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_same_table_multi_times.groovy
index b774273f952..cc09dabff4e 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_same_table_multi_times.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_same_table_multi_times.groovy
@@ -17,7 +17,19 @@
import org.codehaus.groovy.runtime.IOGroovyMethods
-suite("test_warm_up_same_table_multi_times") {
+suite("test_warm_up_same_table_multi_times", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
+ }
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
+
def custoBeConfig = [
enable_evict_file_cache_in_advance : false,
file_cache_enter_disk_resource_limit_mode_percent : 99
@@ -115,6 +127,26 @@ suite("test_warm_up_same_table_multi_times") {
}
}
+ def createdWarmUpJobIds = [] as Set
+ def recordWarmUpJob = { jobId ->
+ if (jobId != null && !jobId.isEmpty()) {
+ createdWarmUpJobIds.add(jobId[0][0])
+ }
+ }
+ def cleanupWarmUpJobs = {
+ createdWarmUpJobIds.each { jobId ->
+ try {
+ def statuses = getJobState(jobId)
+ if (statuses.any { it != null && (it.equals("PENDING") || it.equals("RUNNING")) }) {
+ sql "cancel warm up job where id = ${jobId}"
+ }
+ } catch (Exception e) {
+ logger.info("ignore warm up job cleanup failure, job id: ${jobId}, error: ${e.getMessage()}")
+ }
+ }
+ }
+
+ try {
clearFileCache.call();
sleep(30000)
@@ -124,11 +156,12 @@ suite("test_warm_up_same_table_multi_times") {
load_customer_once()
def jobId = sql "warm up cluster ${validCluster} with table customer;"
+ recordWarmUpJob(jobId)
try {
sql "warm up cluster ${validCluster} with table customer;"
- assertTrue(true) // dup warm up command can be send to fe queue now
- } catch (Exception e) {
assertTrue(false)
+ } catch (Exception e) {
+ assertTrue("${e.getMessage()}".contains("already has a pending job"))
}
int retryTime = 120
int j = 0
@@ -191,6 +224,7 @@ suite("test_warm_up_same_table_multi_times") {
// AGAIN!
jobId = sql "warm up cluster ${validCluster} with table customer;"
+ recordWarmUpJob(jobId)
retryTime = 120
j = 0
@@ -273,5 +307,8 @@ suite("test_warm_up_same_table_multi_times") {
long diff = skip_io_bytes_end - skip_io_bytes_start;
println("skip_io_bytes diff: " + diff);
assertTrue(diff > 1000);
+ } finally {
+ cleanupWarmUpJobs()
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_table.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_table.groovy
index 4e4979ab79c..0b5da4bcdd3 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_table.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_table.groovy
@@ -15,9 +15,22 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.Http
+import org.apache.doris.regression.util.WarmupMetricsUtils
+
+suite("test_warm_up_table", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
+ }
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
-suite("test_warm_up_table") {
def custoBeConfig = [
enable_evict_file_cache_in_advance : false,
file_cache_enter_disk_resource_limit_mode_percent : 99
@@ -113,17 +126,104 @@ suite("test_warm_up_table") {
}
}
- def getMetricsMethod = { ip, port, check_func ->
- httpTest {
- endpoint ip + ":" + port
- uri "/brpc_metrics"
- op "get"
- check check_func
+ def clusterBackends = ["regression_cluster_name0", "regression_cluster_name1"].collectEntries { name ->
+ def members = sql_return_maparray("SHOW BACKENDS").findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == name
+ }.collect { be ->
+ [id: be.BackendId as Long, ip: be.Host.toString(),
+ httpPort: be.HttpPort.toString(), brpcPort: be.BrpcPort.toString()]
+ }
+ assertTrue(!members.isEmpty(), "No alive backends in ${name}")
+ [(name): members]
+ }
+ def tabletIds = []
+ def waitUntil = { String stage, long timeoutMs, Closure ready ->
+ long deadline = System.currentTimeMillis() + timeoutMs
+ while (System.currentTimeMillis() < deadline) {
+ if (ready()) { return }
+ sleep(1000)
+ }
+ assertTrue(false, "Timeout waiting for ${stage}; see last cache state in log")
+ }
+ def getGlobalTtl = {
+ clusterBackends.collectEntries { name, members ->
+ [(name): WarmupMetricsUtils.getBackendMetricSum(members, "ttl_cache_size")]
+ }
+ }
+ def getTableCache = { String clusterName ->
+ sql "use @${clusterName}"
+ def beIds = clusterBackends[clusterName].collect { it.id }
+ def rows = sql """select tablet_id, lower(type), sum(size)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${beIds.join(',')})
+ group by tablet_id, lower(type)"""
+ def ttlByTablet = rows.findAll { it[1]?.toString() == "ttl" }
+ .collectEntries { [(it[0] as Long): it[2] as Long] }
+ [ttlByTablet: ttlByTablet, complete: tabletIds.every { (ttlByTablet[it] ?: 0L) > 0L },
+ nonTtlBytes: rows.findAll { it[1]?.toString() != "ttl" }.sum(0L) { it[2] as Long }]
+ }
+ def waitForTargetTablets = {
+ sql "use @regression_cluster_name1"
+ def targetBes = clusterBackends.regression_cluster_name1.collectEntries { [(it.id): it] }
+ waitUntil("target primary tablet metadata", 120000L) {
+ def ready = [] as Set
+ sql_return_maparray("show tablets from ${table}").each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (id in tabletIds && targetBes.containsKey(primary) && (tablet.BackendId as Long) == primary) {
+ def be = targetBes[primary]
+ try {
+ // Metadata only: a business read here would populate the target cache.
+ def connection = Http.openConnection(
+ "http://${be.ip}:${be.httpPort}/api/meta/header/${id}")
+ connection.connectTimeout = 5000
+ connection.readTimeout = 5000
+ try {
+ def meta = connection.inputStream.withCloseable {
+ parseJson(it.getText("UTF-8"))
+ }
+ if ((meta.tablet_id as Long) == id && (meta.ttl_seconds as Long) == 12000L &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null) {
+ ready.add(id)
+ }
+ } finally {
+ connection.disconnect()
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for target tablet ${id} on BE ${primary}: ${e.message}")
+ }
+ }
+ }
+ logger.info("Target tablet metadata: missing=${tabletIds - ready}")
+ ready.size() == tabletIds.size()
}
}
- clearFileCache.call(ipList[0], httpPortList[0]);
- clearFileCache.call(ipList[1], httpPortList[1]);
+ def createdWarmUpJobIds = [] as Set
+ def recordWarmUpJob = { jobId ->
+ if (jobId != null && !jobId.isEmpty()) {
+ createdWarmUpJobIds.add(jobId[0][0])
+ }
+ }
+ def cleanupWarmUpJobs = {
+ createdWarmUpJobIds.each { jobId ->
+ try {
+ def statuses = getJobState(jobId)
+ if (statuses.any { it != null && (it.equals("PENDING") || it.equals("RUNNING")) }) {
+ sql "cancel warm up job where id = ${jobId}"
+ }
+ } catch (Exception e) {
+ logger.info("ignore warm up job cleanup failure, job id: ${jobId}, error: ${e.getMessage()}")
+ }
+ }
+ }
+
+ try {
+ def ttlBeforeClear = getGlobalTtl()
+ clusterBackends.values().flatten().each { be -> clearFileCache.call(be.ip, be.httpPort) }
+ def ttlAfterClear = getGlobalTtl()
+ logger.info("Single-table global TTL clear: before=${ttlBeforeClear}, after=${ttlAfterClear}")
sleep(30000)
load_customer_once()
@@ -140,12 +240,30 @@ suite("test_warm_up_table") {
load_customer_once()
load_customer_once()
+ tabletIds = sql_return_maparray("show tablets from ${table}")
+ .collect { it.TabletId as Long }.unique().sort()
+ assertTrue(!tabletIds.isEmpty(), "No tablets for ${table}")
+ logger.info("Single-table warmup tablets=${tabletIds}, backends=${clusterBackends}")
+ def sourceCache = null
+ def candidate = null
+ long stableSince = 0L
+ waitUntil("stable nonempty source TTL cache", 120000L) {
+ sourceCache = getTableCache("regression_cluster_name0")
+ boolean complete = sourceCache.complete && sourceCache.nonTtlBytes == 0L
+ if (!complete || sourceCache != candidate) { stableSince = System.currentTimeMillis() }
+ candidate = sourceCache
+ logger.info("Single-table source TTL baseline=${sourceCache}")
+ complete && System.currentTimeMillis() - stableSince >= 3000L
+ }
+ waitForTargetTablets()
+
def jobId = sql "warm up cluster regression_cluster_name1 with table customer;"
+ recordWarmUpJob(jobId)
try {
sql "warm up cluster regression_cluster_name1 with table customer;"
- assertTrue(true)
- } catch (Exception e) {
assertTrue(false)
+ } catch (Exception e) {
+ assertTrue("${e.getMessage()}".contains("already has a pending job"))
}
int retryTime = 120
int j = 0
@@ -166,46 +284,20 @@ suite("test_warm_up_table") {
def tablesString = getTablesFromShowCommand(jobId[0][0])
assertTrue(tablesString.any { it != null && it.contains("customer") })
- sleep(30000)
- long ttl_cache_size = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- flag = true
- break
- }
- }
- assertTrue(flag)
- }
-
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
- }
- }
- assertTrue(flag)
+ // Global TTL includes residual blocks from other tables; assert this table on the target BEs.
+ try {
+ waitUntil("target per-tablet TTL cache to equal the nonzero source baseline", 180000L) {
+ def currentSource = getTableCache("regression_cluster_name0")
+ def targetCache = getTableCache("regression_cluster_name1")
+ logger.info("Single-table TTL cache: source=${currentSource}, target=${targetCache}")
+ currentSource == sourceCache && targetCache.complete && targetCache.nonTtlBytes == 0L &&
+ targetCache.ttlByTablet == sourceCache.ttlByTablet
+ }
+ } finally {
+ def ttlAfterWarmup = getGlobalTtl()
+ def delta = ttlAfterWarmup.collectEntries { name, bytes -> [(name): bytes - ttlAfterClear[name]] }
+ logger.info("Single-table global TTL after warmup=${ttlAfterWarmup}, " +
+ "baseline_after_clear=${ttlAfterClear}, delta_since_clear=${delta}")
}
try {
@@ -222,5 +314,9 @@ suite("test_warm_up_table") {
} catch (Exception e) {
assertTrue(true)
}
+ } finally {
+ cleanupWarmUpJobs()
+ sql "DROP TABLE IF EXISTS ${table} FORCE"
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_tables.groovy b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_tables.groovy
index be4521544fe..9a8e1281deb 100644
--- a/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_tables.groovy
+++ b/regression-test/suites/cloud_p0/cache/multi_cluster/warm_up/table/test_warm_up_tables.groovy
@@ -15,233 +15,325 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.WarmupMetricsUtils
-suite("test_warm_up_tables") {
- def custoBeConfig = [
- enable_evict_file_cache_in_advance : false,
- file_cache_enter_disk_resource_limit_mode_percent : 99
- ]
-
- setBeConfigTemporary(custoBeConfig) {
- def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="12000") """
- def getJobState = { jobId ->
- def jobStateResult = sql """ SHOW WARM UP JOB WHERE ID = ${jobId} """
- return jobStateResult[0]
- }
- def getTablesFromShowCommand = { jobId ->
- def jobStateResult = sql """ SHOW WARM UP JOB WHERE ID = ${jobId} """
- return jobStateResult[0]
+suite("test_warm_up_tables", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
}
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
- List<String> ipList = new ArrayList<>();
- List<String> hbPortList = new ArrayList<>()
- List<String> httpPortList = new ArrayList<>()
- List<String> brpcPortList = new ArrayList<>()
- List<String> beUniqueIdList = new ArrayList<>()
-
- String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
- int num = 0
- for(String values : bes) {
- if (num++ == 2) break;
- println("the value is " + values);
- String[] beInfo = values.split(':');
- ipList.add(beInfo[0]);
- hbPortList.add(beInfo[1]);
- httpPortList.add(beInfo[2]);
- beUniqueIdList.add(beInfo[3]);
- brpcPortList.add(beInfo[4]);
+ String sourceCluster = "regression_cluster_name0"
+ String targetCluster = "regression_cluster_name1"
+ def tableDefinitions = [customer: "customer_with_partition", supplier: "supplier"]
+ long ttlSeconds = 12000L
+ def partitionInfo = [
+ "customer.p1": [table: "customer", partition: "p1"],
+ "customer.p2": [table: "customer", partition: "p2"],
+ "customer.p3": [table: "customer", partition: "p3"],
+ "supplier": [table: "supplier", partition: null]
+ ]
+ def showTabletsSql = { info ->
+ "show tablets from ${info.table}" + (info.partition ? " partition ${info.partition}" : "")
}
-
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
- println("the brpc port is " + brpcPortList);
-
- def s3BucketName = getS3BucketName()
- def s3WithProperties = """WITH S3 (
- |"AWS_ACCESS_KEY" = "${getS3AK()}",
- |"AWS_SECRET_KEY" = "${getS3SK()}",
- |"AWS_ENDPOINT" = "${getS3Endpoint()}",
- |"AWS_REGION" = "${getS3Region()}",
- |"provider" = "${getS3Provider()}")
- |PROPERTIES(
- |"exec_mem_limit" = "8589934592",
- |"load_parallelism" = "3")""".stripMargin()
-
-
- sql """ SET PROPERTY 'default_cloud_cluster' = "regression_cluster_name0"; """
- sql "use @regression_cluster_name0"
-
- def table = "customer"
- sql new File("""${context.file.parent}/../ddl/${table}_delete.sql""").text
- sql new File("""${context.file.parent}/../ddl/supplier_delete.sql""").text
- // create table if not exists
- sql (new File("""${context.file.parent}/../ddl/customer_with_partition.sql""").text + ttlProperties)
- sql (new File("""${context.file.parent}/../ddl/supplier.sql""").text + ttlProperties)
- sleep(10000)
-
- def load_customer_once = {
- def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
- def loadLabel = table + "_" + uniqueID
- // load data from cos
- def loadSql = new File("""${context.file.parent}/../ddl/${table}_load.sql""").text.replaceAll("\\\$\\{s3BucketName\\}", s3BucketName)
- loadSql = loadSql.replaceAll("\\\$\\{loadLabel\\}", loadLabel) + s3WithProperties
- sql loadSql
-
- // check load state
- while (true) {
- def stateResult = sql "show load where Label = '${loadLabel}'"
- def loadState = stateResult[stateResult.size() - 1][2].toString()
- if ("CANCELLED".equalsIgnoreCase(loadState)) {
- throw new IllegalStateException("load ${loadLabel} failed.")
- } else if ("FINISHED".equalsIgnoreCase(loadState)) {
- break
- }
- sleep(5000)
+ def warmupJobs = [] as Set
+ def pendingLoads = [] as Set
+ def backends = sql_return_maparray("SHOW BACKENDS")
+ def clusterBackends = [sourceCluster, targetCluster].collectEntries { clusterName ->
+ def members = backends.findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collect { be ->
+ [id: be.BackendId as Long, ip: be.Host.toString(),
+ httpPort: be.HttpPort.toString(), brpcPort: be.BrpcPort.toString()]
}
+ assertTrue(!members.isEmpty(), "No alive backends in ${clusterName}")
+ [(clusterName): members]
}
+ logger.info("Multi-table warmup backends: ${clusterBackends}")
- def load_supplier_once = {
- def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
- def loadLabel = "supplier_" + uniqueID
- // load data from cos
- def loadSql = new File("""${context.file.parent}/../ddl/supplier_load.sql""").text.replaceAll("\\\$\\{s3BucketName\\}", s3BucketName)
- loadSql = loadSql.replaceAll("\\\$\\{loadLabel\\}", loadLabel) + s3WithProperties
- sql loadSql
-
- // check load state
- while (true) {
- def stateResult = sql "show load where Label = '${loadLabel}'"
- def loadState = stateResult[stateResult.size() - 1][2].toString()
- if ("CANCELLED".equalsIgnoreCase(loadState)) {
- throw new IllegalStateException("load ${loadLabel} failed.")
- } else if ("FINISHED".equalsIgnoreCase(loadState)) {
- break
+ def waitUntil = { String description, long timeoutMs, Closure condition ->
+ long deadline = System.currentTimeMillis() + timeoutMs
+ while (System.currentTimeMillis() < deadline) {
+ if (condition()) {
+ return
}
- sleep(5000)
+ sleep(1000)
}
+ assertTrue(false, "Timeout after ${timeoutMs}ms: ${description}; see last state in log")
}
-
- def clearFileCache = { ip, port ->
- httpTest {
- endpoint ""
- uri ip + ":" + port + """/api/file_cache?op=clear&sync=true"""
- op "get"
- body ""
+ def getGlobalTtlBytes = {
+ [source: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[sourceCluster], "ttl_cache_size"),
+ target: WarmupMetricsUtils.getBackendMetricSum(clusterBackends[targetCluster], "ttl_cache_size")]
+ }
+ def getPartitionCache = { String clusterName ->
+ sql "use @${clusterName}"
+ def tabletIds = partitionInfo.values().collectMany { it.tabletIds }
+ def beIds = clusterBackends[clusterName].collect { it.id }
+ // Read all cache types: a NORMAL block in an untouched partition is also unexpected.
+ def rows = sql """select tablet_id, lower(type), sum(size), count(*)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${beIds.join(',')})
+ group by tablet_id, lower(type)"""
+ partitionInfo.collectEntries { name, info ->
+ def partitionRows = rows.findAll { (it[0] as Long) in info.tabletIds }
+ def ttlRows = partitionRows.findAll { it[1]?.toString() == "ttl" }
+ def ttlByTablet = ttlRows.collectEntries { [(it[0] as Long): it[2] as Long] }
+ [(name): [ttlBytes: ttlByTablet.values().sum(0L) as Long, ttlByTablet: ttlByTablet,
+ bytesByType: partitionRows.groupBy { it[1]?.toString() }.collectEntries { type, entries ->
+ [(type): entries.sum(0L) { it[2] as Long }]
+ },
+ complete: info.tabletIds.every { (ttlByTablet[it] ?: 0L) > 0L },
+ blocks: partitionRows.sum(0L) { it[3] as Long },
+ nonTtlBlocks: partitionRows.findAll { it[1]?.toString() != "ttl" }
+ .sum(0L) { it[3] as Long }]]
}
}
-
- def getMetricsMethod = { ip, port, check_func ->
- httpTest {
- endpoint ip + ":" + port
- uri "/brpc_metrics"
- op "get"
- check check_func
+ def waitForTargetTablets = {
+ sql "use @${targetCluster}"
+ def targetBes = clusterBackends[targetCluster].collectEntries { [(it.id): it] }
+ waitUntil("target primary routing and tablet metadata", 120000L) {
+ def missing = [:]
+ partitionInfo.each { name, info ->
+ def ready = [] as Set
+ def tablets = sql_return_maparray(showTabletsSql(info))
+ tablets.each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (info.tabletIds.contains(id) && targetBes.containsKey(primary) &&
+ (tablet.BackendId as Long) == primary) {
+ def be = targetBes[primary]
+ try {
+ // Fetch metadata only; a target data query could populate the cache itself.
+ httpTest {
+ endpoint "${be.ip}:${be.httpPort}"
+ uri "/api/meta/header/${id}"
+ op "get"
+ check { code, body ->
+ if ((code as int) == 200) {
+ def meta = parseJson(body.toString())
+ // Cloud header JSON omits rowsets. Source data and cache
+ // baselines are checked separately from target readiness.
+ if ((meta.tablet_id as Long) == id &&
+ (meta.ttl_seconds as Long) == ttlSeconds &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null) {
+ ready.add(id)
+ }
+ }
+ }
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for ${name} tablet ${id} on BE ${primary}: ${e.message}")
+ }
+ }
+ }
+ missing[name] = [tablets: info.tabletIds - ready, loadedSourceVersion: info.version]
+ }
+ logger.info("Target tablet readiness: ${missing}")
+ missing.values().every { it.tablets.isEmpty() }
}
}
- clearFileCache.call(ipList[0], httpPortList[0]);
- clearFileCache.call(ipList[1], httpPortList[1]);
-
- load_customer_once()
- load_supplier_once()
-
- def jobId_ = sql "warm up cluster regression_cluster_name1 with table customer partition p1 and table customer partition p2;"
- def waitJobDone = { jobId ->
- int retryTime = 120
- int i = 0
- for (; i < retryTime; i++) {
- sleep(1000)
- def statuses = getJobState(jobId[0][0])
- if (statuses.any { it != null && it.equals("CANCELLED") }) {
- assertTrue(false);
- }
- if (statuses.any { it != null && it.equals("FINISHED") }) {
- break;
+ def clearCaches = { String phase, List members ->
+ def before = getGlobalTtlBytes()
+ members.each { be ->
+ httpTest {
+ endpoint "${be.ip}:${be.httpPort}"
+ uri "/api/file_cache?op=clear&sync=true"
+ op "get"
+ check { code, body ->
+ assertTrue((code as int) == 200, "Clear cache failed on BE ${be.id}: ${code}, ${body}")
+ def response = parseJson(body.toString())
+ assertTrue(response.status == "OK", "Clear cache failed on BE ${be.id}: ${body}")
+ logger.info("${phase} clear BE ${be.id}: ${body}")
+ }
}
}
- if (i == retryTime) {
- sql "cancel warm up job where id = ${jobId[0][0]}"
- assertTrue(false);
+ def after = getGlobalTtlBytes()
+ // Busy blocks belonging to other tables must not determine whether this case passes.
+ logger.info("${phase} global TTL: before=${before}, after=${after}, " +
+ "source_minus_target_after_clear=${after.source - after.target}, " +
+ "source_change_after_clear=${after.source - before.source}, " +
+ "target_change_after_clear=${after.target - before.target}")
+ }
+ def waitForTargetCold = { String phase ->
+ waitUntil("${phase}: target test-table cache cleanup incomplete", 120000L) {
+ def cache = getPartitionCache(targetCluster)
+ logger.info("${phase} target cache: ${cache}")
+ cache.values().every { it.blocks == 0L }
}
}
- waitJobDone(jobId_)
- jobId_ = sql "warm up cluster regression_cluster_name1 with table customer partition p3 and table supplier;"
- waitJobDone(jobId_);
-
- def tablesString = getTablesFromShowCommand(jobId_[0][0])
- assertTrue(tablesString.any { it != null && it.contains("customer") })
- assertTrue(tablesString.any { it != null && it.contains("supplier") })
+ setBeConfigTemporary([
+ enable_evict_file_cache_in_advance: false,
+ file_cache_enter_disk_resource_limit_mode_percent: 99
+ ]) {
+ try {
+ sql "use @${sourceCluster}"
+ String dbName = sql("select database()")[0][0].toString()
+ assertTrue(!dbName.isEmpty(), "Missing database for warmup Tables assertions")
+ tableDefinitions.each { tableName, ddlName ->
+ sql "DROP TABLE IF EXISTS ${tableName} FORCE"
+ sql (new File("${context.file.parent}/../ddl/${ddlName}.sql").text + """
+ PROPERTIES("file_cache_ttl_seconds"="${ttlSeconds}", "disable_auto_compaction"="true")""")
+ }
+ partitionInfo.each { name, info ->
+ def ids = sql(showTabletsSql(info)).collect { it[0] as Long }.unique()
+ assertTrue(!ids.isEmpty(), "Missing tablets for ${name}")
+ info.tabletIds = ids
+ }
+ clearCaches("initial", clusterBackends.values().flatten())
- sleep(30000)
- long ttl_cache_size = 0
- getMetricsMethod.call(ipList[0], brpcPortList[0]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ tableDefinitions.keySet().each { tableName ->
+ String label = "${tableName}_${UUID.randomUUID().toString().replace('-', '')}"
+ String loadSql = new File("${context.file.parent}/../ddl/${tableName}_load.sql").text
+ .replace('${s3BucketName}', getS3BucketName()).replace('${loadLabel}', label)
+ sql (loadSql + """WITH S3 (
+ "AWS_ACCESS_KEY" = "${getS3AK()}", "AWS_SECRET_KEY" = "${getS3SK()}",
+ "AWS_ENDPOINT" = "${getS3Endpoint()}", "AWS_REGION" = "${getS3Region()}",
+ "provider" = "${getS3Provider()}")
+ PROPERTIES("exec_mem_limit"="8589934592", "load_parallelism"="3")""")
+ pendingLoads.add(label)
+ waitUntil("load ${label} to finish", 300000L) {
+ def rows = sql "show load where Label = '${label}'"
+ logger.info("Multi-table warmup load state: ${rows}")
+ if (rows.isEmpty()) {
+ return false
}
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- flag = true
- break
+ String state = rows.last()[2].toString()
+ assertTrue(!state.equalsIgnoreCase("CANCELLED"), "Load ${label} cancelled: ${rows.last()}")
+ if (state.equalsIgnoreCase("FINISHED")) {
+ pendingLoads.remove(label)
+ return true
+ }
+ false
}
- }
- assertTrue(flag)
- }
-
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ def partitions = sql_return_maparray("show partitions from ${tableName}")
+ partitionInfo.findAll { name, info -> info.table == tableName }.each { name, info ->
+ def matching = partitions.findAll {
+ info.partition == null || it.PartitionName.toString() == info.partition
}
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
+ assertTrue(matching.size() == 1 && (matching[0].VisibleVersion as Long) > 1L,
+ "Missing loaded version for ${name}: ${partitions}")
+ info.version = matching[0].VisibleVersion as Long
}
}
- assertTrue(flag)
- }
+ logger.info("Loaded table/partition tablets: ${partitionInfo}")
+ waitForTargetTablets()
- clearFileCache.call(ipList[1], httpPortList[1]);
- jobId_ = sql "warm up cluster regression_cluster_name1 with table customer and table supplier;"
- waitJobDone(jobId_);
+ def sourceBaseline = [:]
+ long stableSince = System.currentTimeMillis()
+ waitUntil("source tables to have stable nonempty TTL cache", 120000L) {
+ def current = getPartitionCache(sourceCluster)
+ logger.info("Source cache before warmup: ${current}")
+ if (current != sourceBaseline) {
+ sourceBaseline = current
+ stableSince = System.currentTimeMillis()
+ }
+ current.values().every { it.complete && it.nonTtlBlocks == 0L } &&
+ System.currentTimeMillis() - stableSince >= 3000L
+ }
+ waitForTargetCold("before first warmup")
- sleep(30000)
- getMetricsMethod.call(ipList[1], brpcPortList[1]) {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ def stages = [
+ [name: "customer_p1_p2",
+ clause: "TABLE customer PARTITION p1 AND TABLE customer PARTITION p2",
+ tables: ["customer.p1", "customer.p2"], warmed: ["customer.p1", "customer.p2"]],
+ [name: "customer_p3_and_supplier",
+ clause: "TABLE customer PARTITION p3 AND TABLE supplier",
+ tables: ["customer.p3", "supplier"], warmed: partitionInfo.keySet().toList()],
+ [name: "whole_tables", clause: "TABLE customer AND TABLE supplier",
+ tables: ["customer", "supplier"], warmed: partitionInfo.keySet().toList()]
+ ]
+ stages.each { stage ->
+ if (stage.name == "whole_tables") {
+ // Keep the source baseline, but require both target tables to become cold again.
+ clearCaches("before whole_tables", clusterBackends[targetCluster])
+ waitForTargetCold("before whole_tables")
+ }
+ sql "use @${targetCluster}"
+ // These explicit table/partition jobs do not depend on hotspot collection.
+ def job = sql "WARM UP CLUSTER ${targetCluster} WITH ${stage.clause}"
+ assertTrue(!job.isEmpty(), "No warmup job returned for ${stage.name}")
+ def jobId = job[0][0]
+ warmupJobs.add(jobId)
+ def expectedTables = stage.tables.collect { "${dbName}.${it}".toString() }.sort()
+ waitUntil("warmup ${stage.name} job ${jobId} to finish", 120000L) {
+ def rows = sql_return_maparray("SHOW WARM UP JOB WHERE ID = ${jobId}")
+ logger.info("${stage.name} warmup job ${jobId}: ${rows}")
+ if (rows.isEmpty()) {
+ return false
+ }
+ assertTrue(rows.size() == 1, "Unexpected warmup rows for ${jobId}: ${rows}")
+ def state = rows[0]
+ assertTrue(state.JobId.toString() == jobId.toString(), "Wrong warmup job: ${state}")
+ assertTrue(state.DstComputeGroup == targetCluster, "Wrong warmup target: ${state}")
+ assertTrue(state.Type == "TABLE", "Wrong warmup type: ${state}")
+ def actualTables = (state.Tables ?: "").toString().split(',').collect { it.trim() }.sort()
+ assertTrue(actualTables == expectedTables,
+ "Wrong Tables for ${stage.name}: expected=${expectedTables}, actual=${actualTables}")
+ assertTrue(state.Status != "CANCELLED", "Warmup ${stage.name} cancelled: ${state}")
+ state.Status == "FINISHED"
+ }
+ waitUntil("after ${stage.name}: per-tablet TTL must match source and untouched tables stay cold",
+ 120000L) {
+ def source = getPartitionCache(sourceCluster)
+ def target = getPartitionCache(targetCluster)
+ logger.info("After ${stage.name} job ${jobId}: source=${source}, target=${target}, " +
+ "baseline=${sourceBaseline}")
+ partitionInfo.keySet().each { name ->
+ if (!stage.warmed.contains(name)) {
+ assertTrue(target[name].blocks == 0L,
+ "Untouched ${name} has cache after ${stage.name}: ${target[name]}")
+ } else {
+ assertTrue(target[name].nonTtlBlocks == 0L,
+ "Expected only TTL cache for ${name}: ${target[name]}")
+ }
+ }
+ partitionInfo.keySet().every { name ->
+ def src = source[name]
+ def dst = target[name]
+ src.complete && src.nonTtlBlocks == 0L &&
+ src.ttlByTablet == sourceBaseline[name].ttlByTablet &&
+ (!stage.warmed.contains(name) ||
+ (dst.complete && dst.ttlBytes == src.ttlBytes && dst.ttlByTablet == src.ttlByTablet))
}
- def i = line.indexOf(' ')
- assertEquals(ttl_cache_size, line.substring(i).toLong())
- flag = true
- break
}
}
- assertTrue(flag)
- }
+ } finally {
+ warmupJobs.each { jobId ->
+ try {
+ def rows = sql_return_maparray("SHOW WARM UP JOB WHERE ID = ${jobId}")
+ if (!rows.isEmpty() && rows[0].Status in ["PENDING", "RUNNING"]) {
+ sql "CANCEL WARM UP JOB WHERE ID = ${jobId}"
+ }
+ } catch (Exception e) {
+ logger.warn("Failed to clean up warmup ${jobId}: ${e.message}")
+ }
+ }
+ try {
+ sql "use @${sourceCluster}"
+ pendingLoads.each { label ->
+ try {
+ sql "CANCEL LOAD WHERE LABEL = '${label}'"
+ } catch (Exception e) {
+ logger.warn("Failed to clean up load ${label}: ${e.message}")
+ }
+ }
+ } finally {
+ try {
+ sql "DROP TABLE IF EXISTS customer FORCE"
+ } finally {
+ sql "DROP TABLE IF EXISTS supplier FORCE"
+ }
+ }
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/test_clear_file_cache_on_load_failure.groovy b/regression-test/suites/cloud_p0/cache/test_clear_file_cache_on_load_failure.groovy
index 88d89eea0bf..a9705f6c03f 100644
--- a/regression-test/suites/cloud_p0/cache/test_clear_file_cache_on_load_failure.groovy
+++ b/regression-test/suites/cloud_p0/cache/test_clear_file_cache_on_load_failure.groovy
@@ -15,71 +15,38 @@
// specific language governing permissions and limitations
// under the License.
-import groovy.json.JsonSlurper
+import org.apache.doris.regression.util.WarmupMetricsUtils
suite("test_clear_file_cache_on_load_failure", "nonConcurrent") {
if (!isCloudMode()) {
return
}
+ String originalAutoAnalyze = sql("SHOW GLOBAL VARIABLES LIKE 'enable_auto_analyze'")[0][1].toString()
+ assertTrue(originalAutoAnalyze.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected auto-analyze setting: ${originalAutoAnalyze}")
+ onFinish {
+ sql "SET GLOBAL enable_auto_analyze = ${originalAutoAnalyze}"
+ logger.info("Restored enable_auto_analyze=${originalAutoAnalyze}")
+ }
+
// Clear any existing debug points
GetDebugPoint().clearDebugPointsForAllFEs()
GetDebugPoint().clearDebugPointsForAllBEs()
- // Helper function to clear file cache on all backends
- def clearFileCache = { ip, port ->
- def url = "http://${ip}:${port}/api/file_cache?op=clear&sync=true"
- def response = new URL(url).text
- def json = new JsonSlurper().parseText(response)
- if (json.status != "OK") {
- throw new RuntimeException("Clear cache on ${ip}:${port} failed: ${json.status}")
- }
- }
-
- def clearFileCacheOnAllBackends = {
- def backends = sql """SHOW BACKENDS"""
- for (be in backends) {
- def ip = be[1]
- def port = be[4]
- clearFileCache(ip, port)
+ def getCacheBackends = {
+ return (sql """SHOW BACKENDS""").collect { be ->
+ [ip: be[1], httpPort: be[4], brpcPort: be[5]]
}
- // Wait for async clear to complete
- sleep(5000)
}
- // Helper function to get brpc metrics
- def getBrpcMetrics = { ip, port, name ->
- def url = "http://${ip}:${port}/brpc_metrics"
- try {
- def metrics = new URL(url).text
- def matcher = metrics =~ ~"${name}\\s+(\\d+)"
- if (matcher.find()) {
- return matcher[0][1] as long
- }
- } catch (Exception e) {
- logger.warn("Failed to get brpc metrics from ${ip}:${port}: ${e.message}")
+ def requestFileCacheClearOnAllBackends = {
+ getCacheBackends().each { be ->
+ WarmupMetricsUtils.clearFileCache(be.ip.toString(), be.httpPort.toString())
}
- return 0L
- }
-
- // Helper function to get index queue cache size
- def getIndexQueueSize = { ip, port ->
- return getBrpcMetrics(ip, port, "file_cache_index_queue_cache_size")
}
- // Helper function to get total index queue cache size across all backends
- def getTotalIndexQueueSize = {
- def backends = sql """SHOW BACKENDS"""
- long totalSize = 0
- for (be in backends) {
- def ip = be[1]
- def brpcPort = be[5]
- def size = getIndexQueueSize(ip, brpcPort)
- totalSize += size
- logger.info("BE ${ip}:${brpcPort} index_queue_size = ${size}")
- }
- return totalSize
- }
+ def cacheSizeMetric = "file_cache_cache_size"
// Create test table with file cache enabled
def tableName = "test_load_failure_cache"
@@ -101,30 +68,37 @@ suite("test_clear_file_cache_on_load_failure", "nonConcurrent") {
// Disable auto analyze to avoid internal loads affecting cache size
sql """SET GLOBAL enable_auto_analyze = false"""
- // Clear file cache and wait for it to complete
- clearFileCacheOnAllBackends()
- sleep(3000)
+ requestFileCacheClearOnAllBackends()
- // Get initial cache size
- def initialCacheSize = getTotalIndexQueueSize()
+ // sync=true synchronously removes releasable blocks, but it does not guarantee that
+ // blocks still held by readers or writers have all been recycled. Use a stable metric
+ // snapshot as the baseline and verify cache-size increments instead of requiring zero.
+ def initialCacheSize = WarmupMetricsUtils.waitForBackendMetricSumStable(
+ getCacheBackends(),
+ cacheSizeMetric,
+ 10000,
+ 60000)
logger.info("Initial file cache size: ${initialCacheSize}")
// First, do a successful load to establish baseline
sql """INSERT INTO ${tableName} VALUES (1, 'test1', 100)"""
- // Wait for cache metrics to update
- sleep(5000)
- def afterSuccessfulLoadSize = getTotalIndexQueueSize()
+ def afterSuccessfulLoadSize = WarmupMetricsUtils.waitForBackendMetricSum(
+ getCacheBackends(),
+ cacheSizeMetric,
+ { value -> value > initialCacheSize },
+ 120000,
+ "Cache should increase after successful load. Initial: ${initialCacheSize}")
logger.info("Cache size after successful load: ${afterSuccessfulLoadSize}")
- assertTrue(afterSuccessfulLoadSize > initialCacheSize,
- "Cache should increase after successful load. Initial: ${initialCacheSize}, After: ${afterSuccessfulLoadSize}")
-
- // Clear cache again to reset
- clearFileCacheOnAllBackends()
- sleep(3000)
- def afterClearSize = getTotalIndexQueueSize()
- logger.info("Cache size after clear: ${afterClearSize}")
+ // The successful load may populate the cache asynchronously. Wait until it is stable,
+ // then use that value as the baseline for the failed-load cache-size increment.
+ afterSuccessfulLoadSize = WarmupMetricsUtils.waitForBackendMetricSumStable(
+ getCacheBackends(),
+ cacheSizeMetric,
+ 10000,
+ 60000)
+ logger.info("Stable cache size before failed load: ${afterSuccessfulLoadSize}")
// Enable debug point to make commit_rowset return error
GetDebugPoint().enableDebugPointForAllBEs("LoadChannel.add_batch.failed")
@@ -137,17 +111,22 @@ suite("test_clear_file_cache_on_load_failure", "nonConcurrent") {
}
// Wait for cleanup to complete and cache metrics to update
- sleep(5000)
+ def afterFailedLoadSize = WarmupMetricsUtils.waitForBackendMetricSumStable(
+ getCacheBackends(),
+ cacheSizeMetric,
+ 10000,
+ 60000)
// Get cache size after failed load
- def afterFailedLoadSize = getTotalIndexQueueSize()
logger.info("Cache size after failed load: ${afterFailedLoadSize}")
- // Verify cache size has not increased
- assertTrue(afterFailedLoadSize == afterClearSize,
+ // Pre-existing deleting blocks may be recycled during this interval, so a negative
+ // delta is valid. A failed load must not add cache bytes over the stable baseline.
+ def failedLoadCacheSizeDelta = afterFailedLoadSize - afterSuccessfulLoadSize
+ assertTrue(failedLoadCacheSizeDelta <= 0,
"Cache should not increase after failed load. " +
- "Before: ${afterClearSize}, After: ${afterFailedLoadSize}, " +
- "Difference: ${afterFailedLoadSize - afterClearSize}")
+ "Before: ${afterSuccessfulLoadSize}, After: ${afterFailedLoadSize}, " +
+ "Difference: ${failedLoadCacheSizeDelta}")
logger.info("Test passed: File cache was properly cleared after load failure")
} finally {
diff --git a/regression-test/suites/cloud_p0/cache/test_file_cache_info.groovy b/regression-test/suites/cloud_p0/cache/test_file_cache_info.groovy
index 7c8841f4544..26c4964bb72 100644
--- a/regression-test/suites/cloud_p0/cache/test_file_cache_info.groovy
+++ b/regression-test/suites/cloud_p0/cache/test_file_cache_info.groovy
@@ -15,120 +15,216 @@
// specific language governing permissions and limitations
// under the License.
-suite("test_file_cache_info") {
- def custoBeConfig = [
- enable_evict_file_cache_in_advance : false,
- file_cache_enter_disk_resource_limit_mode_percent : 99
- ]
-
- setBeConfigTemporary(custoBeConfig) {
+import org.apache.doris.regression.util.Http
- String[][] backends = sql """ show backends """
- def backendSockets = []
- def backendIdToBackendIP = [:]
- def backendIdToBackendHttpPort = [:]
- for (String[] backend in backends) {
- if (backend[9].equals("true")) {
- backendIdToBackendIP.put(backend[0], backend[1])
- backendIdToBackendHttpPort.put(backend[0], backend[4])
+suite("test_file_cache_info", "nonConcurrent") {
+ def customBeConfig = [
+ enable_evict_file_cache_in_advance: false,
+ file_cache_enter_disk_resource_limit_mode_percent: 99
+ ]
+ setBeConfigTemporary(customBeConfig) {
+ String tableName = "test_file_cache_info_lifecycle"
+ def clusters = sql "SHOW CLUSTERS"
+ assertTrue(!clusters.isEmpty(), "No compute group found")
+ String clusterName = clusters[0][0].toString()
+ String dbName = sql("SELECT DATABASE()")[0][0].toString()
+ def backends = sql_return_maparray("SHOW BACKENDS").findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collectEntries { be -> [(be.BackendId as Long): be] }
+ assertTrue(!backends.isEmpty(), "No alive backends in ${clusterName}")
+ def backendIds = backends.keySet()
+ Long tabletId = null
+ def lastState = [:]
+ def clearResponses = [:]
+ def originalQueryCache = sql("select @@enable_sql_cache, @@enable_query_cache")[0].collect { value ->
+ String setting = value.toString().toLowerCase(Locale.ROOT)
+ assertTrue(setting in ["true", "false", "0", "1"], "Unexpected cache setting: ${value}")
+ setting
}
- }
- assertTrue(backendIdToBackendIP.size() > 0, "No alive backends found")
-
- backendIdToBackendIP.each { backendId, ip ->
- def socket = ip + ":" + backendIdToBackendHttpPort.get(backendId)
- backendSockets.add(socket)
- }
-
- sql "drop table IF EXISTS customer"
-
- sql """
- CREATE TABLE IF NOT EXISTS customer (
- `c_custkey` int NULL,
- `c_name` string NULL,
- `c_address` string NULL,
- `c_city` string NULL,
- `c_nation` string NULL,
- `c_region` string NULL,
- `c_phone` string NULL,
- `c_mktsegment` string NULL
- )
- DUPLICATE KEY(`c_custkey`)
- DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
- PROPERTIES (
- "file_cache_ttl_seconds" = "3600"
- )
- """
-
- sql """
- insert into customer values
- (1, 'Customer#000000001', 'address1', 'city1', 'nation1', 'region1', 'phone1', 'segment1'),
- (2, 'Customer#000000002', 'address2', 'city2', 'nation2', 'region2', 'phone2', 'segment2'),
- (3, 'Customer#000000003', 'address3', 'city3', 'nation3', 'region3', 'phone3', 'segment3'),
- (4, 'Customer#000000004', 'address4', 'city4', 'nation4', 'region4', 'phone4', 'segment4'),
- (5, 'Customer#000000005', 'address5', 'city5', 'nation5', 'region5', 'phone5', 'segment5')
- """
- sql "sync"
-
- sql "select count(*) from customer"
-
- Thread.sleep(10000)
-
- def get_tablet_id = { String tbl_name ->
- def tablets = sql "show tablets from ${tbl_name}"
- assertEquals(tablets.size(), 1, "Should have exactly one tablet with BUCKETS=1")
- return tablets[0][0] as Long
- }
-
- def tablet_id = get_tablet_id("customer")
- println "Tablet ID: ${tablet_id}"
-
- def desc_cache_info = sql "desc information_schema.file_cache_info"
- assertTrue(desc_cache_info.size() > 0, "desc information_schema.file_cache_info should not be empty")
- assertEquals(desc_cache_info[0][0].toString().toUpperCase(), "HASH")
- assertEquals(desc_cache_info[1][0].toString().toUpperCase(), "OFFSET")
-
- def cache_info = sql "select * from information_schema.file_cache_info"
-
- assertTrue(cache_info.size() > 0, "file_cache_info should not be empty for tablet_id ${tablet_id}")
-
- println "First query - File cache info for tablet_id ${tablet_id}:"
- cache_info.each { row ->
- println " ${row}"
- }
-
- def clearResults = []
- backendSockets.each { socket ->
- httpTest {
- endpoint ""
- uri socket + "/api/file_cache?op=clear&sync=true"
- op "get"
- check {respCode, body ->
- assertEquals(respCode, 200, "clear local cache fail, maybe you can find something in respond: " + parseJson(body))
- clearResults.add(true)
+ def expectedRows = (1..5).collect { i ->
+ [i as Long, String.format('Customer#%09d', i), "address${i}".toString(),
+ "city${i}".toString(), "nation${i}".toString(), "region${i}".toString(),
+ "phone${i}".toString(), "segment${i}".toString()]
+ }
+ def readTable = {
+ sql "use @${clusterName}"
+ // Read and validate every column; result caches are disabled for both reads.
+ def rows = sql "select * from ${tableName} order by c_custkey"
+ assertTrue(rows.every { it.size() == 8 && it.every { value -> value != null } },
+ "Unexpected data result: ${rows}")
+ def normalized = rows.collect { row -> [row[0] as Long] + row.drop(1).collect { it.toString() } }
+ assertEquals(expectedRows, normalized)
+ }
+ def getCache = {
+ sql "use @${clusterName}"
+ def rows = sql """select be_id, cache_path, tablet_id, `hash`, `offset`, size, lower(type)
+ from information_schema.file_cache_info
+ where tablet_id = ${tabletId} and be_id in (${backendIds.join(',')})"""
+ def blocks = [:]
+ rows.each { row ->
+ def key = [row[0] as Long, row[1].toString(), row[2] as Long,
+ row[3].toString(), row[4] as Long]
+ assertTrue(backendIds.contains(key[0]) && key[2] == tabletId, "Unexpected cache owner: ${row}")
+ assertTrue(key[4] >= 0L && (row[5] as Long) > 0L, "Invalid cache range: ${row}")
+ assertTrue(!blocks.containsKey(key), "Duplicate cache block: ${key}")
+ blocks[key] = [size: row[5] as Long, type: row[6]?.toString()]
+ }
+ blocks
+ }
+ def summarizeCache = { Map blocks ->
+ blocks.groupBy { key, value -> [key[0], key[2]] }.collectEntries { owner, entries ->
+ [(owner): [block_count: entries.size(), bytes: entries.values().sum(0L) { it.size },
+ bytes_by_type: entries.values().groupBy { it.type }.collectEntries { type, values ->
+ [(type): values.sum(0L) { it.size }]
+ }]]
+ }
+ }
+ def waitForCache = { String phase, boolean expectPresent, long timeoutMs ->
+ long startedMs = System.currentTimeMillis()
+ long deadlineMs = startedMs + timeoutMs
+ long stableSince = 0L
+ long lastLogMs = 0L
+ def previous = null
+ while (System.currentTimeMillis() < deadlineMs) {
+ def blocks = getCache()
+ boolean matches = expectPresent ? !blocks.isEmpty() : blocks.isEmpty()
+ lastState = [phase: phase, elapsed_ms: System.currentTimeMillis() - startedMs,
+ cache_by_be_tablet: summarizeCache(blocks), blocks: blocks]
+ if (blocks != previous || System.currentTimeMillis() - lastLogMs >= 30000L) {
+ logger.info("file_cache_info lifecycle: cluster=${clusterName}, ${lastState}")
+ lastLogMs = System.currentTimeMillis()
+ }
+ if (!matches || blocks != previous) {
+ stableSince = System.currentTimeMillis()
+ }
+ previous = blocks
+ // A transient empty metadata snapshot is not a completed clear.
+ if (matches && System.currentTimeMillis() - stableSince >= 3000L) {
+ logger.info("file_cache_info phase complete: ${lastState}")
+ return blocks
+ }
+ sleep(1000)
+ }
+ assertTrue(false, "Timeout waiting for ${phase}: cluster=${clusterName}, tablet=${tabletId}, " +
+ "BEs=${backendIds}, clear_responses=${clearResponses}, last_state=${lastState}")
+ }
+ def waitForReaders = {
+ long deadlineMs = System.currentTimeMillis() + 60000L
+ long quietSince = System.currentTimeMillis()
+ while (System.currentTimeMillis() < deadlineMs) {
+ def active = sql_return_maparray("SHOW PROCESSLIST").findAll { row ->
+ row.Db?.toString() == dbName && row.Info?.toString()?.contains(tableName) &&
+ !"${row.Command}".equalsIgnoreCase("Sleep")
+ }
+ lastState = [phase: "reader_drain", active_queries: active]
+ if (!active.isEmpty()) {
+ quietSince = System.currentTimeMillis()
+ } else if (System.currentTimeMillis() - quietSince >= 3000L) {
+ return
+ }
+ sleep(1000)
+ }
+ assertTrue(false, "Timeout waiting for table queries to finish: ${lastState}")
+ }
+ def clearCache = { long beId ->
+ def be = backends[beId]
+ String url = "http://${be.Host}:${be.HttpPort}/api/file_cache?op=clear&sync=true"
+ long startedMs = System.currentTimeMillis()
+ clearResponses[beId] = [url: url]
+ logger.info("file_cache_info clear start: BE=${beId}, url=${url}")
+ def connection = Http.openConnection(url)
+ connection.setConnectTimeout(5000)
+ connection.setReadTimeout(60000)
+ try {
+ int status = connection.getResponseCode()
+ String body = (status == 200 ? connection.getInputStream() : connection.getErrorStream())
+ ?.withCloseable { it.getText("UTF-8") } ?: ""
+ clearResponses[beId] += [http_status: status, elapsed_ms: System.currentTimeMillis() - startedMs,
+ response: body]
+ logger.info("file_cache_info clear response: BE=${beId}, ${clearResponses[beId]}")
+ assertEquals(200, status, "Clear failed on BE ${beId}: ${body}")
+ def response = parseJson(body)
+ assertTrue("${response.status}".equalsIgnoreCase("OK"), "Clear failed on BE ${beId}: ${body}")
+ def counters = ["num_files_all", "num_cells_all", "num_cells_to_delete", "num_cells_wait_recycle"]
+ .collectEntries { name ->
+ def matcher = response.msg?.toString() =~ /\b${name}=(\d+)/
+ def values = []
+ while (matcher.find()) { values.add(matcher.group(1) as Long) }
+ [(name): values.isEmpty() ? null : values.sum(0L)]
+ }
+ clearResponses[beId].counters = counters
+ logger.info("file_cache_info clear counters: BE=${beId}, ${counters}")
+ } catch (Throwable t) {
+ clearResponses[beId].error = t.toString()
+ clearResponses[beId].elapsed_ms = System.currentTimeMillis() - startedMs
+ throw t
+ } finally {
+ connection.disconnect()
+ }
+ }
+ Throwable failure = null
+ try {
+ sql "use @${clusterName}"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ sql "drop table if exists ${tableName} force"
+ sql """CREATE TABLE ${tableName} (
+ c_custkey INT, c_name STRING, c_address STRING, c_city STRING,
+ c_nation STRING, c_region STRING, c_phone STRING, c_mktsegment STRING
+ ) DUPLICATE KEY(c_custkey)
+ DISTRIBUTED BY HASH(c_custkey) BUCKETS 1
+ PROPERTIES("file_cache_ttl_seconds"="3600", "disable_auto_compaction"="true")"""
+ def tablets = sql_return_maparray("show tablets from ${tableName}")
+ .collect { it.TabletId as Long }.unique()
+ assertEquals(1, tablets.size(), "Expected one tablet: ${tablets}")
+ tabletId = tablets[0]
+ String values = expectedRows.collect { row ->
+ "(${row[0]}, " + row.drop(1).collect { "'${it}'" }.join(", ") + ")"
+ }.join(", ")
+ sql "insert into ${tableName} values ${values}"
+ readTable()
+ def baseline = waitForCache("populated", true, 120000L)
+ waitForReaders()
+ // SQL quiescence does not prove that every BE holder has been released. The scoped
+ // empty-state check below remains the barrier for clear and async Meta Store deletion.
+ def cacheOwners = baseline.keySet().collect { it[0] }.unique().sort()
+ logger.info("file_cache_info before clear: cluster=${clusterName}, tablet=${tabletId}, " +
+ "cache_owners=${cacheOwners}, cache_by_be_tablet=${summarizeCache(baseline)}, blocks=${baseline}")
+ long clearStartedMs = System.currentTimeMillis()
+ cacheOwners.each { clearCache(it as Long) }
+ waitForCache("cleared", false, 600000L)
+ logger.info("file_cache_info clear verified: tablet=${tabletId}, " +
+ "clear_to_empty_ms=${System.currentTimeMillis() - clearStartedMs}, responses=${clearResponses}")
+
+ // Only now read the business table again. No repeated clear or forced release can
+ // bypass a stuck holder; persistent remnants must fail with the block details above.
+ long reloadStartedMs = System.currentTimeMillis()
+ readTable()
+ def reloaded = waitForCache("reloaded", true, 120000L)
+ // Readers may split a file into different blocks than the INSERT writer did.
+ logger.info("file_cache_info reload verified: tablet=${tabletId}, " +
+ "read_to_visible_ms=${System.currentTimeMillis() - reloadStartedMs}, " +
+ "cache_by_be_tablet=${summarizeCache(reloaded)}, blocks=${reloaded}")
+ } catch (Throwable t) {
+ failure = t
+ logger.warn("file_cache_info lifecycle failed: cluster=${clusterName}, tablet=${tabletId}, " +
+ "clear_responses=${clearResponses}, last_state=${lastState}, error=${t}")
+ throw t
+ } finally {
+ def cleanupErrors = []
+ try { sql "drop table if exists ${tableName} force" } catch (Throwable t) { cleanupErrors.add(t) }
+ [enable_sql_cache: originalQueryCache[0], enable_query_cache: originalQueryCache[1]].each { name, value ->
+ try { sql "set ${name} = ${value}" } catch (Throwable t) { cleanupErrors.add(t) }
+ }
+ if (!cleanupErrors.isEmpty()) {
+ if (failure != null) {
+ cleanupErrors.each { failure.addSuppressed(it) }
+ } else {
+ cleanupErrors.tail().each { cleanupErrors[0].addSuppressed(it) }
+ throw cleanupErrors[0]
+ }
}
}
- }
- assertEquals(clearResults.size(), backendSockets.size(), "Failed to clear cache on some backends")
-
- Thread.sleep(5000)
-
- def cache_info_after_clear = sql "select * from information_schema.file_cache_info where tablet_id = ${tablet_id}"
- assertEquals(cache_info_after_clear.size(), 0, "file_cache_info should be empty after clearing cache")
-
- println "After clearing cache - File cache info is empty as expected"
-
- sql "select * from customer"
-
- Thread.sleep(10000)
-
- def cache_info_reloaded = sql "select * from information_schema.file_cache_info where tablet_id = ${tablet_id}"
- assertTrue(cache_info_reloaded.size() > 0, "file_cache_info should not be empty after reloading data")
-
- println "After reloading data - File cache info for tablet_id ${tablet_id}:"
- cache_info_reloaded.each { row ->
- println " ${row}"
- }
-
}
}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_2.groovy b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_2.groovy
index cb804ae3b62..e652f2de6c1 100644
--- a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_2.groovy
+++ b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_2.groovy
@@ -15,9 +15,18 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.WarmupMetricsUtils
-suite("alter_ttl_2") {
+suite("alter_ttl_2", "nonConcurrent") {
+ def originalQueryCache = sql("select @@enable_sql_cache, @@enable_query_cache")[0]
+ .collect { it.toString().toLowerCase(Locale.ROOT) }
+ assertTrue(originalQueryCache.every { it in ["true", "false", "1", "0"] })
+ onFinish {
+ sql "set enable_sql_cache = ${originalQueryCache[0]}"
+ sql "set enable_query_cache = ${originalQueryCache[1]}"
+ }
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
def custoBeConfig = [
enable_evict_file_cache_in_advance : false,
file_cache_enter_disk_resource_limit_mode_percent : 99,
@@ -31,7 +40,9 @@ suite("alter_ttl_2") {
assertTrue(!clusters.isEmpty())
def validCluster = clusters[0][0]
sql """use @${validCluster};""";
- def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="300") """
+ long initialTtlSeconds = 3600L
+ long expiredTtlSeconds = 1L
+ def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="${initialTtlSeconds}") """
String[][] backends = sql """ show backends """
String backendId;
def backendIdToBackendIP = [:]
@@ -47,26 +58,11 @@ suite("alter_ttl_2") {
assertEquals(backendIdToBackendIP.size(), 1)
backendId = backendIdToBackendIP.keySet()[0]
- def url = backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendHttpPort.get(backendId) + """/api/file_cache?op=clear&sync=true"""
- logger.info(url)
- def clearFileCache = { check_func ->
- httpTest {
- endpoint ""
- uri url
- op "get"
- body ""
- check check_func
- }
- }
-
- def getMetricsMethod = { check_func ->
- httpTest {
- endpoint backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendBrpcPort.get(backendId)
- uri "/brpc_metrics"
- op "get"
- check check_func
- }
- }
+ def cacheBackend = [
+ ip: backendIdToBackendIP.get(backendId),
+ httpPort: backendIdToBackendHttpPort.get(backendId),
+ brpcPort: backendIdToBackendBrpcPort.get(backendId)
+ ]
def getTabletIds = { String tableName ->
def tablets = sql "show tablets from ${tableName}"
@@ -74,31 +70,57 @@ suite("alter_ttl_2") {
tablets.collect { it[0] as Long }
}
- def waitForFileCacheType = { List<Long> tabletIds, String expectedType, long timeoutMs = 180000L, long intervalMs = 2000L ->
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- boolean allMatch = true
- for (Long tabletId in tabletIds) {
- def rows = sql "select type from information_schema.file_cache_info where tablet_id = ${tabletId}"
- if (rows.isEmpty()) {
- logger.warn("file_cache_info is empty for tablet ${tabletId} while waiting for ${expectedType}")
- allMatch = false
- break
- }
- def mismatch = rows.find { row -> !row[0]?.toString()?.equalsIgnoreCase(expectedType) }
- if (mismatch) {
- logger.info("tablet ${tabletId} has cache types ${rows.collect { it[0] }} while waiting for ${expectedType}")
- allMatch = false
- break
- }
- }
- if (allMatch) {
- logger.info("All file cache entries for tablets ${tabletIds} are ${expectedType}")
- return
+ def getCache = { List<Long> tabletIds ->
+ sql "use @${validCluster}"
+ def rows = sql """select be_id, cache_path, tablet_id, `hash`, `offset`, size, lower(type)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id = ${backendId}"""
+ def blocks = [:]
+ rows.each { row ->
+ def key = [row[0] as Long, row[1].toString(), row[2] as Long,
+ row[3].toString(), row[4] as Long]
+ assertTrue(!blocks.containsKey(key), "Duplicate cache block: ${key}")
+ assertTrue((row[5] as Long) > 0L, "Invalid block size: ${row}")
+ blocks[key] = [size: row[5] as Long, type: row[6].toString()]
+ }
+ blocks
+ }
+ def sizes = { Map blocks -> blocks.collectEntries { key, value -> [(key): value.size] } }
+ def waitForFileCacheType = { List<Long> tabletIds, String expectedType, Map expectedBlocks = null ->
+ long deadline = System.currentTimeMillis() + 600000L
+ long stableSince = 0L
+ def previous = null
+ def lastState = [:]
+ while (System.currentTimeMillis() < deadline) {
+ def blocks = getCache(tabletIds)
+ def missing = tabletIds.findAll { id -> !blocks.keySet().any { it[2] == id } }
+ boolean ready = missing.isEmpty() && blocks.values().every { it.type == expectedType } &&
+ (expectedBlocks == null || sizes(blocks) == sizes(expectedBlocks))
+ lastState = [expected_type: expectedType, missing_tablets: missing,
+ by_type: blocks.values().groupBy { it.type }.collectEntries { type, values ->
+ [(type): [blocks: values.size(), bytes: values.sum(0L) { it.size }]]
+ }, missing_blocks: expectedBlocks == null ? [] : (expectedBlocks.keySet() - blocks.keySet()).take(10)]
+ if (blocks != previous) { logger.info("Scoped TTL transition: ${lastState}") }
+ if (!ready || blocks != previous) { stableSince = System.currentTimeMillis() }
+ previous = blocks
+ if (ready && System.currentTimeMillis() - stableSince >= 3000L) {
+ logger.info("Verified scoped cache: tablets=${tabletIds}, state=${lastState}")
+ return blocks
}
- sleep(intervalMs)
+ sleep(1000)
+ }
+ assertTrue(false, "Timeout waiting for scoped cache conversion on ${validCluster}/BE ${backendId}: ${lastState}")
+ }
+ def getGlobalMetrics = {
+ ["ttl_cache_size", "normal_queue_cache_size"].collectEntries { name ->
+ [(name): WarmupMetricsUtils.getBrpcMetric(cacheBackend.ip.toString(), cacheBackend.brpcPort.toString(), name)]
}
- assertTrue(false, "Timeout waiting for file_cache_info type ${expectedType} for tablets ${tabletIds}")
+ }
+ def scanTable = {
+ // Force a BE column scan and check fixture data before and after the cache transition.
+ def result = sql "select count(*), sum(C_CUSTKEY) from customer_ttl"
+ assertEquals(200L, result[0][0] as Long)
+ assertEquals(2020100L, result[0][1] as Long)
}
sql new File("""${context.file.parent}/../ddl/customer_ttl_delete.sql""").text
@@ -130,101 +152,30 @@ suite("alter_ttl_2") {
}
}
- clearFileCache.call() {
- respCode, body -> {}
- }
- sleep(30000)
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- logger.info("ttl_cache_size line before assert zero: " + line)
- assertEquals(line.substring(i).toLong(), 0)
- flag1 = true
- }
- }
- assertTrue(flag1)
- }
+ try {
+ def metricsBeforeClear = getGlobalMetrics()
+ WarmupMetricsUtils.clearFileCache(cacheBackend.ip.toString(), cacheBackend.httpPort.toString())
+ def metricsAfterClear = getGlobalMetrics()
+ // sync=true may leave held blocks from other tables. Global counters are diagnostic only.
+ logger.info("TTL clear metrics: before=${metricsBeforeClear}, after=${metricsAfterClear}")
- load_customer_ttl_once("customer_ttl")
- def tabletIds = getTabletIds.call("customer_ttl")
- waitForFileCacheType.call(tabletIds, "ttl", 60000L)
- sql """ select count(*) from customer_ttl """
- sleep(30000)
- long ttl_cache_size = 0
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- logger.info("ttl_cache_size line after load: " + line)
- ttl_cache_size = line.substring(i).toLong()
- flag1 = true
- }
- }
- assertTrue(flag1)
- }
- sql """ ALTER TABLE customer_ttl SET ("file_cache_ttl_seconds"="120") """
- sleep(80000)
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- logger.info("ttl_cache_size line after ttl update: " + line)
- assertEquals(line.substring(i).toLong(), ttl_cache_size)
- flag1 = true
- }
- }
- assertTrue(flag1)
- }
- // the first load data ttl is 300，so need wait for 200s until the ttl timeout
- sleep(200000)
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- assertEquals(line.substring(i).toLong(), 0)
- flag1 = true
- }
- }
- assertTrue(flag1)
- }
+ load_customer_ttl_once("customer_ttl")
+ def tabletIds = getTabletIds.call("customer_ttl")
+ scanTable()
+ def baseline = waitForFileCacheType(tabletIds, "ttl")
+ def metricsBeforeAlter = getGlobalMetrics()
+ logger.info("TTL before shortening: tablets=${tabletIds}, blocks=${baseline}, metrics=${metricsBeforeAlter}")
- waitForFileCacheType.call(tabletIds, "normal", 300000L)
+ // The stable TTL baseline above is already more than one second after tablet creation.
+ // Retain the positive -> positive transition; the original blocks must change type in place.
+ sql """ALTER TABLE customer_ttl SET ("file_cache_ttl_seconds"="${expiredTtlSeconds}")"""
+ waitForFileCacheType(tabletIds, "normal", baseline)
+ def metricsAfterAlter = getGlobalMetrics()
+ def delta = metricsAfterAlter.collectEntries { name, value -> [(name): value - metricsBeforeAlter[name]] }
+ logger.info("TTL after shortening: before=${metricsBeforeAlter}, after=${metricsAfterAlter}, delta=${delta}")
+ scanTable()
+ } finally {
+ sql "DROP TABLE IF EXISTS customer_ttl FORCE"
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_4.groovy b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_4.groovy
index 691994c36c5..c78a140ba47 100644
--- a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_4.groovy
+++ b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_4.groovy
@@ -15,205 +15,324 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
-
-suite("alter_ttl_4") {
- def custoBeConfig = [
- enable_evict_file_cache_in_advance : false,
- file_cache_enter_disk_resource_limit_mode_percent : 99
+suite("alter_ttl_4", "nonConcurrent") {
+ def customBeConfig = [
+ enable_evict_file_cache_in_advance: false,
+ file_cache_enter_disk_resource_limit_mode_percent: 99,
+ file_cache_background_ttl_gc_interval_ms: 1000,
+ file_cache_background_ttl_info_update_interval_ms: 1000,
+ file_cache_background_tablet_id_flush_interval_ms: 1000
]
-
- setBeConfigTemporary(custoBeConfig) {
- def clusters = sql " SHOW CLUSTERS; "
- assertTrue(!clusters.isEmpty())
- def validCluster = clusters[0][0]
- sql """use @${validCluster};""";
- def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="900") """
- String[][] backends = sql """ show backends """
- String backendId;
- def backendIdToBackendIP = [:]
- def backendIdToBackendHttpPort = [:]
- def backendIdToBackendBrpcPort = [:]
- for (String[] backend in backends) {
- if (backend[9].equals("true") && backend[19].contains("${validCluster}")) {
- backendIdToBackendIP.put(backend[0], backend[1])
- backendIdToBackendHttpPort.put(backend[0], backend[4])
- backendIdToBackendBrpcPort.put(backend[0], backend[5])
+ setBeConfigTemporary(customBeConfig) {
+ String targetTable = "alter_ttl_4_target"
+ String pressureTable = "alter_ttl_4_pressure"
+ long initialTtl = 3600L
+ long rowsPerLoad = 15000000L // TPCH SF100 customer.tbl
+ long maxPressureRows = 64L * rowsPerLoad
+ def clusters = sql "SHOW CLUSTERS"
+ assertTrue(!clusters.isEmpty())
+ String clusterName = clusters[0][0].toString()
+ def backends = sql_return_maparray("SHOW BACKENDS").findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collectEntries { be -> [(be.BackendId as Long): be] }
+ assertTrue(!backends.isEmpty(), "No alive backends in ${clusterName}")
+ def backendIds = backends.keySet()
+ def tabletIds = []
+ def pendingLoads = [] as Set
+ def lastState = [:]
+ def cacheDifference = [:]
+ def originalQueryCache = sql("select @@enable_sql_cache, @@enable_query_cache")[0].collect { value ->
+ String setting = value.toString().toLowerCase(Locale.ROOT)
+ assertTrue(setting in ["true", "false", "0", "1"], "Unexpected cache setting: ${value}")
+ setting
}
- }
- assertEquals(backendIdToBackendIP.size(), 1)
-
- backendId = backendIdToBackendIP.keySet()[0]
- def url = backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendHttpPort.get(backendId) + """/api/file_cache?op=clear&sync=true"""
- logger.info(url)
- def clearFileCache = { check_func ->
- httpTest {
- endpoint ""
- uri url
- op "get"
- body ""
- check check_func
+ def waitUntil = { String description, long timeoutMs, Closure ready ->
+ long deadline = System.currentTimeMillis() + timeoutMs
+ while (System.currentTimeMillis() < deadline) {
+ if (ready()) {
+ return
+ }
+ sleep(1000)
+ }
+ assertTrue(false, "Timeout after ${timeoutMs}ms: ${description}; last_state=${lastState}")
}
- }
-
- def getMetricsMethod = { check_func ->
- httpTest {
- endpoint backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendBrpcPort.get(backendId)
- uri "/brpc_metrics"
- op "get"
- check check_func
+ def scanTarget = {
+ sql "use @${clusterName}"
+ // Read every data column, including strings; neither COUNT metadata nor LIMIT can satisfy this scan.
+ def rows = sql """select count(*), sum(C_CUSTKEY), sum(length(C_NAME)), sum(length(C_ADDRESS)),
+ sum(C_NATIONKEY), sum(length(C_PHONE)), sum(C_ACCTBAL),
+ sum(length(C_MKTSEGMENT)), sum(length(C_COMMENT)) from ${targetTable}"""
+ assertTrue(rows.size() == 1 && rows[0].size() == 9 && rows[0].every { it != null },
+ "Unexpected data result: ${rows}")
+ assertEquals(rowsPerLoad, rows[0][0] as Long)
+ rows[0].collect { new BigDecimal(it.toString()).stripTrailingZeros() }
}
- }
-
- def s3BucketName = getS3BucketName()
- def s3WithProperties = """WITH S3 (
- |"AWS_ACCESS_KEY" = "${getS3AK()}",
- |"AWS_SECRET_KEY" = "${getS3SK()}",
- |"AWS_ENDPOINT" = "${getS3Endpoint()}",
- |"AWS_REGION" = "${getS3Region()}",
- |"provider" = "${getS3Provider()}")
- |PROPERTIES(
- |"exec_mem_limit" = "8589934592",
- |"load_parallelism" = "3")""".stripMargin()
-
-
- sql new File("""${context.file.parent}/../ddl/customer_ttl_delete.sql""").text
- sql new File("""${context.file.parent}/../ddl/customer_delete.sql""").text
- def load_customer_ttl_once = { String table ->
- def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
- // def table = "customer"
- // create table if not exists
- sql (new File("""${context.file.parent}/../ddl/${table}.sql""").text + ttlProperties)
- sql """ alter table ${table} set ("disable_auto_compaction" = "true") """ // no influence from compaction
- def loadLabel = table + "_" + uniqueID
- // load data from cos
- def loadSql = new File("""${context.file.parent}/../ddl/${table}_load.sql""").text.replaceAll("\\\$\\{s3BucketName\\}", s3BucketName)
- loadSql = loadSql.replaceAll("\\\$\\{loadLabel\\}", loadLabel) + s3WithProperties
- sql loadSql
-
- // check load state
- while (true) {
- def stateResult = sql "show load where Label = '${loadLabel}'"
- def loadState = stateResult[stateResult.size() - 1][2].toString()
- if ("CANCELLED".equalsIgnoreCase(loadState)) {
- throw new IllegalStateException("load ${loadLabel} failed.")
- } else if ("FINISHED".equalsIgnoreCase(loadState)) {
- break
- }
- sleep(5000)
+ def getCache = {
+ sql "use @${clusterName}"
+ def rows = sql """select be_id, cache_path, tablet_id, `hash`, `offset`, size, lower(type)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${backendIds.join(',')})"""
+ def blocks = [:]
+ rows.each { row ->
+ def key = [row[0] as Long, row[1].toString(), row[2] as Long,
+ row[3].toString(), row[4] as Long]
+ assertTrue(backendIds.contains(key[0]) && tabletIds.contains(key[2]), "Unexpected cache owner: ${row}")
+ assertTrue(key[4] >= 0L && (row[5] as Long) > 0L, "Invalid cache range: ${row}")
+ assertTrue(!blocks.containsKey(key), "Duplicate cache block: ${key}")
+ blocks[key] = [size: row[5] as Long, type: row[6]?.toString()]
+ }
+ blocks
}
- }
-
- def load_customer_once = { String table ->
- def uniqueID = Math.abs(UUID.randomUUID().hashCode()).toString()
- // def table = "customer"
- // create table if not exists
- sql new File("""${context.file.parent}/../ddl/${table}.sql""").text
- def loadLabel = table + "_" + uniqueID
- // load data from cos
- def loadSql = new File("""${context.file.parent}/../ddl/${table}_load.sql""").text.replaceAll("\\\$\\{s3BucketName\\}", s3BucketName)
- loadSql = loadSql.replaceAll("\\\$\\{loadLabel\\}", loadLabel) + s3WithProperties
- sql loadSql
-
- // check load state
- while (true) {
- def stateResult = sql "show load where Label = '${loadLabel}'"
- def loadState = stateResult[stateResult.size() - 1][2].toString()
- if ("CANCELLED".equalsIgnoreCase(loadState)) {
- throw new IllegalStateException("load ${loadLabel} failed.")
- } else if ("FINISHED".equalsIgnoreCase(loadState)) {
- break
- }
- sleep(5000)
+ def blockSizes = { Map blocks -> blocks.collectEntries { key, value -> [(key): value.size] } }
+ def sameBlocks = { Map expected, Map actual ->
+ cacheDifference = [missing: (expected.keySet() - actual.keySet()).take(10),
+ unexpected: (actual.keySet() - expected.keySet()).take(10),
+ changed_sizes: actual.findAll { key, value -> value.size != expected[key]?.size }.take(10)]
+ blockSizes(expected) == blockSizes(actual)
}
- }
-
- clearFileCache.call() {
- respCode, body -> {}
- }
- sleep(30000)
+ def cacheSummary = { Map blocks ->
+ blocks.groupBy { key, value -> [key[0], key[2], value.type] }
+ .collectEntries { key, entries -> [(key): entries.values().sum(0L) { it.size }] }
+ }
+ def coverage = { Map blocks ->
+ blocks.groupBy { key, value -> key[0..3] }.collectEntries { file, entries ->
+ def ranges = []
+ entries.collect { key, value -> [key[4], key[4] + value.size] }.sort { a, b -> a[0] <=> b[0] }
+ .each { range ->
+ assertTrue(ranges.isEmpty() || range[0] >= ranges.last()[1],
+ "Overlapping cache ranges: file=${file}, ranges=${ranges}, next=${range}")
+ if (!ranges.isEmpty() && range[0] == ranges.last()[1]) {
+ ranges.last()[1] = range[1]
+ } else {
+ ranges.add(range)
+ }
+ }
+ [(file): ranges]
+ }
+ }
+ def missingCoverage = { Map expected, Map actual ->
+ expected.findAll { file, ranges ->
+ !ranges.every { range ->
+ actual[file]?.any { present -> present[0] <= range[0] && present[1] >= range[1] }
+ }
+ }
+ }
+ def waitForCache = { String stage, String type, Closure matches ->
+ def previous = null
+ def result = [:]
+ long stableSince = 0L
+ waitUntil(stage, 600000L) {
+ result = getCache()
+ def missingTablets = tabletIds.findAll { id -> !result.keySet().any { it[2] == id } }
+ boolean matching = matches(result)
+ boolean complete = missingTablets.isEmpty() &&
+ result.values().every { it.type == type } && matching
+ lastState = [stage: stage, cache_by_be_tablet_type: cacheSummary(result),
+ missing_tablets: missingTablets, differences: cacheDifference]
+ if (result != previous) {
+ logger.info("${lastState}")
+ }
+ if (!complete || result != previous) {
+ stableSince = System.currentTimeMillis()
+ }
+ previous = result
+ complete && System.currentTimeMillis() - stableSince >= 3000L
+ }
+ result
+ }
+ def metricNames = ["file_cache_ttl_cache_size", "file_cache_capacity", "file_cache_evict_by_self_lru_normal"]
+ def getMetrics = {
+ // Per-BE diagnostic snapshots; residual cache from other tables is not a failure.
+ backends.collectEntries { id, be ->
+ def values = metricNames.collectEntries { [(it): null] }
+ try {
+ String text = new URL("http://${be.Host}:${be.BrpcPort}/brpc_metrics")
+ .getText(connectTimeout: 5000, readTimeout: 5000)
+ text.readLines().each { line ->
+ def fields = line.trim().split(/\s+/)
+ if (fields.size() == 2 && !fields[0].startsWith("#")) {
+ metricNames.each { name ->
+ if (fields[0] == name || fields[0].endsWith("_${name}")) {
+ values[name] = (values[name] ?: 0L) + (fields[1] as Long)
+ }
+ }
+ }
+ }
+ } catch (Exception e) {
+ logger.warn("Cannot read diagnostic metrics for BE ${id}: ${e.message}")
+ }
+ [(id): values]
+ }
+ }
+ def logMetricDelta = { String stage, Map before, Map after ->
+ def delta = backendIds.collectEntries { id ->
+ [(id): metricNames.collectEntries { name ->
+ [(name): before[id][name] != null && after[id][name] != null ?
+ after[id][name] - before[id][name] : null]
+ }]
+ }
+ logger.info("${stage} metrics: before=${before}, after=${after}, delta_by_be=${delta}")
+ }
+ def loadTable = { String table, long timeoutMs ->
+ long deadline = System.currentTimeMillis() + timeoutMs
+ String label = "${table}_${UUID.randomUUID().toString().replace('-', '')}"
+ pendingLoads.add(label)
+ String statement = new File("${context.file.parent}/../ddl/customer_ttl_load.sql").text
+ .replace('${loadLabel}', label).replace('${s3BucketName}', getS3BucketName())
+ .replace('INTO TABLE customer_ttl', "INTO TABLE ${table}")
+ sql(statement + """WITH S3 (
+ "AWS_ACCESS_KEY"="${getS3AK()}", "AWS_SECRET_KEY"="${getS3SK()}",
+ "AWS_ENDPOINT"="${getS3Endpoint()}", "AWS_REGION"="${getS3Region()}",
+ "provider"="${getS3Provider()}")
+ PROPERTIES("exec_mem_limit"="8589934592", "load_parallelism"="3")""")
+ waitUntil("load ${label}", Math.max(0L, deadline - System.currentTimeMillis())) {
+ def rows = sql "show load where Label = '${label}'"
+ lastState = [label: label, state: rows.isEmpty() ? "NOT_VISIBLE" : rows.last()[2]]
+ if (rows.isEmpty()) {
+ return false
+ }
+ String state = rows.last()[2].toString()
+ assertTrue(!state.equalsIgnoreCase("CANCELLED"), "Load cancelled: ${label}, details=${rows.last()}")
+ if (state.equalsIgnoreCase("FINISHED")) {
+ pendingLoads.remove(label)
+ return true
+ }
+ false
+ }
+ }
+ Throwable failure = null
+ try {
+ sql "use @${clusterName}"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ [(targetTable): initialTtl, (pressureTable): 0L].each { table, ttl ->
+ sql "drop table if exists ${table} force"
+ String ddl = new File("${context.file.parent}/../ddl/customer_ttl.sql").text
+ .replace('customer_ttl', table)
+ sql(ddl + """ PROPERTIES("file_cache_ttl_seconds"="${ttl}", "disable_auto_compaction"="true")""")
+ }
+ tabletIds = sql_return_maparray("show tablets from ${targetTable}")
+ .collect { it.TabletId as Long }.unique().sort()
+ assertTrue(tabletIds.size() == 32, "Expected 32 target tablets: ${tabletIds}")
+ loadTable(targetTable, 300000L)
+ def dataBaseline = scanTarget()
- load_customer_ttl_once("customer_ttl")
- sql """ select count(*) from customer_ttl """
- sleep(30000)
- long custom_ttl_size = 0
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag2 = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ def creationTimes = [:]
+ // The full scan above verifies data; cloud header JSON does not include rowsets.
+ waitUntil("target tablet TTL metadata", 600000L) {
+ creationTimes = [:]
+ sql_return_maparray("show tablets from ${targetTable}").each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (!(id in tabletIds) || !backends.containsKey(primary) || (tablet.BackendId as Long) != primary) {
+ return
+ }
+ def be = backends[primary]
+ try {
+ httpTest {
+ endpoint "${be.Host}:${be.HttpPort}"
+ uri "/api/meta/header/${id}"
+ op "get"
+ check { code, body ->
+ if ((code as int) == 200) {
+ def meta = parseJson(body.toString())
+ if ((meta.tablet_id as Long) == id && (meta.creation_time as Long) > 0L &&
+ (meta.ttl_seconds as Long) == initialTtl &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null) {
+ creationTimes[id] = meta.creation_time as Long
+ }
+ }
+ }
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for metadata: tablet=${id}, BE=${primary}, ${e.message}")
}
- def i = line.indexOf(' ')
- custom_ttl_size = line.substring(i).toLong()
- flag2 = true
}
+ lastState = [missing_tablets: tabletIds - creationTimes.keySet()]
+ creationTimes.size() == tabletIds.size()
}
- assertTrue(flag2)
- }
- sleep(60000)
- // one customer table would take about 1.3GB, the total cache size is 20GB
- // the following would take 20G all
- // evict customer_ttl
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
- load_customer_once("customer")
+ def ttlBaseline = waitForCache("initial TTL cache", "ttl") { true }
+ def baselineCoverage = coverage(ttlBaseline)
+ waitUntil("tablet creation time plus one second to have passed", 60000L) {
+ long now = sql("select unix_timestamp()")[0][0] as Long
+ assertTrue(now < creationTimes.values().min() + initialTtl, "Initial TTL already expired")
+ now > creationTimes.values().max() + 1L
+ }
+ logger.info("TTL baseline: creation_times=${creationTimes}, metrics=${getMetrics()}")
- // some datas in s3 and will download them
- sql """ select C_CUSTKEY from customer_ttl order by C_CUSTKEY limit 1"""
- sql """ select C_NAME from customer_ttl order by C_NAME limit 1"""
- sql """ select C_ADDRESS from customer_ttl order by C_ADDRESS limit 1"""
- sql """ select C_NATIONKEY from customer_ttl order by C_NATIONKEY limit 1"""
- sql """ select C_PHONE from customer_ttl order by C_PHONE limit 1 """
- sql """ select C_ACCTBAL from customer_ttl order by C_ACCTBAL limit 1"""
- sql """ select C_MKTSEGMENT from customer_ttl order by C_MKTSEGMENT limit 1"""
- sql """ select C_COMMENT from customer_ttl order by C_COMMENT limit 1"""
+ sql "alter table ${targetTable} set (\"file_cache_ttl_seconds\"=\"1\")"
+ waitForCache("expired TTL blocks to become NORMAL", "normal") { blocks ->
+ sameBlocks(ttlBaseline, blocks)
+ }
- sql """ ALTER TABLE customer_ttl SET ("file_cache_ttl_seconds"="3600") """
- // wait for fetching new tablet meta in BE
- sleep(60000)
- // wait for updating file cache metrics
- sleep(30000)
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ long pressureDeadline = System.currentTimeMillis() + 1200000L
+ long pressureRows = 0L
+ def metricsBeforePressure = getMetrics()
+ def evictedFiles = [:]
+ while (evictedFiles.isEmpty() && pressureRows < maxPressureRows &&
+ System.currentTimeMillis() < pressureDeadline) {
+ loadTable(pressureTable, Math.min(300000L, pressureDeadline - System.currentTimeMillis()))
+ pressureRows += rowsPerLoad
+ // Poll cache metadata between loads without touching target data or clearing caches.
+ long checkDeadline = Math.min(pressureDeadline, System.currentTimeMillis() + 5000L)
+ while (evictedFiles.isEmpty() && System.currentTimeMillis() < checkDeadline) {
+ def current = getCache()
+ assertTrue(current.values().every { it.type == "normal" },
+ "Expired target cache changed type during pressure: ${cacheSummary(current)}")
+ evictedFiles = missingCoverage(baselineCoverage, coverage(current))
+ if (evictedFiles.isEmpty()) {
+ sleep(1000)
}
- def i = line.indexOf(' ')
- assertEquals(custom_ttl_size, line.substring(i).toLong())
- flag1 = true
}
+ logger.info("Cache pressure: imported_rows=${pressureRows}, missing_files=${evictedFiles.keySet()}")
}
- assertTrue(flag1)
- }
+ def metricsAfterPressure = getMetrics()
+ logMetricDelta("Pressure", metricsBeforePressure, metricsAfterPressure)
+ assertTrue(!evictedFiles.isEmpty(), "No target cache eviction before pressure limit: " +
+ "imported_rows=${pressureRows}/${maxPressureRows}, metrics=${metricsAfterPressure}")
+
+ assertEquals(dataBaseline, scanTarget())
+ // Re-reading an evicted file may split blocks differently. Compare merged byte ranges.
+ def reloaded = waitForCache("reloaded NORMAL cache coverage", "normal") { blocks ->
+ def current = coverage(blocks)
+ cacheDifference = [missing: missingCoverage(baselineCoverage, current).take(10),
+ extra: missingCoverage(current, baselineCoverage).take(10)]
+ current == baselineCoverage
+ }
+ def metricsBeforeExtension = getMetrics()
+ long now = sql("select unix_timestamp()")[0][0] as Long
+ long extendedTtl = Math.max(initialTtl, now - (creationTimes.values().min() as Long) + 1200L)
+ sql "alter table ${targetTable} set (\"file_cache_ttl_seconds\"=\"${extendedTtl}\")"
+ // Keep positive -> positive: routing through TTL=0 would hide expired-TTL renewal bugs.
+ // No target reads after ALTER until the background conversion has been verified.
+ waitForCache("renewed TTL on existing NORMAL blocks (1 -> ${extendedTtl})", "ttl") { blocks ->
+ sameBlocks(reloaded, blocks)
+ }
+ logMetricDelta("Extended TTL=${extendedTtl}", metricsBeforeExtension, getMetrics())
+ assertEquals(dataBaseline, scanTarget())
+ } catch (Throwable t) {
+ failure = t
+ throw t
+ } finally {
+ def cleanupErrors = []
+ pendingLoads.each { label ->
+ try { sql "cancel load where label = '${label}'" } catch (Throwable t) { cleanupErrors.add(t) }
+ }
+ [targetTable, pressureTable].each { table ->
+ try { sql "drop table if exists ${table} force" } catch (Throwable t) { cleanupErrors.add(t) }
+ }
+ [enable_sql_cache: originalQueryCache[0], enable_query_cache: originalQueryCache[1]].each { name, value ->
+ try { sql "set ${name} = ${value}" } catch (Throwable t) { cleanupErrors.add(t) }
+ }
+ if (!cleanupErrors.isEmpty()) {
+ if (failure != null) {
+ cleanupErrors.each { failure.addSuppressed(it) }
+ } else {
+ cleanupErrors.tail().each { cleanupErrors[0].addSuppressed(it) }
+ throw cleanupErrors[0]
+ }
+ }
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_random.groovy b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_random.groovy
index 3aa0a3a2c58..c875757807c 100644
--- a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_random.groovy
+++ b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_random.groovy
@@ -91,20 +91,27 @@ suite("test_ttl_random") {
load_customer_once("customer_ttl")
- Random random = new Random()
+ long maxFileCacheTtlSeconds = Long.MAX_VALUE / 2L
+ long randomSeed = 20260902L
+ Random random = new Random(randomSeed)
+ long lastAlterTtlSeconds = 180L
+ logger.info("Use deterministic random seed ${randomSeed} for TTL ALTER values")
for (int j = 0; j < 40; j++) {
- long number = random.nextLong() % 10l;
+ int number = j % 10
if (number < 5) {
sql """ select * from customer_ttl limit 10"""
} else if (number >= 5 && number < 8) {
load_customer_once("customer_ttl")
} else {
- def alterTtlSeconds = random.nextLong()
- if (alterTtlSeconds < 0) {
- alterTtlSeconds *= -1
- }
- sql """ ALTER TABLE customer_ttl SET ("file_cache_ttl_seconds"="${alterTtlSeconds}") """
+ lastAlterTtlSeconds = Math.floorMod(
+ random.nextLong(), maxFileCacheTtlSeconds + 1L)
+ sql """ ALTER TABLE customer_ttl SET ("file_cache_ttl_seconds"="${lastAlterTtlSeconds}") """
sleep(40000)
}
}
+
+ def showCreateTable = sql "show create table customer_ttl"
+ assertTrue(showCreateTable[0][1].toString().contains(
+ "\"file_cache_ttl_seconds\" = \"${lastAlterTtlSeconds}\""),
+ "file_cache_ttl_seconds should converge to the last ALTER value ${lastAlterTtlSeconds}")
}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_seconds.groovy b/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_seconds.groovy
deleted file mode 100644
index 491576e5601..00000000000
--- a/regression-test/suites/cloud_p0/cache/ttl/alter_ttl_seconds.groovy
+++ /dev/null
@@ -1,170 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-import org.codehaus.groovy.runtime.IOGroovyMethods
-
-suite("test_ttl_seconds") {
- def custoBeConfig = [
- enable_evict_file_cache_in_advance : false,
- file_cache_enter_disk_resource_limit_mode_percent : 99,
- file_cache_background_ttl_gc_interval_ms : 1000,
- file_cache_background_ttl_info_update_interval_ms : 1000,
- file_cache_background_tablet_id_flush_interval_ms : 1000
- ]
-
- setBeConfigTemporary(custoBeConfig) {
- sql "set global enable_auto_analyze = false"
- sql "set global enable_audit_plugin = false"
- def clusters = sql " SHOW CLUSTERS; "
- assertTrue(!clusters.isEmpty())
- def validCluster = clusters[0][0]
- sql """use @${validCluster};""";
- def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="5") """
- String[][] backends = sql """ show backends """
- String backendId;
- def backendIdToBackendIP = [:]
- def backendIdToBackendHttpPort = [:]
- def backendIdToBackendBrpcPort = [:]
- for (String[] backend in backends) {
- if (backend[9].equals("true") && backend[19].contains("regression_cluster_name1")) {
- backendIdToBackendIP.put(backend[0], backend[1])
- backendIdToBackendHttpPort.put(backend[0], backend[4])
- backendIdToBackendBrpcPort.put(backend[0], backend[5])
- }
- }
- assertEquals(backendIdToBackendIP.size(), 1)
-
- backendId = backendIdToBackendIP.keySet()[0]
- def url = backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendHttpPort.get(backendId) + """/api/file_cache?op=clear&sync=true"""
- logger.info(url)
- def clearFileCache = { check_func ->
- httpTest {
- endpoint ""
- uri url
- op "get"
- body ""
- check check_func
- }
- }
-
- sql new File("""${context.file.parent}/../ddl/customer_ttl_delete.sql""").text
- def load_customer_once = { String table ->
- sql (new File("""${context.file.parent}/../ddl/${table}.sql""").text + ttlProperties)
- sql """ alter table ${table} set ("disable_auto_compaction" = "true") """ // no influence from compaction
- def totalRows = 200
- def batchSize = 100
- def commentSuffix = ' ' + ('X' * 50)
- for (int offset = 0; offset < totalRows; offset += batchSize) {
- def sb = new StringBuilder()
- int batchEnd = Math.min(totalRows, offset + batchSize)
- for (int idx = offset; idx < batchEnd; idx++) {
- def customerId = 10001 + idx
- def customerName = String.format('Customer#%09d', customerId)
- sb.append("""INSERT INTO ${table} VALUES (
- ${customerId},
- '${customerName}',
- 'Address Line 1',
- 15,
- '123-456-7890',
- 12345.67,
- 'AUTOMOBILE',
- 'This is a test comment for the customer.${commentSuffix}'
- );
- """)
- }
- sql sb.toString()
- }
- }
-
- def getMetricsMethod = { check_func ->
- httpTest {
- endpoint backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendBrpcPort.get(backendId)
- uri "/brpc_metrics"
- op "get"
- check check_func
- }
- }
-
- def getTabletIds = { String tableName ->
- def tablets = sql "show tablets from ${tableName}"
- assertTrue(tablets.size() > 0, "No tablets found for table ${tableName}")
- tablets.collect { it[0] as Long }
- }
-
- def waitForFileCacheType = { List<Long> tabletIds, String expectedType, long timeoutMs = 60000L, long intervalMs = 1000L ->
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- boolean allMatch = true
- for (Long tabletId in tabletIds) {
- def rows = sql "select type from information_schema.file_cache_info where tablet_id = ${tabletId}"
- if (rows.isEmpty()) {
- logger.warn("file_cache_info is empty for tablet ${tabletId} while waiting for ${expectedType}")
- allMatch = false
- break
- }
- def mismatch = rows.find { row -> !row[0]?.toString()?.equalsIgnoreCase(expectedType) }
- if (mismatch) {
- logger.info("tablet ${tabletId} has cache types ${rows.collect { it[0] }} while waiting for ${expectedType}")
- allMatch = false
- break
- }
- }
- if (allMatch) {
- logger.info("All file cache entries for tablets ${tabletIds} are ${expectedType}")
- return
- }
- sleep(intervalMs)
- }
- assertTrue(false, "Timeout waiting for file_cache_info type ${expectedType} for tablets ${tabletIds}")
- }
-
- clearFileCache.call() {
- respCode, body -> {}
- }
- sleep(30000)
-
- load_customer_once("customer_ttl")
- def tabletIds = getTabletIds.call("customer_ttl")
- // No wait for the "ttl" type here. The TTL deadline is the tablet creation time plus
- // file_cache_ttl_seconds, and with a 5s ttl the load itself outlives it, so most of the
- // data is written straight into the normal queue and the table never has all of its
- // blocks in the TTL queue at once.
- sleep(30000) // 30s
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- logger.info("ttl_cache_size after load: " + line)
- assertEquals(line.substring(i).toLong(), 0)
- flag1 = true
- }
- }
- assertTrue(flag1)
- }
-
- waitForFileCacheType.call(tabletIds, "normal", 60000L)
- }
-}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.groovy b/regression-test/suites/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.groovy
index ffc864e0e53..824cb8d7dd0 100644
--- a/regression-test/suites/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.groovy
+++ b/regression-test/suites/cloud_p0/cache/ttl/st04_alter_ttl_n_to_0_runtime.groovy
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-suite("st04_alter_ttl_n_to_0_runtime") {
+suite("st04_alter_ttl_n_to_0_runtime", "nonConcurrent") {
def customBeConfig = [
enable_evict_file_cache_in_advance : false,
file_cache_enter_disk_resource_limit_mode_percent : 99,
@@ -27,119 +27,160 @@ suite("st04_alter_ttl_n_to_0_runtime") {
setBeConfigTemporary(customBeConfig) {
def clusters = sql "SHOW CLUSTERS"
assertTrue(!clusters.isEmpty())
- def validCluster = clusters[0][0]
- sql """use @${validCluster};"""
-
+ String clusterName = clusters[0][0].toString()
+ def backends = sql_return_maparray("SHOW BACKENDS").findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collectEntries { be -> [(be.BackendId as Long): be] }
+ assertTrue(!backends.isEmpty(), "No alive backends in ${clusterName}")
+ def backendIds = backends.keySet()
String tableName = "st04_ttl_n_to_0_tpl"
- def ddl = new File("""${context.file.parent}/../ddl/st04_alter_ttl_n_to_0_runtime.sql""").text
- .replace("\${TABLE_NAME}", tableName)
- sql ddl
-
- String[][] backends = sql """show backends"""
- def backendIdToBackendIP = [:]
- def backendIdToBackendHttpPort = [:]
- def backendIdToBackendBrpcPort = [:]
- for (String[] backend in backends) {
- if (backend[9].equals("true") && backend[19].contains("${validCluster}")) {
- backendIdToBackendIP.put(backend[0], backend[1])
- backendIdToBackendHttpPort.put(backend[0], backend[4])
- backendIdToBackendBrpcPort.put(backend[0], backend[5])
- }
+ String dataQuery = "select count(*), sum(k1), sum(length(v1)) from ${tableName}"
+ def tabletIds = []
+ def originalQueryCache = sql("select @@enable_sql_cache, @@enable_query_cache")[0].collect { value ->
+ String setting = value.toString().toLowerCase(Locale.ROOT)
+ assertTrue(setting in ["true", "false", "0", "1"], "Unexpected cache setting: ${value}")
+ setting
}
- assertEquals(backendIdToBackendIP.size(), 1)
- def backendId = backendIdToBackendIP.keySet()[0]
- def clearUrl = backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendHttpPort.get(backendId) + "/api/file_cache?op=clear&sync=true"
- httpTest {
- endpoint ""
- uri clearUrl
- op "get"
- body ""
- check { respCode, body ->
- assertEquals("${respCode}".toString(), "200")
+ def getCache = {
+ sql "use @${clusterName}"
+ def rows = sql """select be_id, cache_path, tablet_id, `hash`, `offset`, size, lower(type)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${backendIds.join(',')})"""
+ def blocks = [:]
+ rows.each { row ->
+ def key = [row[0] as Long, row[1].toString(), row[2] as Long,
+ row[3].toString(), row[4] as Long]
+ assertTrue(backendIds.contains(key[0]) && tabletIds.contains(key[2]),
+ "Cache outside ${clusterName}/${tabletIds}: ${row}")
+ assertTrue(!blocks.containsKey(key), "Duplicate cache block: ${key}")
+ blocks[key] = [size: row[5] as Long, type: row[6]?.toString()]
}
+ blocks
}
-
- def getTabletIds = { String tbl ->
- def tablets = sql """show tablets from ${tbl}"""
- assertTrue(tablets.size() > 0, "No tablets found for table ${tbl}")
- tablets.collect { it[0] as Long }
+ def summarizeCache = { Map blocks ->
+ blocks.groupBy { key, value -> [key[0], key[2]] }.collectEntries { key, entries ->
+ [(key): [blocks: entries.size(),
+ bytesByType: entries.values().groupBy { it.type }.collectEntries { type, values ->
+ [(type): values.sum(0L) { it.size }]
+ }]]
+ }
}
-
- def waitForFileCacheType = { List<Long> tabletIds, String expectedType, long timeoutMs = 600000L, long intervalMs = 2000L ->
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- boolean allMatch = true
- for (Long tabletId in tabletIds) {
- def rows = sql """select type from information_schema.file_cache_info where tablet_id = ${tabletId}"""
- if (rows.isEmpty()) {
- allMatch = false
- break
- }
- def mismatch = rows.find { row -> !row[0]?.toString()?.equalsIgnoreCase(expectedType) }
- if (mismatch) {
- allMatch = false
- break
- }
+ def waitForCache = { String description, Closure ready ->
+ // Updating the interval cannot interrupt an already sleeping background thread.
+ long deadline = System.currentTimeMillis() + 600000L
+ def blocks = [:]
+ def previous = null
+ while (System.currentTimeMillis() < deadline) {
+ blocks = getCache()
+ if (blocks != previous) {
+ logger.info("${description}: cluster=${clusterName}, " +
+ "cache_by_be_tablet=${summarizeCache(blocks)}, blocks=${blocks}")
+ previous = blocks
}
- if (allMatch) {
- return
+ if (ready(blocks)) {
+ return blocks
}
- sleep(intervalMs)
+ sleep(2000)
}
- assertTrue(false, "Timeout waiting for ${expectedType}, tablets=${tabletIds}")
+ assertTrue(false, "Timeout after 600000ms waiting for ${description}: " +
+ "cluster=${clusterName}, tablets=${tabletIds}, BEs=${backendIds}, blocks=${blocks}")
}
-
- def waitTtlCacheSizeZero = { long timeoutMs = 120000L, long intervalMs = 2000L ->
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- long ttlCacheSize = -1L
- httpTest {
- endpoint backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendBrpcPort.get(backendId)
- uri "/brpc_metrics"
- op "get"
- check { respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def lines = out.split('\n')
- for (String line in lines) {
- if (line.startsWith("#")) {
- continue
- }
- if (line.contains("ttl_cache_size")) {
- def idx = line.indexOf(' ')
- ttlCacheSize = line.substring(idx).trim().toLong()
- break
- }
+ def getGlobalTtlBytes = {
+ // Diagnostic only: other tables and busy blocks may keep the BE's TTL queue nonempty.
+ backends.collectEntries { id, be ->
+ Long bytes = null
+ try {
+ String metrics = new URL("http://${be.Host}:${be.BrpcPort}/brpc_metrics")
+ .getText(connectTimeout: 5000, readTimeout: 5000)
+ def values = metrics.readLines().findResults { line ->
+ def fields = line.trim().split(/\s+/)
+ if (fields.size() == 2 && (fields[0] == "file_cache_ttl_cache_size" ||
+ fields[0].endsWith("_file_cache_ttl_cache_size"))) {
+ return fields[1] as Long
}
+ null
}
+ if (!values.isEmpty()) {
+ bytes = values.sum(0L)
+ }
+ } catch (Exception e) {
+ logger.warn("Cannot read diagnostic TTL metric for BE ${id}: ${e.message}")
}
- if (ttlCacheSize == 0L) {
- return
- }
- sleep(intervalMs)
+ [(id): bytes]
}
- assertTrue(false, "Timeout waiting ttl_cache_size = 0")
}
- def insertBatch = { int start, int end ->
- def values = (start..<end).collect { i -> "(${i}, 'value_${i}')" }.join(",")
- sql """insert into ${tableName} values ${values}"""
- }
- insertBatch(0, 200)
- insertBatch(200, 400)
-
- qt_sql """select count(*) from ${tableName} where v1 like 'value_%'"""
- sleep(5000)
+ try {
+ sql "use @${clusterName}"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ def ddl = new File("${context.file.parent}/../ddl/st04_alter_ttl_n_to_0_runtime.sql").text
+ .replace('${TABLE_NAME}', tableName)
+ sql ddl
+ tabletIds = sql_return_maparray("show tablets from ${tableName}")
+ .collect { it.TabletId as Long }.unique().sort()
+ assertTrue(tabletIds.size() == 8, "Expected 8 tablets: ${tabletIds}")
- def tabletIds = getTabletIds.call(tableName)
- waitForFileCacheType.call(tabletIds, "ttl")
+ // Fresh tablet IDs isolate this case without clearing other tables' caches.
+ [0, 200].each { start ->
+ def values = (start..<(start + 200)).collect { i -> "(${i}, 'value_${i}')" }.join(",")
+ sql "insert into ${tableName} values ${values}"
+ }
+ qt_before_alter dataQuery
- // ST-04 未覆盖点模板：运行期 ALTER N->0 后，缓存类型应从 ttl 转为 normal
- sql """alter table ${tableName} set ("file_cache_ttl_seconds"="0")"""
- waitForFileCacheType.call(tabletIds, "normal")
- waitTtlCacheSizeZero.call()
+ def candidate = null
+ long stableSince = 0L
+ def baseline = waitForCache("stable TTL baseline") { blocks ->
+ boolean allTtl = tabletIds.every { id -> blocks.keySet().any { it[2] == id } } &&
+ blocks.values().every { it.size > 0L && it.type == "ttl" }
+ if (!allTtl || blocks != candidate) {
+ candidate = allTtl ? blocks : null
+ stableSince = System.currentTimeMillis()
+ return false
+ }
+ System.currentTimeMillis() - stableSince >= 3000L
+ }
+ def globalBefore = getGlobalTtlBytes()
+ logger.info("TTL before ALTER: global_by_be=${globalBefore}, " +
+ "case_by_be_tablet=${summarizeCache(baseline)}")
+ sql "alter table ${tableName} set (\"file_cache_ttl_seconds\"=\"0\")"
- sql """drop table if exists ${tableName}"""
+ // Do not read table data here: verify the background conversion of existing blocks.
+ def lastDifference = null
+ def converted = waitForCache("existing TTL blocks to become NORMAL") { blocks ->
+ def difference = [missing: baseline.keySet() - blocks.keySet(),
+ unexpected: blocks.keySet() - baseline.keySet(),
+ changed: blocks.findAll { key, value ->
+ value.type != "normal" || value.size != baseline[key]?.size
+ }]
+ if (difference != lastDifference) {
+ logger.info("TTL conversion differences: ${difference}")
+ lastDifference = difference
+ }
+ difference.values().every { it.isEmpty() }
+ }
+ // Exact block equality also ensures every tablet retains its nonzero baseline bytes.
+ def globalAfter = getGlobalTtlBytes()
+ def globalDelta = backendIds.collectEntries { id ->
+ [(id): globalBefore[id] != null && globalAfter[id] != null ?
+ globalAfter[id] - globalBefore[id] : null]
+ }
+ logger.info("TTL after ALTER: global_by_be=${globalAfter}, global_delta_by_be=${globalDelta}, " +
+ "case_by_be_tablet=${summarizeCache(converted)}")
+ qt_after_alter dataQuery
+ } finally {
+ try {
+ sql "use @${clusterName}"
+ sql "drop table if exists ${tableName} force"
+ } finally {
+ try {
+ sql "set enable_sql_cache = ${originalQueryCache[0]}"
+ } finally {
+ sql "set enable_query_cache = ${originalQueryCache[1]}"
+ }
+ }
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.groovy b/regression-test/suites/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.groovy
index 0d8e55c9da5..fbc90987acc 100644
--- a/regression-test/suites/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.groovy
+++ b/regression-test/suites/cloud_p0/cache/ttl/st06_warmup_ttl_type_assert.groovy
@@ -15,7 +15,19 @@
// specific language governing permissions and limitations
// under the License.
-suite("st06_warmup_ttl_type_assert") {
+suite("st06_warmup_ttl_type_assert", "nonConcurrent") {
+ // This setting is static in FE; isolate it and restore it even when a cache assertion fails.
+ String originalSyncLoad = sql("select @@enable_multi_cluster_sync_load")[0][0].toString()
+ assertTrue(originalSyncLoad.toLowerCase() in ["true", "false", "1", "0"],
+ "Unexpected sync-load setting: ${originalSyncLoad}")
+ onFinish {
+ sql "set enable_multi_cluster_sync_load = ${originalSyncLoad}"
+ logger.info("Restored enable_multi_cluster_sync_load=${originalSyncLoad}")
+ }
+ // Only explicit warm-up should populate the target cache in this case.
+ sql "set enable_multi_cluster_sync_load = false"
+ logger.info("Explicit warmup: disabled automatic sync load, previous=${originalSyncLoad}")
+
def customBeConfig = [
enable_evict_file_cache_in_advance : false,
file_cache_enter_disk_resource_limit_mode_percent : 99
@@ -32,95 +44,154 @@ suite("st06_warmup_ttl_type_assert") {
def targetCluster = clusters[1][0]
String tableName = "st06_warmup_ttl_tpl"
- sql """use @${sourceCluster};"""
- def ddl = new File("""${context.file.parent}/../ddl/st06_warmup_ttl_type_assert.sql""").text
- .replace("\${TABLE_NAME}", tableName)
- sql ddl
-
- def values = (0..<200).collect { i -> "(${i}, 'warmup_tpl_${i}')" }.join(",")
- sql """insert into ${tableName} values ${values}"""
- qt_source_preheat """select count(*) from ${tableName}"""
-
- def sourceTablets = sql """show tablets from ${tableName}"""
- assertTrue(sourceTablets.size() > 0, "No tablets found for table ${tableName} in source cluster ${sourceCluster}")
- def sourceTabletIds = sourceTablets.collect { it[0] as Long }
-
- // ST-06 部分覆盖点模板：显式断言 warmup 后目标集群缓存类型为 ttl
- def jobIdRows = sql """warm up cluster ${targetCluster} with table ${tableName};"""
- assertTrue(!jobIdRows.isEmpty())
- def jobId = jobIdRows[0][0]
-
- def waitWarmUpJobFinished = { Object id, long timeoutMs = 600000L, long intervalMs = 5000L ->
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- def stateRows = sql """SHOW WARM UP JOB WHERE ID = ${id}"""
- if (stateRows.isEmpty()) {
- sleep(intervalMs)
- continue
+ // Read both data columns on BE; count(*) may be evaluated entirely in FE.
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ String dataQuery = "select sum(k1), sum(length(v1)) from ${tableName}"
+
+ def getClusterBackends = { String clusterName ->
+ sql_return_maparray("SHOW BACKENDS").findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collectEntries { be -> [(be.BackendId as Long): be] }
+ }
+
+ def waitForTargetTablets = { List<Long> tabletIds ->
+ sql "use @${targetCluster}"
+ long deadline = System.currentTimeMillis() + 120000L
+ def missing = tabletIds
+ while (System.currentTimeMillis() < deadline) {
+ def backends = getClusterBackends.call(targetCluster)
+ def tablets = sql_return_maparray("show tablets from ${tableName}")
+ def readyIds = []
+ for (def tablet : tablets) {
+ long tabletId = tablet.TabletId as Long
+ long primaryBeId = tablet.PrimaryBackendId as Long
+ // BackendId alone can be computed on demand before the primary route exists.
+ if (!tabletIds.contains(tabletId) || !backends.containsKey(primaryBeId) ||
+ (tablet.BackendId as Long) != primaryBeId) {
+ continue
+ }
+ def be = backends[primaryBeId]
+ boolean ready = false
+ try {
+ // Fetch only tablet metadata, leaving the target data cache cold for warm-up.
+ httpTest {
+ endpoint "${be.Host}:${be.HttpPort}"
+ uri "/api/meta/header/${tabletId}"
+ op "get"
+ check { respCode, body ->
+ if ((respCode as int) == 200) {
+ def meta = parseJson(body.toString())
+ // Cloud header JSON omits rs_metas. Check tablet identity/state here;
+ // the source scan and post-warm-up cache checks establish data readiness.
+ ready = (meta.tablet_id as Long) == tabletId &&
+ (meta.ttl_seconds as Long) == 3600L &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null
+ }
+ }
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for tablet ${tabletId} on ${targetCluster}, " +
+ "BE ${primaryBeId}: ${e.message}")
+ }
+ if (ready) {
+ readyIds.add(tabletId)
+ }
}
- def state = stateRows[0][3].toString()
- if ("FINISHED".equalsIgnoreCase(state)) {
+ missing = tabletIds - readyIds
+ if (missing.isEmpty()) {
+ logger.info("Target tablet metadata is ready: cluster=${targetCluster}, tablets=${tabletIds}")
return
}
- if ("CANCELLED".equalsIgnoreCase(state) || "FAILED".equalsIgnoreCase(state)) {
- assertTrue(false, "Warm up job failed, id=${id}, state=${state}")
- }
- sleep(intervalMs)
+ logger.info("Waiting for target tablet metadata: cluster=${targetCluster}, missing=${missing}")
+ sleep(1000)
}
- assertTrue(false, "Timeout waiting warm up job finished, id=${id}")
+ assertTrue(false, "Timeout waiting for target tablet metadata: cluster=${targetCluster}, missing=${missing}")
}
- waitWarmUpJobFinished.call(jobId)
-
- sql """use @${targetCluster};"""
- qt_target_query """select count(*) from ${tableName}"""
- def targetTablets = sql """show tablets from ${tableName}"""
- assertTrue(targetTablets.size() > 0, "No tablets found for table ${tableName} in target cluster ${targetCluster}")
- def targetTabletIds = targetTablets.collect { it[0] as Long }
- assertTrue(sourceTabletIds.size() == targetTabletIds.size(),
- "Tablet size mismatch between source and target, source=${sourceTabletIds.size()}, target=${targetTabletIds.size()}")
-
- def waitForFileCacheType = { List<Long> sourceIds, List<Long> targetIds, String expectedType, long timeoutMs = 600000L, long intervalMs = 2000L ->
- logger.info("waitForFileCacheType, sourceIds=${sourceIds.toString()}, targetIds=${targetIds.toString()}, expectedType=${expectedType}")
- assertTrue(sourceIds.size() == targetIds.size(),
- "Tablet size mismatch before waiting file cache type, source=${sourceIds.size()}, target=${targetIds.size()}")
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- int sourceMatched = 0
- int targetMatched = 0
- for (Long sourceTabletId in sourceIds) {
- def sourceTabletIdStr = sql """select * from information_schema.file_cache_info where tablet_id = ${sourceTabletId}"""
- logger.info("[source tablet] tablet_id=${sourceTabletId}, tablet_cache_info=${sourceTabletIdStr.toString()}")
- def rows = sql """select type from information_schema.file_cache_info where tablet_id = ${sourceTabletId}"""
- if (!rows.isEmpty()) {
- def mismatch = rows.find { row -> !row[0]?.toString()?.equalsIgnoreCase(expectedType) }
- if (!mismatch) {
- sourceMatched++
- }
- }
- }
- for (Long targetTabletId in targetIds) {
- def targetTabletIdStr = sql """select * from information_schema.file_cache_info where tablet_id = ${targetTabletId}"""
- logger.info("[target tablet] tablet_id=${targetTabletId}, tablet_cache_info=${targetTabletIdStr.toString()}")
- def rows = sql """select type from information_schema.file_cache_info where tablet_id = ${targetTabletId}"""
- if (!rows.isEmpty()) {
- def mismatch = rows.find { row -> !row[0]?.toString()?.equalsIgnoreCase(expectedType) }
- if (!mismatch) {
- targetMatched++
- }
- }
+
+ def waitForFileCacheType = { String clusterName, List<Long> tabletIds ->
+ sql "use @${clusterName}"
+ def backendIds = getClusterBackends.call(clusterName).keySet()
+ assertTrue(!backendIds.isEmpty(), "No alive backends in ${clusterName}")
+ long deadline = System.currentTimeMillis() + 600000L
+ def rows = []
+ def missing = tabletIds
+ while (System.currentTimeMillis() < deadline) {
+ rows = sql """select tablet_id, type, be_id from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')})"""
+ assertTrue(rows.every { backendIds.contains(it[2] as Long) },
+ "Cache query ran outside ${clusterName}: expected BEs=${backendIds}, rows=${rows}")
+ missing = tabletIds.findAll { tabletId ->
+ def tabletRows = rows.findAll { (it[0] as Long) == tabletId }
+ tabletRows.isEmpty() || tabletRows.any { !it[1]?.toString()?.equalsIgnoreCase("ttl") }
}
-
- if (sourceMatched == sourceIds.size() && targetMatched == targetIds.size()) {
+ if (missing.isEmpty()) {
return
}
- sleep(intervalMs)
+ logger.info("Waiting for TTL cache: cluster=${clusterName}, missing=${missing}, rows=${rows}")
+ sleep(2000)
}
- assertTrue(false, "Timeout waiting for ${expectedType}, sourceTablets=${sourceIds}, targetTablets=${targetIds}")
+ assertTrue(false, "Timeout waiting for TTL cache: cluster=${clusterName}, missing=${missing}, rows=${rows}")
}
- waitForFileCacheType.call(sourceTabletIds, targetTabletIds, "ttl")
- // cleanup
- sql """use @${sourceCluster};"""
- sql """drop table if exists ${tableName}"""
+ try {
+ sql """use @${sourceCluster};"""
+ def ddl = new File("""${context.file.parent}/../ddl/st06_warmup_ttl_type_assert.sql""").text
+ .replace("\${TABLE_NAME}", tableName)
+ sql ddl
+
+ def values = (0..<200).collect { i -> "(${i}, 'warmup_tpl_${i}')" }.join(",")
+ sql """insert into ${tableName} values ${values}"""
+ qt_source_preheat dataQuery
+
+ def sourceTablets = sql """show tablets from ${tableName}"""
+ assertTrue(sourceTablets.size() > 0, "No tablets found for table ${tableName} in source cluster ${sourceCluster}")
+ def sourceTabletIds = sourceTablets.collect { it[0] as Long }
+
+ waitForTargetTablets.call(sourceTabletIds)
+
+ // ST-06 部分覆盖点模板：显式断言 warmup 后目标集群缓存类型为 ttl
+ def jobIdRows = sql """warm up cluster ${targetCluster} with table ${tableName};"""
+ assertTrue(!jobIdRows.isEmpty())
+ def jobId = jobIdRows[0][0]
+
+ def waitWarmUpJobFinished = { Object id, long timeoutMs = 600000L, long intervalMs = 5000L ->
+ long start = System.currentTimeMillis()
+ while (System.currentTimeMillis() - start < timeoutMs) {
+ def stateRows = sql """SHOW WARM UP JOB WHERE ID = ${id}"""
+ if (stateRows.isEmpty()) {
+ sleep(intervalMs)
+ continue
+ }
+ def state = stateRows[0][3].toString()
+ if ("FINISHED".equalsIgnoreCase(state)) {
+ return
+ }
+ if ("CANCELLED".equalsIgnoreCase(state) || "FAILED".equalsIgnoreCase(state)) {
+ assertTrue(false, "Warm up job failed, id=${id}, state=${state}")
+ }
+ sleep(intervalMs)
+ }
+ assertTrue(false, "Timeout waiting warm up job finished, id=${id}")
+ }
+ waitWarmUpJobFinished.call(jobId)
+
+ sql """use @${targetCluster};"""
+ def targetTablets = sql """show tablets from ${tableName}"""
+ assertTrue(targetTablets.size() > 0, "No tablets found for table ${tableName} in target cluster ${targetCluster}")
+ def targetTabletIds = targetTablets.collect { it[0] as Long }
+ assertTrue(sourceTabletIds.toSet() == targetTabletIds.toSet(),
+ "Tablet IDs differ between source and target, source=${sourceTabletIds}, target=${targetTabletIds}")
+
+ // Check warm-up before reading target data, so a query cannot populate a missing cache.
+ waitForFileCacheType.call(sourceCluster, sourceTabletIds)
+ waitForFileCacheType.call(targetCluster, targetTabletIds)
+ sql """use @${targetCluster};"""
+ qt_target_query dataQuery
+ } finally {
+ sql """use @${sourceCluster};"""
+ sql """drop table if exists ${tableName}"""
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.groovy b/regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.groovy
new file mode 100644
index 00000000000..91ca261fe5b
--- /dev/null
+++ b/regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.groovy
@@ -0,0 +1,57 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_show_ttl", "nonConcurrent") {
+ String tableName = "test_show_ttl_properties"
+ def checkTtlProperty = { long expectedTtl ->
+ def rows = sql "SHOW CREATE TABLE ${tableName}"
+ assertEquals(1, rows.size())
+ assertEquals(tableName, rows[0][0].toString())
+ String createTable = rows[0][1].toString()
+ def matcher = createTable =~ /"file_cache_ttl_seconds"\s*=\s*"(\d+)"/
+ assertTrue(matcher.find(), "Missing file_cache_ttl_seconds in SHOW CREATE TABLE: ${createTable}")
+ long actualTtl = matcher.group(1).toLong()
+ assertFalse(matcher.find(), "Duplicate file_cache_ttl_seconds in SHOW CREATE TABLE: ${createTable}")
+ logger.info("SHOW CREATE TABLE TTL property: table=${tableName}, expected=${expectedTtl}, actual=${actualTtl}")
+ assertEquals(expectedTtl, actualTtl)
+ }
+
+ try {
+ sql "DROP TABLE IF EXISTS ${tableName}"
+ sql """
+ CREATE TABLE ${tableName} (
+ C_CUSTKEY INTEGER NOT NULL,
+ C_NAME VARCHAR(25) NOT NULL,
+ C_ADDRESS VARCHAR(40) NOT NULL,
+ C_NATIONKEY INTEGER NOT NULL,
+ C_PHONE CHAR(15) NOT NULL,
+ C_ACCTBAL DECIMAL(15,2) NOT NULL,
+ C_MKTSEGMENT CHAR(10) NOT NULL,
+ C_COMMENT VARCHAR(117) NOT NULL
+ )
+ DUPLICATE KEY(C_CUSTKEY, C_NAME)
+ DISTRIBUTED BY HASH(C_CUSTKEY) BUCKETS 32
+ PROPERTIES("file_cache_ttl_seconds"="300")
+ """
+ // Other SHOW CREATE TABLE properties depend on cluster defaults.
+ checkTtlProperty(300L)
+ sql "ALTER TABLE ${tableName} SET (\"file_cache_ttl_seconds\"=\"0\")"
+ checkTtlProperty(0L)
+ } finally {
+ sql "DROP TABLE IF EXISTS ${tableName}"
+ }
+}
diff --git a/regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.sql b/regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.sql
deleted file mode 100644
index ce7c5c1986c..00000000000
--- a/regression-test/suites/cloud_p0/cache/ttl/test_show_ttl.sql
+++ /dev/null
@@ -1,40 +0,0 @@
--- Licensed to the Apache Software Foundation (ASF) under one
--- or more contributor license agreements. See the NOTICE file
--- distributed with this work for additional information
--- regarding copyright ownership. The ASF licenses this file
--- to you under the Apache License, Version 2.0 (the
--- "License"); you may not use this file except in compliance
--- with the License. You may obtain a copy of the License at
---
--- http://www.apache.org/licenses/LICENSE-2.0
---
--- Unless required by applicable law or agreed to in writing,
--- software distributed under the License is distributed on an
--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
--- KIND, either express or implied. See the License for the
--- specific language governing permissions and limitations
--- under the License.
-
-DROP TABLE IF EXISTS customer_ttl;
-
-CREATE TABLE IF NOT EXISTS customer_ttl (
- C_CUSTKEY INTEGER NOT NULL,
- C_NAME VARCHAR(25) NOT NULL,
- C_ADDRESS VARCHAR(40) NOT NULL,
- C_NATIONKEY INTEGER NOT NULL,
- C_PHONE CHAR(15) NOT NULL,
- C_ACCTBAL DECIMAL(15,2) NOT NULL,
- C_MKTSEGMENT CHAR(10) NOT NULL,
- C_COMMENT VARCHAR(117) NOT NULL
-)
-DUPLICATE KEY(C_CUSTKEY, C_NAME)
-DISTRIBUTED BY HASH(C_CUSTKEY) BUCKETS 32
-PROPERTIES(
- "file_cache_ttl_seconds"="300"
-);
-
-show create table customer_ttl;
-
-alter table customer_ttl set ("file_cache_ttl_seconds"="0");
-
-show create table customer_ttl;
diff --git a/regression-test/suites/cloud_p0/cache/ttl/test_ttl.groovy b/regression-test/suites/cloud_p0/cache/ttl/test_ttl.groovy
index 09ee37aef07..2deb8eac632 100644
--- a/regression-test/suites/cloud_p0/cache/ttl/test_ttl.groovy
+++ b/regression-test/suites/cloud_p0/cache/ttl/test_ttl.groovy
@@ -15,190 +15,295 @@
// specific language governing permissions and limitations
// under the License.
-import org.codehaus.groovy.runtime.IOGroovyMethods
+import org.apache.doris.regression.util.Http
-suite("test_ttl") {
- def custoBeConfig = [
- enable_evict_file_cache_in_advance : false,
- file_cache_enter_disk_resource_limit_mode_percent : 99,
- file_cache_background_ttl_gc_interval_ms : 1000,
- file_cache_background_ttl_info_update_interval_ms : 1000,
- file_cache_background_tablet_id_flush_interval_ms : 1000
+suite("test_ttl", "nonConcurrent") {
+ def customBeConfig = [
+ enable_evict_file_cache_in_advance: false,
+ file_cache_enter_disk_resource_limit_mode_percent: 99,
+ file_cache_background_ttl_gc_interval_ms: 1000,
+ file_cache_background_ttl_info_update_interval_ms: 1000,
+ file_cache_background_tablet_id_flush_interval_ms: 1000
]
-
- setBeConfigTemporary(custoBeConfig) {
- def clusters = sql " SHOW CLUSTERS; "
- assertTrue(!clusters.isEmpty())
- def validCluster = clusters[0][0]
- sql """use @${validCluster};""";
- def ttlProperties = """ PROPERTIES("file_cache_ttl_seconds"="180") """
- String[][] backends = sql """ show backends """
- String backendId;
- def backendIdToBackendIP = [:]
- def backendIdToBackendHttpPort = [:]
- def backendIdToBackendBrpcPort = [:]
- for (String[] backend in backends) {
- if (backend[9].equals("true") && backend[19].contains("${validCluster}")) {
- backendIdToBackendIP.put(backend[0], backend[1])
- backendIdToBackendHttpPort.put(backend[0], backend[4])
- backendIdToBackendBrpcPort.put(backend[0], backend[5])
+ setBeConfigTemporary(customBeConfig) {
+ String tableName = "test_ttl_natural_expiration"
+ long ttlSeconds = 300L
+ int bucketCount = 4
+ def clusters = sql "SHOW CLUSTERS"
+ assertTrue(!clusters.isEmpty())
+ String clusterName = clusters[0][0].toString()
+ def backends = sql_return_maparray("SHOW BACKENDS").findAll { be ->
+ "${be.Alive}".equalsIgnoreCase("true") &&
+ parseJson(be.Tag.toString()).compute_group_name == clusterName
+ }.collectEntries { be -> [(be.BackendId as Long): be] }
+ assertTrue(!backends.isEmpty(), "No alive backends in ${clusterName}")
+ def backendIds = backends.keySet()
+ def tabletIds = []
+ def metadata = [:]
+ def lastState = [:]
+ def metricsBaseline = null
+ def originalQueryCache = sql("select @@enable_sql_cache, @@enable_query_cache")[0].collect { value ->
+ String setting = value.toString().toLowerCase(Locale.ROOT)
+ assertTrue(setting in ["true", "false", "0", "1"], "Unexpected cache setting: ${value}")
+ setting
}
- }
- assertEquals(backendIdToBackendIP.size(), 1)
-
- backendId = backendIdToBackendIP.keySet()[0]
- def url = backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendHttpPort.get(backendId) + """/api/file_cache?op=clear&sync=true"""
- logger.info(url)
- def clearFileCache = { check_func ->
- try {
- httpTest {
- endpoint ""
- uri url
- op "get"
- body ""
- check check_func
+ def serverTime = { sql("select unix_timestamp()")[0][0] as Long }
+ def waitUntil = { String stage, long deadlineMs, Closure ready ->
+ while (System.currentTimeMillis() < deadlineMs) {
+ if (ready()) {
+ return
+ }
+ sleep(1000)
}
- } catch (Exception e) {
- logger.error("Failed to clear file cache: ${e.message}")
- throw e
+ assertTrue(false, "Timeout waiting for ${stage}: cluster=${clusterName}, " +
+ "tablets=${tabletIds}, BEs=${backendIds}, last_state=${lastState}")
}
- }
-
- sql new File("""${context.file.parent}/../ddl/customer_ttl_delete.sql""").text
- def load_customer_once = { String table ->
- try {
- sql (new File("""${context.file.parent}/../ddl/${table}.sql""").text + ttlProperties)
- sql """ alter table ${table} set ("disable_auto_compaction" = "true") """ // no influence from compaction
- def totalRows = 200
- def batchSize = 100
- def commentSuffix = ' ' + ('X' * 50)
- for (int offset = 0; offset < totalRows; offset += batchSize) {
- def sb = new StringBuilder()
- int batchEnd = Math.min(totalRows, offset + batchSize)
- for (int idx = offset; idx < batchEnd; idx++) {
- def customerId = 10001 + idx
- def customerName = String.format('Customer#%09d', customerId)
- sb.append("""INSERT INTO ${table} VALUES (
- ${customerId},
- '${customerName}',
- 'Address Line 1',
- 15,
- '123-456-7890',
- 12345.67,
- 'AUTOMOBILE',
- 'This is a test comment for the customer.${commentSuffix}'
- );
- """)
- }
- sql sb.toString()
+ def getCache = {
+ sql "use @${clusterName}"
+ def blocks = [:]
+ def rows = sql """select be_id, cache_path, tablet_id, `hash`, `offset`, size, lower(type)
+ from information_schema.file_cache_info
+ where tablet_id in (${tabletIds.join(',')}) and be_id in (${backendIds.join(',')})"""
+ rows.each { row ->
+ def key = [row[0] as Long, row[1].toString(), row[2] as Long,
+ row[3].toString(), row[4] as Long]
+ assertTrue(backendIds.contains(key[0]) && tabletIds.contains(key[2]), "Unexpected cache owner: ${row}")
+ assertTrue(key[4] >= 0L && (row[5] as Long) > 0L, "Invalid cache range: ${row}")
+ assertTrue(!blocks.containsKey(key), "Duplicate cache block: ${key}")
+ blocks[key] = [size: row[5] as Long, type: row[6]?.toString()]
}
- } catch (Exception e) {
- logger.error("Failed to load customer data: ${e.message}")
- throw e
+ blocks
}
- }
-
- def getMetricsMethod = { check_func ->
- try {
- httpTest {
- endpoint backendIdToBackendIP.get(backendId) + ":" + backendIdToBackendBrpcPort.get(backendId)
- uri "/brpc_metrics"
- op "get"
- check check_func
+ def blockSizes = { Map blocks -> blocks.collectEntries { key, value -> [(key): value.size] } }
+ def summarizeCache = { Map blocks ->
+ blocks.groupBy { key, value -> [key[0], key[2]] }.collectEntries { owner, entries ->
+ [(owner): [block_count: entries.size(), bytes: entries.values().sum(0L) { it.size },
+ by_type: (["ttl", "normal"] + entries.values().collect { it.type }).unique().collectEntries { type ->
+ def values = entries.values().findAll { it.type == type }
+ [(type): [block_count: values.size(), bytes: values.sum(0L) { it.size }]]
+ }]]
}
- } catch (Exception e) {
- logger.error("Failed to get metrics: ${e.message}")
- throw e
}
- }
-
- def getTabletIds = { String tableName ->
- def tablets = sql "show tablets from ${tableName}"
- assertTrue(tablets.size() > 0, "No tablets found for table ${tableName}")
- tablets.collect { it[0] as Long }
- }
-
- def waitForFileCacheType = { List<Long> tabletIds, String expectedType, long timeoutMs = 60000L, long intervalMs = 2000L ->
- long start = System.currentTimeMillis()
- while (System.currentTimeMillis() - start < timeoutMs) {
- boolean allMatch = true
- for (Long tabletId in tabletIds) {
- def rows = sql "select type from information_schema.file_cache_info where tablet_id = ${tabletId}"
- if (rows.isEmpty()) {
- logger.warn("file_cache_info is empty for tablet ${tabletId} while waiting for ${expectedType}")
- allMatch = false
- break
+ def metricNames = ["file_cache_ttl_cache_size", "file_cache_normal_queue_cache_size",
+ "file_cache_ttl_cache_lru_queue_size", "file_cache_ttl_cache_lru_queue_element_count",
+ "file_cache_normal_queue_element_count", "file_cache_ttl_cache_evict_size",
+ "file_cache_normal_queue_evict_size", "file_cache_ttl_mgr_tablet_id_set_size"]
+ def getMetrics = {
+ // These include other tables and update asynchronously. Only scoped cache data is asserted.
+ backends.collectEntries { id, be ->
+ def values = metricNames.collectEntries { [(it): null] }
+ try {
+ String text = new URL("http://${be.Host}:${be.BrpcPort}/brpc_metrics")
+ .getText(connectTimeout: 5000, readTimeout: 5000)
+ text.readLines().each { line ->
+ def fields = line.trim().split(/\s+/)
+ if (fields.size() == 2 && !fields[0].startsWith("#")) {
+ metricNames.each { name ->
+ if (fields[0] == name || fields[0].endsWith("_${name}")) {
+ values[name] = (values[name] ?: 0L) + (fields[1] as Long)
+ }
+ }
+ }
+ }
+ } catch (Exception e) {
+ logger.warn("Cannot read diagnostic metrics for BE ${id}: ${e.message}")
}
- def mismatches = rows.findAll { row -> !row[0]?.toString()?.equalsIgnoreCase(expectedType) }
- if (!mismatches.isEmpty()) {
- logger.info("tablet ${tabletId} has cache types ${rows.collect { it[0] }} while waiting for ${expectedType}")
- allMatch = false
- break
+ def missing = values.findAll { name, value -> value == null }.keySet()
+ if (!missing.isEmpty()) {
+ logger.warn("Unavailable diagnostic metrics: BE=${id}, names=${missing}")
}
+ [(id): values]
}
- if (allMatch) {
- logger.info("All file cache entries for tablets ${tabletIds} are ${expectedType}")
- return
+ }
+ def logMetrics = { String stage ->
+ def current = getMetrics()
+ def delta = backendIds.collectEntries { id ->
+ [(id): metricNames.collectEntries { name ->
+ [(name): metricsBaseline != null && metricsBaseline[id][name] != null && current[id][name] != null ?
+ current[id][name] - metricsBaseline[id][name] : null]
+ }]
}
- sleep(intervalMs)
+ logger.info("TTL metrics: stage=${stage}, cluster=${clusterName}, by_be=${current}, " +
+ "delta_from_ttl_baseline=${delta}")
+ current
}
- assertTrue(false, "Timeout waiting for file_cache_info type ${expectedType} for tablets ${tabletIds}")
- }
-
- clearFileCache.call() {
- respCode, body -> {}
- }
- sleep(10000)
+ String address = "Address Line 1"
+ String phone = "123-456-7890"
+ String segment = "AUTOMOBILE"
+ String comment = "This is a test comment for the customer. " + ("X" * 50)
+ def customerIds = (10001..10200).toList()
+ def customerNames = customerIds.collect { String.format('Customer#%09d', it) }
+ def normalize = { List values -> values.collect { new BigDecimal(it.toString()).stripTrailingZeros() } }
+ def expectedData = normalize([customerIds.size(), customerIds.sum(0L), customerNames.sum(0L) { it.length() },
+ 200L * address.length(), 200L * 15L, 200L * phone.length(),
+ 200G * 12345.67G, 200L * segment.length(), 200L * comment.length()])
+ def scanTable = {
+ // Read every column; result caches and metadata-only COUNT cannot satisfy this query.
+ def rows = sql """select count(*), sum(C_CUSTKEY), sum(length(C_NAME)), sum(length(C_ADDRESS)),
+ sum(C_NATIONKEY), sum(length(rtrim(C_PHONE))), sum(C_ACCTBAL),
+ sum(length(rtrim(C_MKTSEGMENT))), sum(length(C_COMMENT)) from ${tableName}"""
+ assertTrue(rows.size() == 1 && rows[0].size() == 9 && rows[0].every { it != null },
+ "Unexpected data result: ${rows}")
+ assertEquals(expectedData, normalize(rows[0]))
+ }
+ Throwable failure = null
+ try {
+ sql "use @${clusterName}"
+ sql "set enable_sql_cache = false"
+ sql "set enable_query_cache = false"
+ logMetrics("before_create")
+ sql "drop table if exists ${tableName} force"
+ long prepareStartedMs = System.currentTimeMillis()
+ // Leave a minute for observing TTL before expiry, even on ASAN and with delayed metadata flushes.
+ long prepareDeadlineMs = prepareStartedMs + (ttlSeconds - 60L) * 1000L
+ String ddl = new File("${context.file.parent}/../ddl/customer_ttl.sql").text
+ .replace('customer_ttl', tableName).replace('BUCKETS 32', "BUCKETS ${bucketCount}")
+ sql(ddl + """ PROPERTIES("file_cache_ttl_seconds"="${ttlSeconds}", "disable_auto_compaction"="true")""")
+ tabletIds = sql_return_maparray("show tablets from ${tableName}")
+ .collect { it.TabletId as Long }.unique().sort()
+ assertTrue(tabletIds.size() == bucketCount, "Expected ${bucketCount} tablets: ${tabletIds}")
+ String values = customerIds.withIndex().collect { id, i ->
+ "(${id}, '${customerNames[i]}', '${address}', 15, '${phone}', 12345.67, '${segment}', '${comment}')"
+ }.join(",")
+ long insertStartedMs = System.currentTimeMillis()
+ sql "insert into ${tableName} values ${values}"
+ logger.info("TTL fixture: rows=200, tablets=${tabletIds}, insert_elapsed_ms=${System.currentTimeMillis() - insertStartedMs}")
+ scanTable()
- def tabletIds = []
- load_customer_once("customer_ttl")
- sleep(10000)
- tabletIds = getTabletIds.call("customer_ttl")
- waitForFileCacheType.call(tabletIds, "ttl")
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- long ttl_cache_size = 0;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
+ // The full scan above verifies data; cloud header JSON does not include rowsets.
+ waitUntil("tablet TTL metadata", prepareDeadlineMs) {
+ metadata = [:]
+ sql_return_maparray("show tablets from ${tableName}").each { tablet ->
+ long id = tablet.TabletId as Long
+ long primary = tablet.PrimaryBackendId as Long
+ if (!(id in tabletIds) || !backends.containsKey(primary) || (tablet.BackendId as Long) != primary) {
+ return
+ }
+ def be = backends[primary]
+ try {
+ def connection = Http.openConnection(
+ "http://${be.Host}:${be.HttpPort}/api/meta/header/${id}")
+ connection.connectTimeout = 5000
+ connection.readTimeout = 5000
+ try {
+ def meta = connection.inputStream.withCloseable {
+ parseJson(it.getText("UTF-8"))
+ }
+ if ((meta.tablet_id as Long) == id && (meta.creation_time as Long) > 0L &&
+ (meta.ttl_seconds as Long) == ttlSeconds &&
+ meta.tablet_state == "PB_RUNNING" && meta.schema != null) {
+ metadata[id] = [be_id: primary, creation_time: meta.creation_time as Long,
+ expires_at: (meta.creation_time as Long) + ttlSeconds]
+ }
+ } finally {
+ connection.disconnect()
+ }
+ } catch (Exception e) {
+ logger.info("Waiting for loaded metadata: tablet=${id}, BE=${primary}, ${e.message}")
}
- def i = line.indexOf(' ')
- ttl_cache_size = line.substring(i).toLong()
- logger.info("ttl_cache_size (initial) line: " + line)
- flag1 = true
}
+ lastState = [missing_tablets: tabletIds - metadata.keySet(), metadata: metadata]
+ metadata.size() == tabletIds.size()
}
- assertTrue(flag1)
- assertTrue(ttl_cache_size > 10737)
- }
- sleep(180000)
- getMetricsMethod.call() {
- respCode, body ->
- assertEquals("${respCode}".toString(), "200")
- String out = "${body}".toString()
- def strs = out.split('\n')
- Boolean flag1 = false;
- for (String line in strs) {
- if (flag1) break;
- if (line.contains("ttl_cache_size")) {
- if (line.startsWith("#")) {
- continue
- }
- def i = line.indexOf(' ')
- logger.info("ttl_cache_size line before assert zero: " + line)
- assertEquals(line.substring(i).toLong(), 0)
- flag1 = true
+ long now = serverTime()
+ assertTrue(metadata.values().every { it.creation_time <= now }, "Future tablet creation time: ${metadata}, now=${now}")
+ long earliestExpiry = metadata.values().min { it.expires_at }.expires_at
+ long latestExpiry = metadata.values().max { it.expires_at }.expires_at
+ logger.info("TTL deadlines: ttl_seconds=${ttlSeconds}, server_now=${now}, by_tablet=${metadata}")
+ def baseline = [:]
+ def candidate = null
+ long stableSince = 0L
+ waitUntil("stable TTL baseline before expiry", prepareDeadlineMs) {
+ baseline = getCache()
+ now = serverTime()
+ assertTrue(now + 30L < earliestExpiry, "Preparation exhausted TTL window: now=${now}, metadata=${metadata}")
+ def missing = tabletIds.findAll { id -> !baseline.keySet().any { it[2] == id } }
+ boolean allTtl = missing.isEmpty() && baseline.values().every { it.type == "ttl" }
+ lastState = [missing_tablets: missing, cache_by_be_tablet: summarizeCache(baseline), seconds_to_expiry: earliestExpiry - now]
+ if (baseline != candidate) {
+ logger.info("TTL baseline progress: ${lastState}")
+ }
+ if (!allTtl || baseline != candidate) {
+ stableSince = System.currentTimeMillis()
}
+ candidate = baseline
+ allTtl && System.currentTimeMillis() - stableSince >= 3000L
}
- assertTrue(flag1)
- }
+ logger.info("TTL baseline: preparation_elapsed_ms=${System.currentTimeMillis() - prepareStartedMs}, " +
+ "cache_by_be_tablet=${summarizeCache(baseline)}, blocks=${baseline}")
+ metricsBaseline = logMetrics("ttl_baseline")
- waitForFileCacheType.call(tabletIds, "normal")
+ // Keep TTL unchanged: this case covers natural expiry, separately from ALTER TTL cases.
+ // A new interval does not wake a thread already sleeping for the old 180-second interval.
+ now = serverTime()
+ long conversionDeadlineMs = System.currentTimeMillis() + Math.max(0L, latestExpiry + 1L - now) * 1000L + 600000L
+ def firstNormalAt = [:]
+ def converted = [:]
+ def previous = null
+ long lastLogMs = 0L
+ stableSince = 0L
+ waitUntil("natural TTL expiration", conversionDeadlineMs) {
+ // Do not read business data until the original blocks have converted completely.
+ converted = getCache()
+ now = serverTime() // Read time after the snapshot to avoid a boundary crossing during the query.
+ def differences = [missing: (baseline.keySet() - converted.keySet()).take(10),
+ unexpected: (converted.keySet() - baseline.keySet()).take(10),
+ changed_sizes: converted.findAll { key, value -> value.size != baseline[key]?.size }.take(10)]
+ converted.each { key, value ->
+ // FE sets creation_time; BE checks its own clock. Allow two seconds of clock skew.
+ assertTrue(value.type != "normal" || now + 2L >= metadata[key[2]].expires_at,
+ "Cache converted before expiry: block=${key}, now=${now}, metadata=${metadata[key[2]]}")
+ }
+ tabletIds.each { id ->
+ def expected = baseline.findAll { key, value -> key[2] == id }
+ def actual = converted.findAll { key, value -> key[2] == id }
+ if (!firstNormalAt.containsKey(id) && blockSizes(expected) == blockSizes(actual) &&
+ actual.values().every { it.type == "normal" }) {
+ firstNormalAt[id] = now
+ }
+ }
+ boolean complete = blockSizes(baseline) == blockSizes(converted) &&
+ converted.values().every { it.type == "normal" } && now > latestExpiry
+ lastState = [server_now: now, seconds_to_last_expiry: latestExpiry - now,
+ cache_by_be_tablet: summarizeCache(converted), differences: differences,
+ tablets_not_yet_normal: tabletIds - firstNormalAt.keySet()]
+ if (converted != previous || System.currentTimeMillis() - lastLogMs >= 30000L) {
+ logger.info("TTL expiration progress: ${lastState}")
+ lastLogMs = System.currentTimeMillis()
+ }
+ if (!complete || converted != previous) {
+ stableSince = System.currentTimeMillis()
+ }
+ previous = converted
+ complete && System.currentTimeMillis() - stableSince >= 3000L
+ }
+ def timing = metadata.collectEntries { id, meta ->
+ [(id): meta + [first_normal_observed_at: firstNormalAt[id],
+ observed_conversion_lag_seconds: firstNormalAt[id] - meta.expires_at]]
+ }
+ logger.info("TTL natural expiration verified: by_tablet=${timing}, " +
+ "before=${summarizeCache(baseline)}, after=${summarizeCache(converted)}")
+ logMetrics("normal_after_expiry")
+ scanTable()
+ } catch (Throwable t) {
+ failure = t
+ logger.warn("TTL case failed: cluster=${clusterName}, metadata=${metadata}, last_state=${lastState}")
+ try { logMetrics("failure") } catch (Throwable diagnosticError) { t.addSuppressed(diagnosticError) }
+ throw t
+ } finally {
+ def cleanupErrors = []
+ try { sql "drop table if exists ${tableName} force" } catch (Throwable t) { cleanupErrors.add(t) }
+ [enable_sql_cache: originalQueryCache[0], enable_query_cache: originalQueryCache[1]].each { name, value ->
+ try { sql "set ${name} = ${value}" } catch (Throwable t) { cleanupErrors.add(t) }
+ }
+ if (!cleanupErrors.isEmpty()) {
+ if (failure != null) {
+ cleanupErrors.each { failure.addSuppressed(it) }
+ } else {
+ cleanupErrors.tail().each { cleanupErrors[0].addSuppressed(it) }
+ throw cleanupErrors[0]
+ }
+ }
+ }
}
}
diff --git a/regression-test/suites/cloud_p0/cache/warm_up/test_file_cache_warmup_read_metrics_docker.groovy b/regression-test/suites/cloud_p0/cache/warm_up/test_file_cache_warmup_read_metrics_docker.groovy
index e2fc6d6fe65..8ef6eed35d7 100644
--- a/regression-test/suites/cloud_p0/cache/warm_up/test_file_cache_warmup_read_metrics_docker.groovy
+++ b/regression-test/suites/cloud_p0/cache/warm_up/test_file_cache_warmup_read_metrics_docker.groovy
@@ -60,8 +60,8 @@ suite('test_file_cache_warmup_read_metrics_docker', 'docker') {
'DORIS_CLOUD_AK=minioadmin',
'DORIS_CLOUD_SK=minioadmin',
"DORIS_CLOUD_BUCKET=${minioBucket}",
- "DORIS_CLOUD_ENDPOINT=host.docker.internal:${minioPort}",
- "DORIS_CLOUD_EXTERNAL_ENDPOINT=host.docker.internal:${minioPort}",
+ "DORIS_CLOUD_ENDPOINT=http://host.docker.internal:${minioPort}",
+ "DORIS_CLOUD_EXTERNAL_ENDPOINT=http://host.docker.internal:${minioPort}",
'DORIS_CLOUD_REGION=us-east-1',
'DORIS_CLOUD_PROVIDER=S3',
]
diff --git a/regression-test/suites/cloud_p0/cache/warm_up/test_warmup_table_docker.groovy b/regression-test/suites/cloud_p0/cache/warm_up/test_warmup_table_docker.groovy
index e93560b389e..14c02d48f8b 100644
--- a/regression-test/suites/cloud_p0/cache/warm_up/test_warmup_table_docker.groovy
+++ b/regression-test/suites/cloud_p0/cache/warm_up/test_warmup_table_docker.groovy
@@ -27,7 +27,7 @@ suite('test_warmup_table_docker', 'docker') {
options.beConfigs += [
'file_cache_enter_disk_resource_limit_mode_percent=99',
'enable_evict_file_cache_in_advance=false',
- 'enable_only_warm_up_idx=true',
+ 'file_cache_enable_only_warm_up_idx=true',
'file_cache_background_gc_interval_ms=10',
'file_cache_background_monitor_interval_ms=100',
'file_cache_background_block_lru_update_interval_ms=100',
@@ -189,8 +189,8 @@ suite('test_warmup_table_docker', 'docker') {
def cache_size_after_clear = getClusterFileCacheSize(clusterName)
assertEquals(0, cache_size_after_clear)
- // Set enable_only_warm_up_idx = true
- updateBeConf(clusterName, "enable_only_warm_up_idx", "true")
+ // Set file_cache_enable_only_warm_up_idx = true
+ updateBeConf(clusterName, "file_cache_enable_only_warm_up_idx", "true")
// Trigger warm up
def jobId = sql "WARM UP CLUSTER ${clusterName} WITH TABLE ${testTable}"
diff --git a/regression-test/suites/cloud_p0/compaction/test_cloud_alter_disable_auto_compaction.groovy b/regression-test/suites/cloud_p0/compaction/test_cloud_alter_disable_auto_compaction.groovy
index 799cc05810b..65d78d48ed7 100644
--- a/regression-test/suites/cloud_p0/compaction/test_cloud_alter_disable_auto_compaction.groovy
+++ b/regression-test/suites/cloud_p0/compaction/test_cloud_alter_disable_auto_compaction.groovy
@@ -28,8 +28,10 @@ suite("test_cloud_alter_disable_auto_compaction", "p0,docker") {
'cloud_cluster_check_interval_second=1',
]
options.beConfigs += [
- // Short sync interval for faster property propagation
- 'tablet_sync_interval_s=5',
+ // A tablet becomes eligible for meta sync after this interval.
+ 'tablet_sync_interval_s=1',
+ // The background sync thread must also run frequently enough to propagate ALTER properties.
+ 'schedule_sync_tablets_interval_s=1',
// Trigger cumulative compaction after 5 rowsets
'cumulative_compaction_min_deltas=5',
// Enable file cache (required for sync_meta to work)
@@ -116,7 +118,7 @@ suite("test_cloud_alter_disable_auto_compaction", "p0,docker") {
assertTrue(result[0][1].contains('"disable_auto_compaction" = "true"'))
// Wait for sync_meta to propagate the change to BE
- // tablet_sync_interval_s is set to 5 seconds, so wait a bit longer
+ // Both tablet staleness and background scheduling intervals are set to 1 second.
logger.info("Waiting for sync_meta to propagate disable_auto_compaction=true to BE...")
sleep(10000)
diff --git a/regression-test/suites/cloud_p0/multi_cluster/async_copy_into/async_load.groovy b/regression-test/suites/cloud_p0/multi_cluster/async_copy_into/async_load.groovy
index 0e346d0389a..89cb6124e1c 100644
--- a/regression-test/suites/cloud_p0/multi_cluster/async_copy_into/async_load.groovy
+++ b/regression-test/suites/cloud_p0/multi_cluster/async_copy_into/async_load.groovy
@@ -18,6 +18,7 @@
import groovy.json.JsonOutput
suite("async_load") {
+ withRestoredMultiClusterState(false) {
def externalStageName = "regression_test_tpch"
def prefix = "tpch/sf1"
@@ -51,7 +52,7 @@ suite("async_load") {
}
}
}
- wait_cluster_change()
+ sleep(20000)
List<List<Object>> result = sql "show clusters"
assertTrue(result.size() == 0);
@@ -60,7 +61,7 @@ suite("async_load") {
"regression_cluster_name0", "regression_cluster_id0");
add_cluster.call(beUniqueIdList[1], ipList[1], hbPortList[1],
"regression_cluster_name1", "regression_cluster_id1");
- wait_cluster_change()
+ sleep(20000)
result = sql "show clusters"
assertEquals(result.size(), 2);
@@ -97,7 +98,7 @@ suite("async_load") {
'prefix' = 'regression' ,
'ak' = '${getS3AK()}' ,
'sk' = '${getS3SK()}' ,
- 'provider' = '${getProvider()}',
+ 'provider' = '${getS3Provider()}',
'access_type' = 'aksk',
'default.file.column_separator' = "|"
);
@@ -109,7 +110,7 @@ suite("async_load") {
assertTrue(result.size() == 1)
assertTrue(result[0].size() == 8)
//assertTrue(result[0][1].equals("FINISHED"))
- id = result[0][0]
+ def id = result[0][0]
while(true) {
result = sql "show copy where id = \'${id}\'"
logger.info("copy result: " + result)
@@ -272,4 +273,5 @@ suite("async_load") {
assertTrue(before_cluster0_query_scan_rows == after_cluster0_query_scan_rows)
assertTrue(before_cluster1_query_scan_rows < after_cluster1_query_scan_rows)
+ }
}
diff --git a/regression-test/suites/cloud_p0/multi_cluster/copy_into/sync_laod.groovy b/regression-test/suites/cloud_p0/multi_cluster/copy_into/sync_laod.groovy
index 72c84017185..32c7344e838 100644
--- a/regression-test/suites/cloud_p0/multi_cluster/copy_into/sync_laod.groovy
+++ b/regression-test/suites/cloud_p0/multi_cluster/copy_into/sync_laod.groovy
@@ -16,8 +16,25 @@
// under the License.
import groovy.json.JsonOutput
+import org.apache.doris.regression.action.ProfileAction
suite("sync_load") {
+ withRestoredMultiClusterState(false) {
+
+ def queryFrontend = sql_return_maparray('SHOW FRONTENDS').find {
+ it.CurrentConnected?.toString()?.equalsIgnoreCase('Yes')
+ }
+ assertNotNull(queryFrontend, 'Cannot locate the FE executing profile queries')
+ def profileAction = new ProfileAction(context,
+ new InetSocketAddress(queryFrontend.Host.toString(), queryFrontend.HttpPort.toString().toInteger()))
+ def checkQueryProfile = { query, pattern ->
+ // Match this execution, even when the profile list normalizes or truncates the SQL text.
+ def marker = "multi_cluster_${UUID.randomUUID().toString().replace('-', '')}"
+ sql 'sync'
+ sql "/* ${marker} */ ${query}"
+ profileAction.getProfileBySql(marker, [pattern.toString()], 60000L, 500L)
+ }
+
def externalStageName = "regression_test_tpch"
def prefix = "tpch/sf1"
@@ -25,7 +42,6 @@ suite("sync_load") {
List<String> hbPortList = new ArrayList<>()
List<String> httpPortList = new ArrayList<>()
List<String> beUniqueIdList = new ArrayList<>()
- List<String> bePortList = new ArrayList<>()
String[] bes = context.config.multiClusterBes.split(',');
println("the value is " + context.config.multiClusterBes);
@@ -51,7 +67,7 @@ suite("sync_load") {
}
}
}
- wait_cluster_change()
+ sleep(20000)
List<List<Object>> result = sql "show clusters"
assertTrue(result.size() == 0);
@@ -60,7 +76,7 @@ suite("sync_load") {
"regression_cluster_name0", "regression_cluster_id0");
add_cluster.call(beUniqueIdList[1], ipList[1], hbPortList[1],
"regression_cluster_name1", "regression_cluster_id1");
- wait_cluster_change()
+ sleep(20000)
result = sql "show clusters"
assertEquals(result.size(), 2);
@@ -98,7 +114,7 @@ suite("sync_load") {
'prefix' = 'regression' ,
'ak' = '${getS3AK()}' ,
'sk' = '${getS3SK()}' ,
- 'provider' = '${getProvider()}',
+ 'provider' = '${getS3Provider()}',
'access_type' = 'aksk',
'default.file.column_separator' = "|"
);
@@ -205,21 +221,18 @@ suite("sync_load") {
// sql new File("""${context.file.parent}/ddl/${sql}.sql""").text
//}
- // fill bePortList
- for (int i = 0; i < ipList.size(); ++i) {
- result = sql """show backends"""
- for (row : result) {
- println row
- println row[2]
- if (ipList[i] == row[1] && hbPortList[i] == row[2]) {
- bePortList.add(row[3]);
- }
+ // QeProcessorImpl attaches execution profiles using Backend.getHeartbeatAddress().
+ def profileBackends = sql_return_maparray('SHOW BACKENDS')
+ def profileHeartbeatPorts = (0..1).collect { index ->
+ def backend = profileBackends.find {
+ it.Host == ipList[index] && it.HeartbeatPort.toString() == hbPortList[index]
}
+ assertNotNull(backend, "Missing profile backend ${ipList[index]}:${hbPortList[index]}")
+ backend.HeartbeatPort.toString()
}
- assertEquals(bePortList.size(), 2);
// q01
- sql """
+ def q01 = """
SELECT
l_returnflag,
l_linestatus,
@@ -243,11 +256,10 @@ suite("sync_load") {
l_linestatus
"""
- def set = [ipList[0] + ":" +bePortList[0]] as Set
- checkProfile.call(set)
+ checkQueryProfile(q01, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
// q02
- sql """
+ def q02 = """
SELECT
s_acctbal,
s_name,
@@ -290,10 +302,10 @@ suite("sync_load") {
p_partkey
LIMIT 100
"""
- checkProfile.call(set)
+ checkQueryProfile(q02, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
// q03
- sql """
+ def q03 = """
SELECT
l_orderkey,
sum(l_extendedprice * (1 - l_discount)) AS revenue,
@@ -318,10 +330,10 @@ suite("sync_load") {
o_orderdate
LIMIT 10
"""
- checkProfile.call(set)
+ checkQueryProfile(q03, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
// q04
- sql """
+ def q04 = """
SELECT
o_orderpriority,
count(*) AS order_count
@@ -341,10 +353,10 @@ suite("sync_load") {
ORDER BY
o_orderpriority
"""
- checkProfile.call(set)
+ checkQueryProfile(q04, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
// q05
- sql """
+ def q05 = """
SELECT
n_name,
sum(l_extendedprice * (1 - l_discount)) AS revenue
@@ -370,10 +382,10 @@ suite("sync_load") {
ORDER BY
revenue DESC
"""
- checkProfile.call(set)
+ checkQueryProfile(q05, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
// q06
- sql """
+ def q06 = """
SELECT sum(l_extendedprice * l_discount) AS revenue
FROM
lineitem
@@ -383,10 +395,10 @@ suite("sync_load") {
AND l_discount BETWEEN 0.06 - 0.01 AND .06 + 0.01
AND l_quantity < 24
"""
- checkProfile.call(set)
+ checkQueryProfile(q06, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
// q07
- sql """
+ def q07 = """
SELECT
supp_nation,
cust_nation,
@@ -426,13 +438,12 @@ suite("sync_load") {
cust_nation,
l_year
"""
- checkProfile.call(set)
+ checkQueryProfile(q07, "hostname:${ipList[0]}, port:${profileHeartbeatPorts[0]}")
sql "use @regression_cluster_name1"
- set = [ipList[1] + ":" +bePortList[1]] as Set
// q08
- sql """
+ def q08 = """
SELECT
o_year,
sum(CASE
@@ -471,10 +482,10 @@ suite("sync_load") {
ORDER BY
o_year
"""
- checkProfile.call(set)
+ checkQueryProfile(q08, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q09
- sql """
+ def q09 = """
SELECT
nation,
o_year,
@@ -507,10 +518,10 @@ suite("sync_load") {
nation,
o_year DESC
"""
- checkProfile.call(set)
+ checkQueryProfile(q09, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q10
- sql """
+ def q10 = """
SELECT
c_custkey,
c_name,
@@ -544,10 +555,10 @@ suite("sync_load") {
revenue DESC
LIMIT 20
"""
- checkProfile.call(set)
+ checkQueryProfile(q10, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q11
- sql """
+ def q11 = """
SELECT
ps_partkey,
sum(ps_supplycost * ps_availqty) AS value
@@ -576,10 +587,10 @@ suite("sync_load") {
ORDER BY
value DESC
"""
- checkProfile.call(set)
+ checkQueryProfile(q11, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q12
- sql """
+ def q12 = """
SELECT
l_shipmode,
sum(CASE
@@ -609,10 +620,10 @@ suite("sync_load") {
ORDER BY
l_shipmode
"""
- checkProfile.call(set)
+ checkQueryProfile(q12, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q13
- sql """
+ def q13 = """
SELECT
c_count,
count(*) AS custdist
@@ -634,10 +645,10 @@ suite("sync_load") {
custdist DESC,
c_count DESC
"""
- checkProfile.call(set)
+ checkQueryProfile(q13, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q14
- sql """
+ def q14 = """
SELECT 100.00 * sum(CASE
WHEN p_type LIKE 'PROMO%'
THEN l_extendedprice * (1 - l_discount)
@@ -651,10 +662,10 @@ suite("sync_load") {
AND l_shipdate >= DATE '1995-09-01'
AND l_shipdate < DATE '1995-09-01' + INTERVAL '1' MONTH
"""
- checkProfile.call(set)
+ checkQueryProfile(q14, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q15
- sql """
+ def q15 = """
SELECT
s_suppkey,
s_name,
@@ -674,10 +685,10 @@ suite("sync_load") {
ORDER BY
s_suppkey;
"""
- checkProfile.call(set)
+ checkQueryProfile(q15, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q16
- sql """
+ def q16 = """
SELECT
p_brand,
p_type,
@@ -708,10 +719,10 @@ suite("sync_load") {
p_type,
p_size
"""
- checkProfile.call(set)
+ checkQueryProfile(q16, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q17
- sql """
+ def q17 = """
SELECT sum(l_extendedprice) / 7.0 AS avg_yearly
FROM
lineitem,
@@ -728,10 +739,10 @@ suite("sync_load") {
l_partkey = p_partkey
)
"""
- checkProfile.call(set)
+ checkQueryProfile(q17, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q18
- sql """
+ def q18 = """
SELECT
c_name,
c_custkey,
@@ -766,10 +777,10 @@ suite("sync_load") {
o_orderdate
LIMIT 100
"""
- checkProfile.call(set)
+ checkQueryProfile(q18, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q19
- sql """
+ def q19 = """
SELECT sum(l_extendedprice * (1 - l_discount)) AS revenue
FROM
lineitem,
@@ -805,10 +816,10 @@ suite("sync_load") {
AND l_shipinstruct = 'DELIVER IN PERSON'
)
"""
- checkProfile.call(set)
+ checkQueryProfile(q19, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
// q20
- sql """
+ def q20 = """
SELECT
s_name,
s_address
@@ -842,5 +853,6 @@ suite("sync_load") {
AND n_name = 'CANADA'
ORDER BY s_name
"""
- checkProfile.call(set)
+ checkQueryProfile(q20, "hostname:${ipList[1]}, port:${profileHeartbeatPorts[1]}")
+ }
}
diff --git a/regression-test/suites/cloud_p0/multi_cluster/default_cluster.groovy b/regression-test/suites/cloud_p0/multi_cluster/default_cluster.groovy
index 7580ca13a8b..5d7f1368c4b 100644
--- a/regression-test/suites/cloud_p0/multi_cluster/default_cluster.groovy
+++ b/regression-test/suites/cloud_p0/multi_cluster/default_cluster.groovy
@@ -18,6 +18,7 @@
import groovy.json.JsonOutput
suite("default_cluster") {
+ withRestoredMultiClusterState(false) {
def token = context.config.metaServiceToken
def instance_id = context.config.multiClusterInstance
@@ -27,9 +28,9 @@ suite("default_cluster") {
List<String> beUniqueIdList = new ArrayList<>()
String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
+ logger.info("the value is " + context.config.multiClusterBes);
for(String values : bes) {
- println("the value is " + values);
+ logger.info("the value is " + values);
String[] beInfo = values.split(':');
ipList.add(beInfo[0]);
hbPortList.add(beInfo[1]);
@@ -37,10 +38,10 @@ suite("default_cluster") {
beUniqueIdList.add(beInfo[3]);
}
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
+ logger.info("the ip is " + ipList);
+ logger.info("the heartbeat port is " + hbPortList);
+ logger.info("the http port is " + httpPortList);
+ logger.info("the be unique id is " + beUniqueIdList);
for (unique_id : beUniqueIdList) {
resp = get_cluster.call(unique_id);
@@ -50,7 +51,7 @@ suite("default_cluster") {
}
}
}
- wait_cluster_change()
+ sleep(20000)
List<List<Object>> result = sql "show clusters"
assertTrue(result.size() == 0);
@@ -59,22 +60,22 @@ suite("default_cluster") {
"regression_cluster_name2", "regression_cluster_id2");
add_cluster.call(beUniqueIdList[1], ipList[1], hbPortList[1],
"regression_cluster_name3", "regression_cluster_id3");
- wait_cluster_change()
+ sleep(20000)
result = sql "show clusters"
assertTrue(result.size() == 2);
for (row : result) {
- log.info("show cluster row: ${row}".toString())
+ logger.info("show cluster row: ${row}".toString())
}
sql "SET PROPERTY 'default_cloud_cluster' = 'regression_cluster_name3'"
context.reconnectFe()
def before_cluster0_flush = get_be_metric(ipList[0], httpPortList[0], "memtable_flush_total");
- log.info("before_cluster0_flush : ${before_cluster0_flush}".toString())
+ logger.info("before_cluster0_flush : ${before_cluster0_flush}".toString())
def before_cluster1_flush = get_be_metric(ipList[1], httpPortList[1], "memtable_flush_total");
- log.info("before_cluster1_flush : ${before_cluster1_flush}".toString())
+ logger.info("before_cluster1_flush : ${before_cluster1_flush}".toString())
sql """
@@ -92,17 +93,17 @@ suite("default_cluster") {
"""
def after_cluster0_flush = get_be_metric(ipList[0], httpPortList[0], "memtable_flush_total");
- log.info("after_cluster0_flush : ${after_cluster0_flush}".toString())
+ logger.info("after_cluster0_flush : ${after_cluster0_flush}".toString())
def after_cluster1_flush = get_be_metric(ipList[1], httpPortList[1], "memtable_flush_total");
- log.info("after_cluster1_flush : ${after_cluster1_flush}".toString())
+ logger.info("after_cluster1_flush : ${after_cluster1_flush}".toString())
assertTrue(before_cluster0_flush == after_cluster0_flush)
assertTrue(before_cluster1_flush < after_cluster1_flush)
result = sql "show clusters"
for (row : result) {
- println row
+ logger.info("row: ${row.toString()}")
if(row[0] == "regression_cluster_name2") {
assertTrue(row[1].toString().toLowerCase() == "false")
}
@@ -115,7 +116,7 @@ suite("default_cluster") {
sql "use @regression_cluster_name2"
result = sql "show clusters"
for (row : result) {
- println row
+ logger.info("row: ${row.toString()}")
if(row[0] == "regression_cluster_name2") {
assertTrue(row[1].toString().toLowerCase() == "true")
}
@@ -128,7 +129,7 @@ suite("default_cluster") {
context.reconnectFe()
result = sql "show clusters"
for (row : result) {
- println row
+ logger.info("row: ${row.toString()}")
if(row[0] == "regression_cluster_name2") {
assertTrue(row[1].toString().toLowerCase() == "true")
}
@@ -140,7 +141,7 @@ suite("default_cluster") {
sql "use @regression_cluster_name3"
result = sql "show clusters"
for (row : result) {
- println row
+ logger.info("row: ${row.toString()}")
if(row[0] == "regression_cluster_name2") {
assertTrue(row[1].toString().toLowerCase() == "false")
}
@@ -160,7 +161,7 @@ suite("default_cluster") {
result = sql "show property"
for (row : result) {
- println row
+ logger.info("row: ${row.toString()}")
if(row[0] == "default_cloud_cluster") {
assertTrue(row[1].toString().isEmpty())
}
@@ -169,4 +170,5 @@ suite("default_cluster") {
sql """
drop table if exists test_table
"""
+ }
}
diff --git a/regression-test/suites/cloud_p0/multi_cluster/multi_cluster_s3_load/load.groovy b/regression-test/suites/cloud_p0/multi_cluster/multi_cluster_s3_load/load.groovy
index 20108615679..aeed4a1c95b 100644
--- a/regression-test/suites/cloud_p0/multi_cluster/multi_cluster_s3_load/load.groovy
+++ b/regression-test/suites/cloud_p0/multi_cluster/multi_cluster_s3_load/load.groovy
@@ -27,52 +27,33 @@
// sed -nr 's/.*tables: (.*)$/\1/gp' /path/to/*.sql | sed -nr 's/,/\n/gp' | sort | uniq
import groovy.json.JsonOutput
+import org.apache.doris.regression.util.MultiClusterStateGuard
suite("load") {
- List<String> ipList = new ArrayList<>()
- List<String> hbPortList = new ArrayList<>()
- List<String> httpPortList = new ArrayList<>()
- List<String> beUniqueIdList = new ArrayList<>()
-
- String[] bes = context.config.multiClusterBes.split(',');
- println("the value is " + context.config.multiClusterBes);
- for(String values : bes) {
- println("the value is " + values);
- String[] beInfo = values.split(':');
- ipList.add(beInfo[0]);
- hbPortList.add(beInfo[1]);
- httpPortList.add(beInfo[2]);
- beUniqueIdList.add(beInfo[3]);
- }
-
- println("the ip is " + ipList);
- println("the heartbeat port is " + hbPortList);
- println("the http port is " + httpPortList);
- println("the be unique id is " + beUniqueIdList);
-
- for (unique_id : beUniqueIdList) {
- resp = get_cluster.call(unique_id);
- for (cluster : resp) {
- if (cluster.type == "COMPUTE") {
- drop_cluster.call(cluster.cluster_name, cluster.cluster_id);
- }
+ withRestoredMultiClusterState(true) {
+ def baseline = multiClusterBaseline()
+ def ipList = baseline.collect { it.nodes[0].ip }
+ def httpPortList = baseline.collect { it.nodes[0].http_port.toString() }
+
+ // Establish the shared baseline after a previous suite or run left different group names.
+ def existing = get_cluster.call(baseline[0].nodes[0].cloud_unique_id)
+ .findAll { it.type == 'COMPUTE' }
+ if (!MultiClusterStateGuard.frontendMatches(baseline,
+ sql_return_maparray('SHOW CLUSTERS'), sql_return_maparray('SHOW BACKENDS'))) {
+ existing.each { group -> drop_cluster.call(group.cluster_name, group.cluster_id) }
+ sleep(20000)
+ baseline.each { group ->
+ def node = group.nodes[0]
+ add_cluster.call(node.cloud_unique_id, node.ip, node.heartbeat_port.toString(),
+ group.cluster_name, group.cluster_id)
}
+ sleep(20000)
}
- wait_cluster_change()
-
- List<List<Object>> result = sql "show clusters"
- assertTrue(result.size() == 0);
+ checkMultiClusterBaseline(baseline)
+ sql "SET PROPERTY 'default_cloud_cluster' = '${baseline[0].cluster_name}'"
+ List<List<Object>> result
- add_cluster.call(beUniqueIdList[0], ipList[0], hbPortList[0],
- "regression_cluster_name0", "regression_cluster_id0");
- add_cluster.call(beUniqueIdList[1], ipList[1], hbPortList[1],
- "regression_cluster_name1", "regression_cluster_id1");
- wait_cluster_change()
-
- result = sql "show clusters"
- assertEquals(result.size(), 2);
-
... 12426 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org