You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by li...@apache.org on 2024/03/29 07:15:41 UTC
(doris) branch high-priority-column updated: Recover (#33021)
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch high-priority-column
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/high-priority-column by this push:
new 19189e00294 Recover (#33021)
19189e00294 is described below
commit 19189e002949b2c101d9ed239fccc2142466ce9a
Author: Jibing-Li <64...@users.noreply.github.com>
AuthorDate: Fri Mar 29 15:15:33 2024 +0800
Recover (#33021)
* [fix](merge cloud) Fix cloud be set be tag map (#32864)
* [chore] Add gavinchou to collaborators (#32881)
* [chore](show) support statement to show views from table (#32358)
MySQL [test]> show views;
+----------------+
| Tables_in_test |
+----------------+
| t1_view |
| t2_view |
+----------------+
2 rows in set (0.00 sec)
MySQL [test]> show views like '%t1%';
+----------------+
| Tables_in_test |
+----------------+
| t1_view |
+----------------+
1 row in set (0.01 sec)
MySQL [test]> show views where create_time > '2024-03-18';
+----------------+
| Tables_in_test |
+----------------+
| t2_view |
+----------------+
1 row in set (0.02 sec)
* [Enhancement](ranger) Disable some permission operations when Ranger or LDAP are enabled (#32538)
Disable some permission operations when Ranger or LDAP are enabled.
* [chore](ci) exclude unstable trino_connector case (#32892)
Co-authored-by: stephen <he...@qq.com>
* [fix](Nereids) NPE when create table with implicit index type (#32893)
* [improvement](mtmv) Support more join types for query rewriting by materialized view (#32685)
This pattern of rewriting is supported for multi-table joins and supported join types is as following:
INNER JOIN
LEFT OUTER JOIN
RIGHT OUTER JOIN
FULL OUTER JOIN
LEFT SEMI JOIN
RIGHT SEMI JOIN
LEFT ANTI JOIN
RIGHT ANTI JOIN
* [Serde](Variant) support arrow serialization for varint type (#32780)
* [fix](multicatalog) fix no data error when read hive table on cosn (#32815)
Currently, when reading a hive on cosn table, doris return empty result, but the table has data.
iceberg on cosn is ok.
The reason is misuse of cosn's file sytem. according to cosn's doc, its fs.cosn.impl should be org.apache.hadoop.fs.CosFileSystem
* [fix](nereids)EliminateGroupByConstant should replace agg's output after removing constant group by keys (#32878)
* [Fix](executor)Fix regression test for test_active_queries/test_backend_active_tasks #32899
* [fix](iceberg) fix iceberg catalog bug and p2 test cases (#32898)
1. Fix iceberg catalog bug
This PR #30198 change the logic of `IcebergHMSExternalCatalog.java`,
to get locationUrl by calling hive metastore's `getCatalog()` method.
But this method only exists in hive 3+. So it will fail if we using hive 2.x.
I temporary remove this logic, because this logic is only used from iceberg table writing.
Which is still under development. We will rethink this logic later.
2. Fix test cases
Some of P2 test cases missed `order_qt`. And because the output format of the floating point
type is changed, some result in `out` files need to be regenerated.
* [revert](jni) revert part of #32455 (#32904)
* [fix](spill) Avoid releasing resources while spill tasks are executing (#32783)
* [chore](log) print query id before logging profile in be.INFO (#32922)
* [fix](grace-exit) Stop incorrectly of reportwork cause heap use after free #32929
* [improvement](decommission be) decommission check replica num (#32748)
* [fix](arrow-flight) Fix reach limit of connections error (#32911)
Fix Reach limit of connections error
in fe.conf , arrow_flight_token_cache_size is mandatory less than qe_max_connection/2. arrow flight sql is a stateless protocol, connection is usually not actively disconnected, bearer token is evict from the cache will unregister ConnectContext.
Fix ConnectContext.command not be reset to COM_SLEEP in time, this will result in frequent kill connection after query timeout.
Fix bearer token evict log and exception.
TODO: use arrow flight session: https://mail.google.com/mail/u/0/#inbox/FMfcgzGxRdxBLQLTcvvtRpqsvmhrHpdH
* [bugfix](cloud) few variable not initialized (#32868)
../../cloud/src/recycler/meta_checker.cpp
can cause uninitialised memory read.
* [fix](arrow-flight) Fix arrow flight sql compatible with JDK 17 and upgrade arrow 15.0.2 (#32796)
--add-opens=java.base/java.nio=ALL-UNNAMED, see: https://arrow.apache.org/docs/java/install.html#java-compatibility
groovy use flight sql connection to execute query SUM(MAX(c1) OVER (PARTITION BY)) report error: AGGREGATE clause must not contain analytic expressions, but no problem in Java execute it with jdbc::arrow-flight-sql.
groovy not support print arrow array type, throw IndexOutOfBoundsException.
"arrow_flight_sql" not support two phase read
./run-regression-test.sh --run --clean -g arrow_flight_sql
* [fix](spill) SpillStream's writer maybe may not have been finalized (#32931)
* [improvement](spill) Disable DistinctStreamingAgg when spill is enabled (#32932)
* [Improve](inverted_index) update clucene and improve array inverted index writer (#32436)
* [Performance](exec) replace SipHash in function by XXHash (#32919)
* [feature](agg) add aggregate function sum0 (#32541)
* [improvement](mtmv) Support to get tables in materialized view when collecting table in plan (#32797)
Support to get tables in materialized view when collecting table in plan
table scehma as fllowing:
create materialized view mv1
BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
DISTRIBUTED BY RANDOM BUCKETS 1
PROPERTIES ('replication_num' = '1')
as
select
t1.c1,
t3.c2
from
table1 t1
inner join table3 t3 on t1.c1 = t3.c2
if get table from the plan as follwoing, we can get [table1, table3, table2], the mv1 is expanded to get base tables;
SELECT
mv1.*,
uuid()
FROM
mv1 LEFT SEMI
JOIN table2 ON mv1.c1 = table2.c1
WHERE
mv1.c1 IN (
SELECT
c1
FROM
table2
)
OR mv1.c1 < 10
* [enhance](mtmv)support olap table partition column is null (#32698)
* [enhancement](cloud) add table version to cloud (#32738)
Add table version to cloud.
In Fe:
Get: If Fe is cloud mode, get table version from meta service.
Update: Op drop/replace temp partition, commit transaction.
In meta service:
Add: create Index. init value is 1.
Remove: by recycler.
Update: commit/drop partition rpc, commit txn rpc. Atomic++.
* [fix](cloud) schema change from not null to null (#32913)
1. Use equals instead of == for type comparing
2. null bitmap size is reisze by size of ref column.
* [feature](Nereids): add ColumnPruningPostProcessor. (#32800)
* [case](rowpolicy)fix row policy has been exist (#32880)
* [fix](pipeline) fix use error row desc when origin block clear (#32803)
* [fix](Nereids) support variant column with index when create table (#32948)
* [opt](Nereids) support create table with variant type (#32953)
* [test](insert-overwrite) Add insert overwrite auto detect concurrency cases (#32935)
* [fix](compile) fe cannot compile in idea (#32955)
* [enhancement](plsql) Support select * from routines (#32866)
Support show of plsql procedure using select * from routines.
* [fix](trino-connector) fix `NoClassDefFoundError` of hudi `Utils` class (#32846)
Due to the change of this PR #32455 , the `trino-connector-scanner` package cannot access the `hudi_scanner` package, so the exception NoclassDeffounderror will appear.
We need to write a separate Utils class.
* [exec](column) change some complex column move to noexcept (#32954)
* [Enhancement](data skew) extends show data skew (#32732)
* [chore](test) let suite compatible with Nereids (#32964)
* Support identical column name in different index. (#32792)
* Limit the max string length to 1024 while collecting column stats to control BE memory usage. (#32470)
* [fix](merge-iterator) fix NOT_IMPLEMENTED_ERROR when read next block view (#32961)
* [improvement](executor)Add tag property for workload group #32874
* [fix](auth)unified workload and resource permission logic (#32907)
- `Grant resource` can no longer grant global `usage_priv`
- `grant resource %` instead of `grant resource *`
before change:
```
grant usage_priv on resource * to f;
show grants for f\G
*************************** 1. row ***************************
UserIdentity: 'f'@'%'
Comment:
Password: No
Roles:
GlobalPrivs: Usage_priv
CatalogPrivs: NULL
DatabasePrivs: internal.information_schema: Select_priv ; internal.mysql: Select_priv
TablePrivs: NULL
ColPrivs: NULL
ResourcePrivs: NULL
CloudClusterPrivs: NULL
WorkloadGroupPrivs: normal: Usage_priv
```
after change
```
grant usage_priv on resource '%' to f;
show grants for f\G
*************************** 1. row ***************************
UserIdentity: 'f'@'%'
Comment:
Password: No
Roles:
GlobalPrivs: NULL
CatalogPrivs: NULL
DatabasePrivs: internal.information_schema: Select_priv ; internal.mysql: Select_priv
TablePrivs: NULL
ColPrivs: NULL
ResourcePrivs: %: Usage_priv
CloudClusterPrivs: NULL
WorkloadGroupPrivs: normal: Usage_priv
```
---------
Co-authored-by: yujun <yu...@gmail.com>
Co-authored-by: Gavin Chou <ga...@gmail.com>
Co-authored-by: xy720 <22...@users.noreply.github.com>
Co-authored-by: yongjinhou <10...@users.noreply.github.com>
Co-authored-by: Dongyang Li <he...@qq.com>
Co-authored-by: stephen <he...@qq.com>
Co-authored-by: morrySnow <10...@users.noreply.github.com>
Co-authored-by: seawinde <14...@users.noreply.github.com>
Co-authored-by: lihangyu <15...@163.com>
Co-authored-by: Yulei-Yang <yu...@gmail.com>
Co-authored-by: starocean999 <40...@users.noreply.github.com>
Co-authored-by: wangbo <wa...@apache.org>
Co-authored-by: Mingyu Chen <mo...@163.com>
Co-authored-by: Jerry Hu <mr...@gmail.com>
Co-authored-by: zhiqiang <se...@163.com>
Co-authored-by: Xinyi Zou <zo...@gmail.com>
Co-authored-by: Vallish Pai <va...@gmail.com>
Co-authored-by: amory <wa...@selectdb.com>
Co-authored-by: HappenLee <ha...@hotmail.com>
Co-authored-by: Jensen <cz...@163.com>
Co-authored-by: zhangdong <49...@qq.com>
Co-authored-by: Yongqiang YANG <98...@users.noreply.github.com>
Co-authored-by: jakevin <ja...@gmail.com>
Co-authored-by: Mryange <59...@users.noreply.github.com>
Co-authored-by: zclllyybb <zh...@selectdb.com>
Co-authored-by: Tiewei Fang <43...@users.noreply.github.com>
Co-authored-by: Xin Liao <li...@126.com>
---
.asf.yaml | 2 +-
be/src/agent/task_worker_pool.cpp | 3 +-
be/src/cloud/cloud_schema_change_job.cpp | 4 +
be/src/clucene | 2 +-
be/src/exec/exec_node.h | 6 +-
be/src/exec/schema_scanner.cpp | 3 +
...oups_scanner.cpp => schema_routine_scanner.cpp} | 103 +--
.../exec/schema_scanner/schema_routine_scanner.h | 52 ++
.../schema_workload_groups_scanner.cpp | 1 +
be/src/olap/hll.h | 4 +-
.../rowset/segment_v2/inverted_index_writer.cpp | 92 +-
be/src/olap/schema_change.cpp | 27 +-
be/src/olap/schema_change.h | 4 +-
be/src/pipeline/exec/join_probe_operator.cpp | 4 +-
.../exec/partitioned_aggregation_sink_operator.cpp | 14 +-
.../exec/partitioned_aggregation_sink_operator.h | 5 +
.../partitioned_aggregation_source_operator.cpp | 11 +-
.../exec/partitioned_aggregation_source_operator.h | 5 +
.../exec/partitioned_hash_join_probe_operator.cpp | 128 +--
.../exec/partitioned_hash_join_probe_operator.h | 5 +
.../exec/partitioned_hash_join_sink_operator.cpp | 9 +-
.../exec/partitioned_hash_join_sink_operator.h | 5 +
be/src/pipeline/exec/spill_sort_sink_operator.cpp | 11 +-
be/src/pipeline/exec/spill_sort_sink_operator.h | 5 +
.../pipeline/exec/spill_sort_source_operator.cpp | 10 +-
be/src/pipeline/exec/spill_sort_source_operator.h | 5 +
be/src/pipeline/pipeline_fragment_context.cpp | 10 +-
be/src/pipeline/pipeline_x/dependency.h | 33 +-
be/src/pipeline/pipeline_x/operator.cpp | 4 +-
be/src/pipeline/pipeline_x/operator.h | 4 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 6 +-
be/src/runtime/exec_env.h | 3 +
be/src/service/doris_main.cpp | 2 +
be/src/util/bitmap_value.h | 4 +-
be/src/util/quantile_state.h | 5 +
.../aggregate_function_simple_factory.cpp | 2 +
.../aggregate_functions/aggregate_function_sum.cpp | 4 +
.../aggregate_functions/aggregate_function_uniq.h | 10 +-
.../data_types/serde/data_type_object_serde.cpp | 23 +
.../vec/data_types/serde/data_type_object_serde.h | 5 +-
be/src/vec/exec/join/vjoin_node_base.cpp | 3 +-
be/src/vec/functions/uuid_numeric.cpp | 20 +-
be/src/vec/olap/vgeneric_iterators.cpp | 6 +-
.../segment_v2/inverted_index_array_test.cpp | 225 +++++
cloud/src/meta-service/http_encode_key.cpp | 3 +-
cloud/src/meta-service/keys.cpp | 27 +-
cloud/src/meta-service/keys.h | 13 +-
cloud/src/meta-service/meta_service.cpp | 70 +-
cloud/src/meta-service/meta_service_partition.cpp | 29 +-
cloud/src/meta-service/meta_service_txn.cpp | 15 +-
cloud/src/recycler/meta_checker.cpp | 4 +-
cloud/src/recycler/recycler.cpp | 43 +-
cloud/test/http_encode_key_test.cpp | 13 +-
cloud/test/keys_test.cpp | 50 +-
cloud/test/meta_service_test.cpp | 100 +++
cloud/test/recycler_test.cpp | 67 +-
conf/fe.conf | 2 +-
.../sql-reference/Show-Statements/SHOW-VIEWS.md | 78 ++
.../sql-reference/Show-Statements/SHOW-VIEWS.md | 78 ++
.../hudi-scanner/src/main/resources/package.xml | 20 +-
.../java-common/src/main/resources/package.xml | 20 +-
.../jdbc-scanner/src/main/resources/package.xml | 20 +-
.../src/main/resources/package.xml | 20 +-
.../paimon-scanner/src/main/resources/package.xml | 20 +-
.../src/main/resources/package-deps.xml | 5 +
.../trino-connector-scanner/pom.xml | 10 +-
.../apache/doris/trinoconnector/ProcessUtils.java | 74 ++
.../doris/trinoconnector/TrinoConnectorCache.java | 10 +-
.../src/main/resources/package.xml | 17 +-
.../main/java/org/apache/doris/common/Config.java | 12 +-
fe/fe-core/pom.xml | 10 +-
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 1 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 1 +
fe/fe-core/src/main/cup/sql_parser.cup | 14 +
.../org/apache/doris/alter/CloudRollupJobV2.java | 4 +-
.../apache/doris/alter/CloudSchemaChangeJobV2.java | 5 +-
.../org/apache/doris/alter/SchemaChangeJobV2.java | 2 +-
.../java/org/apache/doris/alter/SystemHandler.java | 84 +-
.../org/apache/doris/analysis/CreateRoleStmt.java | 7 +
.../org/apache/doris/analysis/CreateUserStmt.java | 8 +
.../doris/analysis/CreateWorkloadGroupStmt.java | 6 +
.../org/apache/doris/analysis/DropRoleStmt.java | 7 +
.../org/apache/doris/analysis/DropUserStmt.java | 7 +
.../java/org/apache/doris/analysis/GrantStmt.java | 6 +
.../org/apache/doris/analysis/ResourcePattern.java | 31 +-
.../java/org/apache/doris/analysis/RevokeStmt.java | 5 +
.../apache/doris/analysis/ShowColumnStatsStmt.java | 9 +-
.../apache/doris/analysis/ShowDataSkewStmt.java | 7 +-
.../org/apache/doris/analysis/ShowTableStmt.java | 34 +-
.../doris/catalog/BuiltinAggregateFunctions.java | 4 +
.../main/java/org/apache/doris/catalog/Env.java | 9 +-
.../java/org/apache/doris/catalog/FunctionSet.java | 109 ++-
.../org/apache/doris/catalog/MetadataViewer.java | 88 +-
.../java/org/apache/doris/catalog/OlapTable.java | 110 ++-
.../java/org/apache/doris/catalog/SchemaTable.java | 1 +
.../main/java/org/apache/doris/catalog/Table.java | 11 +-
.../java/org/apache/doris/catalog/TableIf.java | 7 +-
.../doris/cloud/catalog/CloudClusterChecker.java | 35 +-
.../org/apache/doris/cloud/catalog/CloudEnv.java | 6 +-
.../apache/doris/cloud/catalog/CloudPartition.java | 58 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 2 +-
.../cloud/datasource/CloudInternalCatalog.java | 20 +-
.../apache/doris/cloud/load/CloudLoadManager.java | 5 +-
.../org/apache/doris/cloud/qe/SnapshotProxy.java | 92 ++
.../doris/cloud/system/CloudSystemInfoService.java | 19 +-
.../doris/common/profile/SummaryProfile.java | 27 +-
.../org/apache/doris/datasource/ExternalTable.java | 26 +-
.../apache/doris/datasource/InternalCatalog.java | 13 +-
.../iceberg/IcebergHMSExternalCatalog.java | 20 +-
.../datasource/property/PropertyConverter.java | 10 +-
.../insertoverwrite/InsertOverwriteManager.java | 2 +
.../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 6 +-
.../doris/mysql/privilege/ResourcePrivEntry.java | 9 +-
.../doris/mysql/privilege/ResourcePrivTable.java | 19 +-
.../org/apache/doris/mysql/privilege/Role.java | 23 +-
.../mysql/privilege/WorkloadGroupPrivTable.java | 15 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 2 +-
.../processor/post/ColumnPruningPostProcessor.java | 102 +++
.../nereids/processor/post/PlanPostProcessors.java | 1 +
.../rules/analysis/EliminateGroupByConstant.java | 8 +-
.../mv/AbstractMaterializedViewRule.java | 12 +-
.../rules/exploration/mv/ComparisonResult.java | 16 +-
.../rules/exploration/mv/HyperGraphComparator.java | 21 +-
.../mv/InitMaterializationContextHook.java | 4 +-
.../functions/agg/MultiDistinctSum0.java | 82 ++
.../trees/expressions/functions/agg/Sum0.java | 125 +++
.../visitor/AggregateFunctionVisitor.java | 10 +
.../trees/plans/commands/info/CreateMTMVInfo.java | 4 +-
.../trees/plans/commands/info/IndexDefinition.java | 4 +-
.../trees/plans/logical/LogicalTestScan.java | 6 +
.../trees/plans/physical/AbstractPhysicalPlan.java | 2 +-
.../trees/plans/visitor/RelationVisitor.java | 43 +-
.../trees/plans/visitor/TableCollector.java | 75 +-
.../org/apache/doris/nereids/types/DataType.java | 3 +
.../apache/doris/plsql/metastore/PlsqlManager.java | 16 +-
.../java/org/apache/doris/qe/ShowExecutor.java | 40 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 3 +-
.../resource/workloadgroup/WorkloadGroup.java | 47 +-
.../resource/workloadgroup/WorkloadGroupMgr.java | 59 +-
.../apache/doris/service/FrontendServiceImpl.java | 75 +-
.../arrowflight/DorisFlightSqlProducer.java | 8 +-
.../service/arrowflight/DorisFlightSqlService.java | 6 +-
.../auth2/FlightBearerTokenAuthenticator.java | 2 +-
.../sessions/FlightSessionsWithTokenManager.java | 46 +-
.../arrowflight/tokens/FlightTokenManagerImpl.java | 47 +-
.../org/apache/doris/statistics/AnalysisInfo.java | 20 +-
.../doris/statistics/AnalysisInfoBuilder.java | 12 +-
.../org/apache/doris/statistics/AnalysisJob.java | 4 +-
.../apache/doris/statistics/AnalysisManager.java | 143 +---
.../apache/doris/statistics/BaseAnalysisTask.java | 24 +
.../doris/statistics/ExternalAnalysisTask.java | 6 +-
.../apache/doris/statistics/OlapAnalysisTask.java | 15 +-
.../doris/statistics/StatisticsAutoCollector.java | 40 +-
.../doris/statistics/StatisticsCollector.java | 2 +-
.../doris/statistics/StatisticsRepository.java | 3 +-
.../apache/doris/statistics/TableStatsMeta.java | 76 +-
.../doris/tablefunction/MetadataGenerator.java | 105 ++-
.../doris/transaction/PublishVersionDaemon.java | 8 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 1 +
.../org/apache/doris/analysis/GrantStmtTest.java | 4 +-
.../org/apache/doris/analysis/ShowReplicaTest.java | 6 +-
.../apache/doris/analysis/ShowTableStmtTest.java | 23 +
.../org/apache/doris/mysql/privilege/AuthTest.java | 55 +-
.../ColumnPruningPostProcessorTest.java | 62 ++
.../exploration/mv/MaterializedViewUtilsTest.java | 2 +-
.../doris/nereids/trees/plans/PlanVisitorTest.java | 108 ++-
.../java/org/apache/doris/qe/ShowExecutorTest.java | 10 +
.../doris/statistics/AnalysisManagerTest.java | 65 +-
.../doris/statistics/AnalysisTaskExecutorTest.java | 11 +-
.../org/apache/doris/statistics/AnalyzeTest.java | 11 +-
.../apache/doris/statistics/HistogramTaskTest.java | 1 -
.../doris/statistics/OlapAnalysisTaskTest.java | 12 +-
.../statistics/StatisticsAutoCollectorTest.java | 121 +--
.../doris/statistics/TableStatsMetaTest.java | 4 +-
fe/pom.xml | 9 +-
gensrc/proto/cloud.proto | 5 +
gensrc/thrift/FrontendService.thrift | 1 +
.../data/correctness_p0/test_probe_clean.out | 10 +
.../external_table_p2/hive/test_complex_types.out | 8 +-
.../hive/test_external_credit_data.out | 8 +-
.../external_table_p2/hive/test_hive_to_array.out | 8 +-
.../insert_overwrite_auto_detect.out | 13 +
.../test_iot_auto_detect_concurrent.out | 10 +
.../data/mtmv_p0/test_null_partition_mtmv.out | 19 +
.../data/nereids_function_p0/agg_function/agg.out | 926 ++++++++++++++++-----
.../data/nereids_hint_tpch_p0/shape/q15.out | 39 +-
.../data/nereids_p0/hint/test_distribute.out | 120 +--
.../data/nereids_ssb_shape_sf100_p0/shape/q4.3.out | 21 +-
.../nereids_tpcds_shape_sf1000_p0/shape/query1.out | 33 +-
.../shape/query11.out | 23 +-
.../shape/query15.out | 13 +-
.../shape/query16.out | 37 +-
.../shape/query21.out | 13 +-
.../shape/query26.out | 37 +-
.../shape/query29.out | 47 +-
.../nereids_tpcds_shape_sf1000_p0/shape/query4.out | 44 +-
.../shape/query40.out | 21 +-
.../shape/query42.out | 13 +-
.../shape/query44.out | 57 +-
.../shape/query48.out | 15 +-
.../shape/query58.out | 109 +--
.../shape/query62.out | 19 +-
.../nereids_tpcds_shape_sf1000_p0/shape/query7.out | 37 +-
.../shape/query72.out | 77 +-
.../shape/query74.out | 23 +-
.../shape/query76.out | 43 +-
.../shape/query83.out | 97 +--
.../shape/query93.out | 13 +-
.../shape/query94.out | 37 +-
.../shape/query95.out | 37 +-
.../shape/query99.out | 19 +-
.../noStatsRfPrune/query16.out | 41 +-
.../noStatsRfPrune/query2.out | 29 +-
.../noStatsRfPrune/query23.out | 46 +-
.../noStatsRfPrune/query25.out | 67 +-
.../noStatsRfPrune/query29.out | 46 +-
.../noStatsRfPrune/query30.out | 29 +-
.../noStatsRfPrune/query40.out | 21 +-
.../noStatsRfPrune/query42.out | 13 +-
.../noStatsRfPrune/query62.out | 19 +-
.../noStatsRfPrune/query75.out | 63 +-
.../noStatsRfPrune/query81.out | 29 +-
.../noStatsRfPrune/query94.out | 41 +-
.../noStatsRfPrune/query95.out | 45 +-
.../noStatsRfPrune/query99.out | 19 +-
.../no_stats_shape/query16.out | 41 +-
.../no_stats_shape/query2.out | 29 +-
.../no_stats_shape/query23.out | 46 +-
.../no_stats_shape/query25.out | 67 +-
.../no_stats_shape/query29.out | 46 +-
.../no_stats_shape/query30.out | 29 +-
.../no_stats_shape/query40.out | 21 +-
.../no_stats_shape/query42.out | 13 +-
.../no_stats_shape/query62.out | 19 +-
.../no_stats_shape/query75.out | 63 +-
.../no_stats_shape/query81.out | 29 +-
.../no_stats_shape/query94.out | 41 +-
.../no_stats_shape/query95.out | 45 +-
.../no_stats_shape/query99.out | 19 +-
.../rf_prune/query1.out | 33 +-
.../rf_prune/query11.out | 23 +-
.../rf_prune/query15.out | 13 +-
.../rf_prune/query16.out | 41 +-
.../rf_prune/query21.out | 13 +-
.../rf_prune/query26.out | 37 +-
.../rf_prune/query29.out | 47 +-
.../rf_prune/query4.out | 44 +-
.../rf_prune/query40.out | 21 +-
.../rf_prune/query42.out | 13 +-
.../rf_prune/query44.out | 57 +-
.../rf_prune/query46.out | 15 +-
.../rf_prune/query48.out | 15 +-
.../rf_prune/query58.out | 109 +--
.../rf_prune/query62.out | 19 +-
.../rf_prune/query7.out | 37 +-
.../rf_prune/query74.out | 23 +-
.../rf_prune/query76.out | 39 +-
.../rf_prune/query83.out | 95 +--
.../rf_prune/query93.out | 13 +-
.../rf_prune/query94.out | 37 +-
.../rf_prune/query95.out | 37 +-
.../rf_prune/query99.out | 19 +-
.../nereids_tpcds_shape_sf100_p0/shape/query1.out | 33 +-
.../nereids_tpcds_shape_sf100_p0/shape/query11.out | 23 +-
.../nereids_tpcds_shape_sf100_p0/shape/query15.out | 13 +-
.../nereids_tpcds_shape_sf100_p0/shape/query16.out | 41 +-
.../nereids_tpcds_shape_sf100_p0/shape/query21.out | 13 +-
.../nereids_tpcds_shape_sf100_p0/shape/query26.out | 37 +-
.../nereids_tpcds_shape_sf100_p0/shape/query29.out | 47 +-
.../nereids_tpcds_shape_sf100_p0/shape/query4.out | 44 +-
.../nereids_tpcds_shape_sf100_p0/shape/query40.out | 21 +-
.../nereids_tpcds_shape_sf100_p0/shape/query42.out | 13 +-
.../nereids_tpcds_shape_sf100_p0/shape/query44.out | 57 +-
.../nereids_tpcds_shape_sf100_p0/shape/query46.out | 15 +-
.../nereids_tpcds_shape_sf100_p0/shape/query48.out | 15 +-
.../nereids_tpcds_shape_sf100_p0/shape/query58.out | 109 +--
.../nereids_tpcds_shape_sf100_p0/shape/query62.out | 19 +-
.../nereids_tpcds_shape_sf100_p0/shape/query7.out | 37 +-
.../nereids_tpcds_shape_sf100_p0/shape/query74.out | 23 +-
.../nereids_tpcds_shape_sf100_p0/shape/query76.out | 39 +-
.../nereids_tpcds_shape_sf100_p0/shape/query83.out | 95 +--
.../nereids_tpcds_shape_sf100_p0/shape/query93.out | 13 +-
.../nereids_tpcds_shape_sf100_p0/shape/query94.out | 37 +-
.../nereids_tpcds_shape_sf100_p0/shape/query95.out | 37 +-
.../nereids_tpcds_shape_sf100_p0/shape/query99.out | 19 +-
.../nostats_rf_prune/q15.out | 39 +-
.../nostats_rf_prune/q22.out | 23 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q15.out | 39 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q16.out | 13 +-
.../rf_prune/q20-rewrite.out | 13 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q20.out | 13 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q21.out | 35 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q22.out | 23 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q5.out | 21 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q7.out | 13 +-
.../nereids_tpch_shape_sf1000_p0/rf_prune/q9.out | 25 +-
.../nereids_tpch_shape_sf1000_p0/shape/q15.out | 39 +-
.../nereids_tpch_shape_sf1000_p0/shape/q16.out | 13 +-
.../shape/q20-rewrite.out | 13 +-
.../nereids_tpch_shape_sf1000_p0/shape/q20.out | 13 +-
.../nereids_tpch_shape_sf1000_p0/shape/q21.out | 35 +-
.../nereids_tpch_shape_sf1000_p0/shape/q22.out | 23 +-
.../data/nereids_tpch_shape_sf1000_p0/shape/q5.out | 21 +-
.../data/nereids_tpch_shape_sf1000_p0/shape/q7.out | 13 +-
.../data/nereids_tpch_shape_sf1000_p0/shape/q9.out | 25 +-
.../shape_no_stats/q15.out | 39 +-
.../shape_no_stats/q22.out | 23 +-
.../data/workload_manager_p0/test_curd_wlg.out | 24 +-
regression-test/framework/pom.xml | 2 +-
.../regression/action/StreamLoadAction.groovy | 31 +-
.../cloud_p0/conf/regression-conf-custom.groovy | 1 -
.../pipeline/external/conf/regression-conf.groovy | 5 -
.../pipeline/p0/conf/regression-conf.groovy | 2 -
.../account_p0/test_nereids_row_policy.groovy | 9 +
.../test_decommission_with_replica_num_fail.groovy | 59 ++
.../recover_with_schema_change.groovy | 2 +-
.../suites/correctness_p0/test_probe_clean.groovy | 95 +++
.../hive/test_external_credit_data.groovy | 2 +-
.../hive/test_hive_to_array.groovy | 6 +-
.../insert_overwrite_auto_detect.groovy | 47 ++
.../ddl/test_iot_auto_detect_concurrent.sql | 108 +++
.../test_iot_auto_detect_concurrent.groovy | 124 +++
.../suites/mtmv_p0/test_null_partition_mtmv.groovy | 169 ++++
.../nereids_function_p0/agg_function/agg.groovy | 185 ++++
.../aggregate/aggregate_groupby_constant.groovy | 50 ++
.../mv/dimension/dimension_1.groovy | 264 +++---
...ion_2_2.groovy => dimension_2_full_join.groovy} | 143 ++--
...on_2_2.groovy => dimension_2_inner_join.groovy} | 0
..._1.groovy => dimension_2_left_anti_join.groovy} | 236 ++----
...ion_2_1.groovy => dimension_2_left_join.groovy} | 4 +-
..._1.groovy => dimension_2_left_semi_join.groovy} | 236 ++----
...1.groovy => dimension_2_right_anti_join.groovy} | 242 ++----
...on_2_1.groovy => dimension_2_right_join.groovy} | 139 ++--
...1.groovy => dimension_2_right_semi_join.groovy} | 242 ++----
.../dimension_2_join_agg.groovy | 26 +-
.../nereids_rules_p0/mv/tpch/mv_tpch_test.groovy | 4 +-
regression-test/suites/node_p0/test_backend.groovy | 26 +-
.../schema_table/test_active_queries.groovy | 2 +-
.../schema_table/test_backend_active_tasks.groovy | 2 +-
.../test_multi_string_position.groovy | 8 +-
.../test_multi_string_search.groovy | 8 +-
.../query_p0/sql_functions/test_in_expr.groovy | 2 +-
.../window_functions/test_window_fn.groovy | 4 +-
.../test_two_phase_read_with_having.groovy | 2 +-
.../test_alter_table_replace.groovy | 2 +-
.../suites/show_p0/test_show_data_skew.groovy | 36 +-
.../suites/statistics/analyze_stats.groovy | 4 +-
.../suites/statistics/test_analyze_mtmv.groovy | 40 +-
.../suites/statistics/test_analyze_mv.groovy | 38 +-
...artial_update_insert_light_schema_change.groovy | 2 +-
.../workload_manager_p0/test_curd_wlg.groovy | 92 +-
run-regression-test.sh | 1 +
352 files changed, 7838 insertions(+), 4605 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 5aa08854a67..0a2e83fcd51 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -109,8 +109,8 @@ github:
- shuke987
- wm1581066
- KassieZ
- - yujun777
- gavinchou
+ - yujun777
notifications:
pullrequests_status: commits@doris.apache.org
diff --git a/be/src/agent/task_worker_pool.cpp b/be/src/agent/task_worker_pool.cpp
index b9a55518c46..183a5744aae 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -190,7 +190,8 @@ void alter_tablet(StorageEngine& engine, const TAgentTaskRequest& agent_task_req
Status res = Status::OK();
try {
DCHECK(agent_task_req.alter_tablet_req_v2.__isset.job_id);
- SchemaChangeJob job(engine, agent_task_req.alter_tablet_req_v2);
+ SchemaChangeJob job(engine, agent_task_req.alter_tablet_req_v2,
+ std::to_string(agent_task_req.alter_tablet_req_v2.job_id));
status = job.process_alter_tablet(agent_task_req.alter_tablet_req_v2);
} catch (const Exception& e) {
status = e.to_status();
diff --git a/be/src/cloud/cloud_schema_change_job.cpp b/be/src/cloud/cloud_schema_change_job.cpp
index e919e3f37aa..d1d323e19a9 100644
--- a/be/src/cloud/cloud_schema_change_job.cpp
+++ b/be/src/cloud/cloud_schema_change_job.cpp
@@ -201,6 +201,10 @@ Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParam
"Don't support to add materialized view by linked schema change");
}
+ LOG(INFO) << "schema change type, sc_sorting: " << sc_sorting
+ << ", sc_directly: " << sc_directly << ", base_tablet=" << _base_tablet->tablet_id()
+ << ", new_tablet=" << _new_tablet->tablet_id();
+
// 2. Generate historical data converter
auto sc_procedure = get_sc_procedure(changer, sc_sorting);
diff --git a/be/src/clucene b/be/src/clucene
index ef95e67ae31..ff2cd82f9e5 160000
--- a/be/src/clucene
+++ b/be/src/clucene
@@ -1 +1 @@
-Subproject commit ef95e67ae3123409f006072194f742a079603159
+Subproject commit ff2cd82f9e545a24318f1256eba312b4d0562a82
diff --git a/be/src/exec/exec_node.h b/be/src/exec/exec_node.h
index 5d7b3a91651..f2303068437 100644
--- a/be/src/exec/exec_node.h
+++ b/be/src/exec/exec_node.h
@@ -127,7 +127,7 @@ public:
bool has_output_row_descriptor() const { return _output_row_descriptor != nullptr; }
// If use projection, we should clear `_origin_block`.
void clear_origin_block() {
- _origin_block.clear_column_data(_row_descriptor.num_materialized_slots());
+ _origin_block.clear_column_data(intermediate_row_desc().num_materialized_slots());
}
// Emit data, both need impl with method: sink
@@ -326,8 +326,8 @@ protected:
std::shared_ptr<QueryStatistics> _query_statistics = nullptr;
//_keep_origin is used to avoid copying during projection,
- // currently set to true only in the nestloop join.
- bool _keep_origin = false;
+ // currently set to false only in the nestloop join.
+ bool _keep_origin = true;
private:
static Status create_tree_helper(RuntimeState* state, ObjectPool* pool,
diff --git a/be/src/exec/schema_scanner.cpp b/be/src/exec/schema_scanner.cpp
index 32197e37e1e..908bb7b6f09 100644
--- a/be/src/exec/schema_scanner.cpp
+++ b/be/src/exec/schema_scanner.cpp
@@ -37,6 +37,7 @@
#include "exec/schema_scanner/schema_partitions_scanner.h"
#include "exec/schema_scanner/schema_processlist_scanner.h"
#include "exec/schema_scanner/schema_profiling_scanner.h"
+#include "exec/schema_scanner/schema_routine_scanner.h"
#include "exec/schema_scanner/schema_rowsets_scanner.h"
#include "exec/schema_scanner/schema_schema_privileges_scanner.h"
#include "exec/schema_scanner/schema_schemata_scanner.h"
@@ -161,6 +162,8 @@ std::unique_ptr<SchemaScanner> SchemaScanner::create(TSchemaTableType::type type
return SchemaWorkloadGroupsScanner::create_unique();
case TSchemaTableType::SCH_PROCESSLIST:
return SchemaProcessListScanner::create_unique();
+ case TSchemaTableType::SCH_PROCEDURES:
+ return SchemaRoutinesScanner::create_unique();
default:
return SchemaDummyScanner::create_unique();
break;
diff --git a/be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp b/be/src/exec/schema_scanner/schema_routine_scanner.cpp
similarity index 59%
copy from be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp
copy to be/src/exec/schema_scanner/schema_routine_scanner.cpp
index 24e23a3e336..7db46ada650 100644
--- a/be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp
+++ b/be/src/exec/schema_scanner/schema_routine_scanner.cpp
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-#include "exec/schema_scanner/schema_workload_groups_scanner.h"
+#include "exec/schema_scanner/schema_routine_scanner.h"
#include "runtime/client_cache.h"
#include "runtime/exec_env.h"
@@ -26,83 +26,79 @@
#include "vec/data_types/data_type_factory.hpp"
namespace doris {
-std::vector<SchemaScanner::ColumnDesc> SchemaWorkloadGroupsScanner::_s_tbls_columns = {
- {"ID", TYPE_BIGINT, sizeof(int64_t), true},
- {"NAME", TYPE_VARCHAR, sizeof(StringRef), true},
- {"CPU_SHARE", TYPE_BIGINT, sizeof(int64_t), true},
- {"MEMORY_LIMIT", TYPE_VARCHAR, sizeof(StringRef), true},
- {"ENABLE_MEMORY_OVERCOMMIT", TYPE_VARCHAR, sizeof(StringRef), true},
- {"MAX_CONCURRENCY", TYPE_BIGINT, sizeof(int64_t), true},
- {"MAX_QUEUE_SIZE", TYPE_BIGINT, sizeof(int64_t), true},
- {"QUEUE_TIMEOUT", TYPE_BIGINT, sizeof(int64_t), true},
- {"CPU_HARD_LIMIT", TYPE_VARCHAR, sizeof(StringRef), true},
- {"SCAN_THREAD_NUM", TYPE_BIGINT, sizeof(int64_t), true},
- {"MAX_REMOTE_SCAN_THREAD_NUM", TYPE_BIGINT, sizeof(int64_t), true},
- {"MIN_REMOTE_SCAN_THREAD_NUM", TYPE_BIGINT, sizeof(int64_t), true},
- {"SPILL_THRESHOLD_LOW_WATERMARK", TYPE_VARCHAR, sizeof(StringRef), true},
- {"SPILL_THRESHOLD_HIGH_WATERMARK", TYPE_VARCHAR, sizeof(StringRef), true},
+std::vector<SchemaScanner::ColumnDesc> SchemaRoutinesScanner::_s_tbls_columns = {
+ {"SPECIFIC_NAME", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_CATALOG", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_SCHEMA", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_NAME", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_TYPE", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"DTD_IDENTIFIER", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_BODY", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_DEFINITION", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"EXTERNAL_NAME", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"EXTERNAL_LANGUAGE", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"PARAMETER_STYLE", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"IS_DETERMINISTIC", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"SQL_DATA_ACCESS", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"SQL_PATH", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"SECURITY_TYPE", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"CREATED", TYPE_DATETIME, sizeof(int64_t), true},
+ {"LAST_ALTERED", TYPE_DATETIME, sizeof(int64_t), true},
+ {"SQL_MODE", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"ROUTINE_COMMENT", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"DEFINER", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"CHARACTER_SET_CLIENT", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"COLLATION_CONNECTION", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"DATABASE_COLLATION", TYPE_VARCHAR, sizeof(StringRef), true},
};
-SchemaWorkloadGroupsScanner::SchemaWorkloadGroupsScanner()
- : SchemaScanner(_s_tbls_columns, TSchemaTableType::SCH_WORKLOAD_GROUPS) {}
+SchemaRoutinesScanner::SchemaRoutinesScanner()
+ : SchemaScanner(_s_tbls_columns, TSchemaTableType::SCH_PROCEDURES) {}
-SchemaWorkloadGroupsScanner::~SchemaWorkloadGroupsScanner() {}
-
-Status SchemaWorkloadGroupsScanner::start(RuntimeState* state) {
+Status SchemaRoutinesScanner::start(RuntimeState* state) {
_block_rows_limit = state->batch_size();
_rpc_timeout = state->execution_timeout() * 1000;
return Status::OK();
}
-Status SchemaWorkloadGroupsScanner::_get_workload_groups_block_from_fe() {
+Status SchemaRoutinesScanner::get_block_from_fe() {
TNetworkAddress master_addr = ExecEnv::GetInstance()->master_info()->network_address;
-
TSchemaTableRequestParams schema_table_request_params;
for (int i = 0; i < _s_tbls_columns.size(); i++) {
schema_table_request_params.__isset.columns_name = true;
schema_table_request_params.columns_name.emplace_back(_s_tbls_columns[i].name);
}
schema_table_request_params.__set_current_user_ident(*_param->common_param->current_user_ident);
-
TFetchSchemaTableDataRequest request;
- request.__set_schema_table_name(TSchemaTableName::WORKLOAD_GROUPS);
+ request.__set_schema_table_name(TSchemaTableName::ROUTINES_INFO);
request.__set_schema_table_params(schema_table_request_params);
-
TFetchSchemaTableDataResult result;
-
RETURN_IF_ERROR(ThriftRpcHelper::rpc<FrontendServiceClient>(
master_addr.hostname, master_addr.port,
[&request, &result](FrontendServiceConnection& client) {
client->fetchSchemaTableData(result, request);
},
_rpc_timeout));
-
Status status(Status::create(result.status));
if (!status.ok()) {
- LOG(WARNING) << "fetch workload groups from FE failed, errmsg=" << status;
+ LOG(WARNING) << "fetch routines from FE failed, errmsg=" << status;
return status;
}
std::vector<TRow> result_data = result.data_batch;
-
- _workload_groups_block = vectorized::Block::create_unique();
+ _routines_block = vectorized::Block::create_unique();
for (int i = 0; i < _s_tbls_columns.size(); ++i) {
TypeDescriptor descriptor(_s_tbls_columns[i].type);
auto data_type = vectorized::DataTypeFactory::instance().create_data_type(descriptor, true);
- _workload_groups_block->insert(vectorized::ColumnWithTypeAndName(
+ _routines_block->insert(vectorized::ColumnWithTypeAndName(
data_type->create_column(), data_type, _s_tbls_columns[i].name));
}
-
- _workload_groups_block->reserve(_block_rows_limit);
-
+ _routines_block->reserve(_block_rows_limit);
if (result_data.size() > 0) {
int col_size = result_data[0].column_value.size();
if (col_size != _s_tbls_columns.size()) {
- return Status::InternalError<false>(
- "workload groups schema is not match for FE and BE");
+ return Status::InternalError<false>("routine table schema is not match for FE and BE");
}
}
-
- // todo(wb) reuse this callback function
auto insert_string_value = [&](int col_index, std::string str_val, vectorized::Block* block) {
vectorized::MutableColumnPtr mutable_col_ptr;
mutable_col_ptr = std::move(*block->get_by_position(col_index).column).assume_mutable();
@@ -113,14 +109,16 @@ Status SchemaWorkloadGroupsScanner::_get_workload_groups_block_from_fe() {
str_val.size());
nullable_column->get_null_map_data().emplace_back(0);
};
- auto insert_int_value = [&](int col_index, int64_t int_val, vectorized::Block* block) {
+ auto insert_datetime_value = [&](int col_index, const std::vector<void*>& datas,
+ vectorized::Block* block) {
vectorized::MutableColumnPtr mutable_col_ptr;
mutable_col_ptr = std::move(*block->get_by_position(col_index).column).assume_mutable();
auto* nullable_column =
reinterpret_cast<vectorized::ColumnNullable*>(mutable_col_ptr.get());
vectorized::IColumn* col_ptr = &nullable_column->get_nested_column();
- reinterpret_cast<vectorized::ColumnVector<vectorized::Int64>*>(col_ptr)->insert_value(
- int_val);
+ auto data = datas[0];
+ reinterpret_cast<vectorized::ColumnVector<vectorized::Int64>*>(col_ptr)->insert_data(
+ reinterpret_cast<char*>(data), 0);
nullable_column->get_null_map_data().emplace_back(0);
};
@@ -128,17 +126,22 @@ Status SchemaWorkloadGroupsScanner::_get_workload_groups_block_from_fe() {
TRow row = result_data[i];
for (int j = 0; j < _s_tbls_columns.size(); j++) {
- if (_s_tbls_columns[j].type == TYPE_BIGINT) {
- insert_int_value(j, row.column_value[j].longVal, _workload_groups_block.get());
+ if (_s_tbls_columns[j].type == TYPE_DATETIME) {
+ std::vector<void*> datas(1);
+ VecDateTimeValue src[1];
+ src[0].from_date_str(row.column_value[j].stringVal.data(),
+ row.column_value[j].stringVal.size());
+ datas[0] = src;
+ insert_datetime_value(j, datas, _routines_block.get());
} else {
- insert_string_value(j, row.column_value[j].stringVal, _workload_groups_block.get());
+ insert_string_value(j, row.column_value[j].stringVal, _routines_block.get());
}
}
}
return Status::OK();
}
-Status SchemaWorkloadGroupsScanner::get_next_block(vectorized::Block* block, bool* eos) {
+Status SchemaRoutinesScanner::get_next_block(vectorized::Block* block, bool* eos) {
if (!_is_init) {
return Status::InternalError("Used before initialized.");
}
@@ -147,9 +150,9 @@ Status SchemaWorkloadGroupsScanner::get_next_block(vectorized::Block* block, boo
return Status::InternalError("input pointer is nullptr.");
}
- if (_workload_groups_block == nullptr) {
- RETURN_IF_ERROR(_get_workload_groups_block_from_fe());
- _total_rows = _workload_groups_block->rows();
+ if (_routines_block == nullptr) {
+ RETURN_IF_ERROR(get_block_from_fe());
+ _total_rows = _routines_block->rows();
}
if (_row_idx == _total_rows) {
@@ -159,7 +162,7 @@ Status SchemaWorkloadGroupsScanner::get_next_block(vectorized::Block* block, boo
int current_batch_rows = std::min(_block_rows_limit, _total_rows - _row_idx);
vectorized::MutableBlock mblock = vectorized::MutableBlock::build_mutable_block(block);
- mblock.add_rows(_workload_groups_block.get(), _row_idx, current_batch_rows);
+ mblock.add_rows(_routines_block.get(), _row_idx, current_batch_rows);
_row_idx += current_batch_rows;
*eos = _row_idx == _total_rows;
diff --git a/be/src/exec/schema_scanner/schema_routine_scanner.h b/be/src/exec/schema_scanner/schema_routine_scanner.h
new file mode 100644
index 00000000000..543f9e8e8f6
--- /dev/null
+++ b/be/src/exec/schema_scanner/schema_routine_scanner.h
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <vector>
+
+#include "common/status.h"
+#include "exec/schema_scanner.h"
+
+namespace doris {
+class RuntimeState;
+namespace vectorized {
+class Block;
+} // namespace vectorized
+
+class SchemaRoutinesScanner : public SchemaScanner {
+ ENABLE_FACTORY_CREATOR(SchemaRoutinesScanner);
+
+public:
+ SchemaRoutinesScanner();
+ ~SchemaRoutinesScanner() override = default;
+
+ Status start(RuntimeState* state) override;
+ Status get_next_block(vectorized::Block* block, bool* eos) override;
+
+ static std::vector<SchemaScanner::ColumnDesc> _s_tbls_columns;
+
+private:
+ Status get_block_from_fe();
+
+ int _block_rows_limit = 4096;
+ int _row_idx = 0;
+ int _total_rows = 0;
+ std::unique_ptr<vectorized::Block> _routines_block = nullptr;
+ int _rpc_timeout = 3000;
+};
+}; // namespace doris
diff --git a/be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp b/be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp
index 24e23a3e336..03bf9782dcd 100644
--- a/be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp
+++ b/be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp
@@ -41,6 +41,7 @@ std::vector<SchemaScanner::ColumnDesc> SchemaWorkloadGroupsScanner::_s_tbls_colu
{"MIN_REMOTE_SCAN_THREAD_NUM", TYPE_BIGINT, sizeof(int64_t), true},
{"SPILL_THRESHOLD_LOW_WATERMARK", TYPE_VARCHAR, sizeof(StringRef), true},
{"SPILL_THRESHOLD_HIGH_WATERMARK", TYPE_VARCHAR, sizeof(StringRef), true},
+ {"TAG", TYPE_VARCHAR, sizeof(StringRef), true},
};
SchemaWorkloadGroupsScanner::SchemaWorkloadGroupsScanner()
diff --git a/be/src/olap/hll.h b/be/src/olap/hll.h
index 1f97f30ae8b..0ae5539163c 100644
--- a/be/src/olap/hll.h
+++ b/be/src/olap/hll.h
@@ -109,7 +109,7 @@ public:
}
}
- HyperLogLog(HyperLogLog&& other) {
+ HyperLogLog(HyperLogLog&& other) noexcept {
this->_type = other._type;
switch (other._type) {
case HLL_DATA_EMPTY:
@@ -131,7 +131,7 @@ public:
}
}
- HyperLogLog& operator=(HyperLogLog&& other) {
+ HyperLogLog& operator=(HyperLogLog&& other) noexcept {
if (this != &other) {
if (_registers != nullptr) {
delete[] _registers;
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp b/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp
index 078668cb0cf..87827464f15 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp
@@ -80,8 +80,11 @@ public:
explicit InvertedIndexColumnWriterImpl(const std::string& field_name,
InvertedIndexFileWriter* index_file_writer,
- const TabletIndex* index_meta)
- : _index_meta(index_meta), _index_file_writer(index_file_writer) {
+ const TabletIndex* index_meta,
+ const bool single_field = true)
+ : _single_field(single_field),
+ _index_meta(index_meta),
+ _index_file_writer(index_file_writer) {
_parser_type = get_inverted_index_parser_type_from_string(
get_parser_string_from_properties(_index_meta->properties()));
_value_key_coder = get_key_coder(field_type);
@@ -237,9 +240,14 @@ public:
RETURN_IF_ERROR(create_char_string_reader(_char_string_reader));
RETURN_IF_ERROR(create_analyzer(_analyzer));
RETURN_IF_ERROR(create_index_writer(_index_writer));
- RETURN_IF_ERROR(create_field(&_field));
_doc = std::make_unique<lucene::document::Document>();
- _doc->add(*_field);
+ if (_single_field) {
+ RETURN_IF_ERROR(create_field(&_field));
+ _doc->add(*_field);
+ } else {
+ // array's inverted index do need create field first
+ _doc->setNeedResetFieldData(true);
+ }
return Status::OK();
}
@@ -282,11 +290,9 @@ public:
return Status::OK();
}
- void new_fulltext_field(const char* field_value_data, size_t field_value_size) {
- if (_parser_type == InvertedIndexParserType::PARSER_ENGLISH ||
- _parser_type == InvertedIndexParserType::PARSER_CHINESE ||
- _parser_type == InvertedIndexParserType::PARSER_UNICODE ||
- _parser_type == InvertedIndexParserType::PARSER_STANDARD) {
+ void new_inverted_index_field(const char* field_value_data, size_t field_value_size) {
+ if (_parser_type != InvertedIndexParserType::PARSER_UNKNOWN &&
+ _parser_type != InvertedIndexParserType::PARSER_NONE) {
new_char_token_stream(field_value_data, field_value_size, _field);
} else {
new_field_char_value(field_value_data, field_value_size, _field);
@@ -328,7 +334,7 @@ public:
(_parser_type != InvertedIndexParserType::PARSER_NONE && v->empty())) {
RETURN_IF_ERROR(add_null_document());
} else {
- new_fulltext_field(v->get_data(), v->get_size());
+ new_inverted_index_field(v->get_data(), v->get_size());
RETURN_IF_ERROR(add_document());
}
++v;
@@ -348,39 +354,58 @@ public:
}
const auto* offsets = reinterpret_cast<const uint64_t*>(offsets_ptr);
if constexpr (field_is_slice_type(field_type)) {
- if (_field == nullptr || _index_writer == nullptr) {
- LOG(ERROR) << "field or index writer is null in inverted index writer.";
- return Status::InternalError(
- "field or index writer is null in inverted index writer");
+ if (_index_writer == nullptr) {
+ LOG(ERROR) << "index writer is null in inverted index writer.";
+ return Status::InternalError("index writer is null in inverted index writer");
}
auto ignore_above_value =
get_parser_ignore_above_value_from_properties(_index_meta->properties());
auto ignore_above = std::stoi(ignore_above_value);
for (int i = 0; i < count; ++i) {
// offsets[i+1] is now row element count
- std::vector<std::string> strings;
// [0, 3, 6]
// [10,20,30] [20,30,40], [30,40,50]
auto start_off = offsets[i];
auto end_off = offsets[i + 1];
+ // TODO(Amory).later we use object pool to avoid field creation
+ lucene::document::Field* new_field = nullptr;
+ CL_NS(analysis)::TokenStream* ts = nullptr;
for (auto j = start_off; j < end_off; ++j) {
if (null_map[j] == 1) {
continue;
}
+ // now we temp create field . later make a pool
+ if (Status st = create_field(&new_field); st != Status::OK()) {
+ LOG(ERROR)
+ << "create field " << string(_field_name.begin(), _field_name.end())
+ << " error:" << st;
+ return st;
+ }
auto* v = (Slice*)((const uint8_t*)value_ptr + j * field_size);
- strings.emplace_back(v->get_data(), v->get_size());
- }
-
- auto value = join(strings, " ");
- // only ignore_above UNTOKENIZED strings and empty strings not tokenized
- if ((_parser_type == InvertedIndexParserType::PARSER_NONE &&
- value.length() > ignore_above) ||
- (_parser_type != InvertedIndexParserType::PARSER_NONE && value.empty())) {
- RETURN_IF_ERROR(add_null_document());
- } else {
- new_fulltext_field(value.c_str(), value.length());
- RETURN_IF_ERROR(add_document());
+ if ((_parser_type == InvertedIndexParserType::PARSER_NONE &&
+ v->get_size() > ignore_above) ||
+ (_parser_type != InvertedIndexParserType::PARSER_NONE && v->empty())) {
+ // is here a null value?
+ // TODO. Maybe here has performance problem for large size string.
+ continue;
+ } else {
+ if (_parser_type != InvertedIndexParserType::PARSER_UNKNOWN &&
+ _parser_type != InvertedIndexParserType::PARSER_NONE) {
+ // in this case stream need to delete after add_document, because the
+ // stream can not reuse for different field
+ _char_string_reader->init(v->get_data(), v->get_size(), false);
+ ts = _analyzer->tokenStream(new_field->name(),
+ _char_string_reader.get());
+ new_field->setValue(ts);
+ } else {
+ new_field_char_value(v->get_data(), v->get_size(), new_field);
+ }
+ _doc->add(*new_field);
+ }
}
+ RETURN_IF_ERROR(add_document());
+ _doc->clear();
+ _CLDELETE(ts);
_rid++;
}
} else if constexpr (field_is_numeric_type(field_type)) {
@@ -426,7 +451,7 @@ public:
item_data_ptr = (uint8_t*)item_data_ptr + field_size;
}
auto value = join(strings, " ");
- new_fulltext_field(value.c_str(), value.length());
+ new_inverted_index_field(value.c_str(), value.length());
_rid++;
RETURN_IF_ERROR(add_document());
values++;
@@ -579,6 +604,7 @@ private:
std::unique_ptr<lucene::document::Document> _doc = nullptr;
lucene::document::Field* _field = nullptr;
+ bool _single_field = true;
std::unique_ptr<lucene::index::IndexWriter> _index_writer = nullptr;
std::unique_ptr<lucene::analysis::Analyzer> _analyzer = nullptr;
std::unique_ptr<lucene::util::Reader> _char_string_reader = nullptr;
@@ -598,11 +624,13 @@ Status InvertedIndexColumnWriter::create(const Field* field,
const auto* typeinfo = field->type_info();
FieldType type = typeinfo->type();
std::string field_name = field->name();
+ bool single_field = true;
if (type == FieldType::OLAP_FIELD_TYPE_ARRAY) {
const auto* array_typeinfo = dynamic_cast<const ArrayTypeInfo*>(typeinfo);
if (array_typeinfo != nullptr) {
typeinfo = array_typeinfo->item_type_info();
type = typeinfo->type();
+ single_field = false;
} else {
return Status::NotSupported("unsupported array type for inverted index: " +
std::to_string(int(type)));
@@ -610,10 +638,10 @@ Status InvertedIndexColumnWriter::create(const Field* field,
}
switch (type) {
-#define M(TYPE) \
- case TYPE: \
- *res = std::make_unique<InvertedIndexColumnWriterImpl<TYPE>>( \
- field_name, index_file_writer, index_meta); \
+#define M(TYPE) \
+ case TYPE: \
+ *res = std::make_unique<InvertedIndexColumnWriterImpl<TYPE>>( \
+ field_name, index_file_writer, index_meta, single_field); \
break;
M(FieldType::OLAP_FIELD_TYPE_TINYINT)
M(FieldType::OLAP_FIELD_TYPE_SMALLINT)
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 1a0836014ae..a4e81184b10 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -347,7 +347,7 @@ Status BlockChanger::change_block(vectorized::Block* ref_block,
assert_cast<vectorized::ColumnNullable*>(new_col->assume_mutable().get());
new_nullable_col->change_nested_column(ref_col);
- new_nullable_col->get_null_map_data().resize_fill(new_nullable_col->size());
+ new_nullable_col->get_null_map_data().resize_fill(ref_col->size());
} else {
// nullable to not nullable:
// suppose column `c_phone` is originally varchar(16) NOT NULL,
@@ -394,11 +394,22 @@ Status BlockChanger::_check_cast_valid(vectorized::ColumnPtr ref_column,
return Status::DataQualityError("Null data is changed to not nullable");
}
} else {
- const auto* new_null_map =
+ const auto& null_map_column =
vectorized::check_and_get_column<vectorized::ColumnNullable>(new_column)
- ->get_null_map_column()
- .get_data()
- .data();
+ ->get_null_map_column();
+ const auto& nested_column =
+ vectorized::check_and_get_column<vectorized::ColumnNullable>(new_column)
+ ->get_nested_column();
+ const auto* new_null_map = null_map_column.get_data().data();
+
+ if (null_map_column.size() != new_column->size() ||
+ nested_column.size() != new_column->size()) {
+ DCHECK(false);
+ return Status::InternalError(
+ "null_map_column size is changed, null_map_column_size={}, "
+ "new_column_size={}",
+ null_map_column.size(), new_column->size());
+ }
bool is_changed = false;
for (size_t i = 0; i < ref_column->size(); i++) {
@@ -713,7 +724,7 @@ Status SchemaChangeJob::process_alter_tablet(const TAlterTabletReqV2& request) {
}
SchemaChangeJob::SchemaChangeJob(StorageEngine& local_storage_engine,
- const TAlterTabletReqV2& request)
+ const TAlterTabletReqV2& request, const std::string& job_id)
: _local_storage_engine(local_storage_engine) {
_base_tablet = _local_storage_engine.tablet_manager()->get_tablet(request.base_tablet_id);
_new_tablet = _local_storage_engine.tablet_manager()->get_tablet(request.new_tablet_id);
@@ -726,6 +737,7 @@ SchemaChangeJob::SchemaChangeJob(StorageEngine& local_storage_engine,
// the complete variant is constructed by reading all the sub-columns of the variant.
_new_tablet_schema = _new_tablet->tablet_schema()->copy_without_extracted_columns();
}
+ _job_id = job_id;
}
// In the past schema change and rollup will create new tablet and will wait for txns starting before the task to finished
@@ -1017,7 +1029,7 @@ Status SchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc
int64_t* real_alter_version) {
LOG(INFO) << "begin to convert historical rowsets for new_tablet from base_tablet."
<< " base_tablet=" << _base_tablet->tablet_id()
- << ", new_tablet=" << _new_tablet->tablet_id();
+ << ", new_tablet=" << _new_tablet->tablet_id() << ", job_id=" << _job_id;
// find end version
int32_t end_version = -1;
@@ -1291,6 +1303,7 @@ Status SchemaChangeJob::parse_request(const SchemaChangeParams& sc_params,
// use directly schema change instead.
if (!(*sc_directly) && !(*sc_sorting)) {
// check has remote rowset
+ // work for cloud and cold storage
for (const auto& rs_reader : sc_params.ref_rowset_readers) {
if (!rs_reader->rowset()->is_local()) {
*sc_directly = true;
diff --git a/be/src/olap/schema_change.h b/be/src/olap/schema_change.h
index 2e1198e181a..dc77cff9d06 100644
--- a/be/src/olap/schema_change.h
+++ b/be/src/olap/schema_change.h
@@ -274,7 +274,8 @@ struct SchemaChangeParams {
class SchemaChangeJob {
public:
- SchemaChangeJob(StorageEngine& local_storage_engine, const TAlterTabletReqV2& request);
+ SchemaChangeJob(StorageEngine& local_storage_engine, const TAlterTabletReqV2& request,
+ const std::string& job_id);
Status process_alter_tablet(const TAlterTabletReqV2& request);
bool tablet_in_converting(int64_t tablet_id);
@@ -323,5 +324,6 @@ private:
std::shared_mutex _mutex;
std::unordered_set<int64_t> _tablet_ids_in_converting;
std::set<std::string> _supported_functions;
+ std::string _job_id;
};
} // namespace doris
diff --git a/be/src/pipeline/exec/join_probe_operator.cpp b/be/src/pipeline/exec/join_probe_operator.cpp
index 5c89075e8b9..c78e5423709 100644
--- a/be/src/pipeline/exec/join_probe_operator.cpp
+++ b/be/src/pipeline/exec/join_probe_operator.cpp
@@ -87,9 +87,7 @@ Status JoinProbeLocalState<SharedStateArg, Derived>::_build_output_block(
// and you could see a 'todo' in the Thrift definition.
// Here, we have refactored it, but considering upgrade compatibility, we still need to retain the old code.
if (!output_block->mem_reuse()) {
- vectorized::MutableBlock tmp(
- vectorized::VectorizedUtils::create_columns_with_type_and_name(p.row_desc()));
- output_block->swap(tmp.to_block());
+ output_block->swap(origin_block->clone_empty());
}
output_block->swap(*origin_block);
return Status::OK();
diff --git a/be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp b/be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp
index 0a3eb7c7063..87eae081d08 100644
--- a/be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp
+++ b/be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp
@@ -170,6 +170,9 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
if (revocable_mem_size(state) > 0) {
RETURN_IF_ERROR(revoke_memory(state));
} else {
+ for (auto& partition : local_state._shared_state->spill_partitions) {
+ RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+ }
local_state._dependency->set_ready_to_read();
}
}
@@ -244,8 +247,17 @@ Status PartitionedAggSinkLocalState::revoke_memory(RuntimeState* state) {
}
}
}};
+
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+
status = ExecEnv::GetInstance()->spill_stream_mgr()->get_async_task_thread_pool()->submit_func(
- [this, &parent, state] {
+ [this, &parent, state, execution_context] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return Status::Cancelled("Cancelled");
+ }
SCOPED_ATTACH_TASK(state);
SCOPED_TIMER(Base::_spill_timer);
Defer defer {[&]() {
diff --git a/be/src/pipeline/exec/partitioned_aggregation_sink_operator.h b/be/src/pipeline/exec/partitioned_aggregation_sink_operator.h
index 9628fb5766e..5e617386812 100644
--- a/be/src/pipeline/exec/partitioned_aggregation_sink_operator.h
+++ b/be/src/pipeline/exec/partitioned_aggregation_sink_operator.h
@@ -276,6 +276,11 @@ public:
std::mutex _spill_lock;
std::condition_variable _spill_cv;
+ /// Resources in shared state will be released when the operator is closed,
+ /// but there may be asynchronous spilling tasks at this time, which can lead to conflicts.
+ /// So, we need hold the pointer of shared state.
+ std::shared_ptr<PartitionedAggSharedState> _shared_state_holder;
+
// temp structures during spilling
vectorized::MutableColumns key_columns_;
vectorized::MutableColumns value_columns_;
diff --git a/be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp b/be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp
index 82bb25e674c..916a4e82681 100644
--- a/be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp
+++ b/be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp
@@ -196,9 +196,18 @@ Status PartitionedAggLocalState::initiate_merge_spill_partition_agg_data(Runtime
RETURN_IF_ERROR(Base::_shared_state->in_mem_shared_state->reset_hash_table());
_dependency->Dependency::block();
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
RETURN_IF_ERROR(
ExecEnv::GetInstance()->spill_stream_mgr()->get_async_task_thread_pool()->submit_func(
- [this, state] {
+ [this, state, execution_context] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ // FIXME: return status is meaningless?
+ return Status::Cancelled("Cancelled");
+ }
+
SCOPED_ATTACH_TASK(state);
Defer defer {[&]() {
if (!_status.ok()) {
diff --git a/be/src/pipeline/exec/partitioned_aggregation_source_operator.h b/be/src/pipeline/exec/partitioned_aggregation_source_operator.h
index cc408c07e21..ac63402f227 100644
--- a/be/src/pipeline/exec/partitioned_aggregation_source_operator.h
+++ b/be/src/pipeline/exec/partitioned_aggregation_source_operator.h
@@ -63,6 +63,11 @@ protected:
std::mutex _merge_spill_lock;
std::condition_variable _merge_spill_cv;
+ /// Resources in shared state will be released when the operator is closed,
+ /// but there may be asynchronous spilling tasks at this time, which can lead to conflicts.
+ /// So, we need hold the pointer of shared state.
+ std::shared_ptr<PartitionedAggSharedState> _shared_state_holder;
+
std::unique_ptr<RuntimeProfile> _internal_runtime_profile;
RuntimeProfile::Counter* _get_results_timer = nullptr;
RuntimeProfile::Counter* _serialize_result_timer = nullptr;
diff --git a/be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp b/be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp
index 93ce24e8e72..25ac9d333b1 100644
--- a/be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp
+++ b/be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp
@@ -174,29 +174,37 @@ Status PartitionedHashJoinProbeLocalState::spill_build_block(RuntimeState* state
auto* spill_io_pool = ExecEnv::GetInstance()->spill_stream_mgr()->get_spill_io_thread_pool(
build_spilling_stream->get_spill_root_dir());
- return spill_io_pool->submit_func([state, &build_spilling_stream, &mutable_block, this] {
- (void)state; // avoid ut compile error
- SCOPED_ATTACH_TASK(state);
- if (_spill_status_ok) {
- auto build_block = mutable_block->to_block();
- DCHECK_EQ(mutable_block->rows(), 0);
- auto st = build_spilling_stream->spill_block(build_block, false);
- if (!st.ok()) {
- std::unique_lock<std::mutex> lock(_spill_lock);
- _spill_status_ok = false;
- _spill_status = std::move(st);
- } else {
- COUNTER_UPDATE(_spill_build_rows, build_block.rows());
- COUNTER_UPDATE(_spill_build_blocks, 1);
- }
- }
- --_spilling_task_count;
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+ return spill_io_pool->submit_func(
+ [execution_context, state, &build_spilling_stream, &mutable_block, this] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return;
+ }
+ (void)state; // avoid ut compile error
+ SCOPED_ATTACH_TASK(state);
+ if (_spill_status_ok) {
+ auto build_block = mutable_block->to_block();
+ DCHECK_EQ(mutable_block->rows(), 0);
+ auto st = build_spilling_stream->spill_block(build_block, false);
+ if (!st.ok()) {
+ std::unique_lock<std::mutex> lock(_spill_lock);
+ _spill_status_ok = false;
+ _spill_status = std::move(st);
+ } else {
+ COUNTER_UPDATE(_spill_build_rows, build_block.rows());
+ COUNTER_UPDATE(_spill_build_blocks, 1);
+ }
+ }
+ --_spilling_task_count;
- if (_spilling_task_count == 0) {
- std::unique_lock<std::mutex> lock(_spill_lock);
- _dependency->set_ready();
- }
- });
+ if (_spilling_task_count == 0) {
+ std::unique_lock<std::mutex> lock(_spill_lock);
+ _dependency->set_ready();
+ }
+ });
}
Status PartitionedHashJoinProbeLocalState::spill_probe_blocks(RuntimeState* state,
@@ -223,34 +231,42 @@ Status PartitionedHashJoinProbeLocalState::spill_probe_blocks(RuntimeState* stat
}
if (!blocks.empty()) {
- return spill_io_pool->submit_func([state, &blocks, &spilling_stream, this] {
- (void)state; // avoid ut compile error
- SCOPED_ATTACH_TASK(state);
- COUNTER_UPDATE(_spill_probe_blocks, blocks.size());
- while (!blocks.empty()) {
- auto block = std::move(blocks.back());
- blocks.pop_back();
- if (_spill_status_ok) {
- auto st = spilling_stream->spill_block(block, false);
- if (!st.ok()) {
- std::unique_lock<std::mutex> lock(_spill_lock);
- _spill_status_ok = false;
- _spill_status = std::move(st);
- break;
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+ return spill_io_pool->submit_func(
+ [execution_context, state, &blocks, spilling_stream, this] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ _dependency->set_ready();
+ return;
+ }
+ SCOPED_ATTACH_TASK(state);
+ COUNTER_UPDATE(_spill_probe_blocks, blocks.size());
+ while (!blocks.empty() && !state->is_cancelled()) {
+ auto block = std::move(blocks.back());
+ blocks.pop_back();
+ if (_spill_status_ok) {
+ auto st = spilling_stream->spill_block(block, false);
+ if (!st.ok()) {
+ std::unique_lock<std::mutex> lock(_spill_lock);
+ _spill_status_ok = false;
+ _spill_status = std::move(st);
+ break;
+ }
+ COUNTER_UPDATE(_spill_probe_rows, block.rows());
+ } else {
+ break;
+ }
}
- COUNTER_UPDATE(_spill_probe_rows, block.rows());
- } else {
- break;
- }
- }
- --_spilling_task_count;
+ --_spilling_task_count;
- if (_spilling_task_count == 0) {
- std::unique_lock<std::mutex> lock(_spill_lock);
- _dependency->set_ready();
- }
- });
+ if (_spilling_task_count == 0) {
+ std::unique_lock<std::mutex> lock(_spill_lock);
+ _dependency->set_ready();
+ }
+ });
} else {
--_spilling_task_count;
if (_spilling_task_count == 0) {
@@ -296,7 +312,14 @@ Status PartitionedHashJoinProbeLocalState::recovery_build_blocks_from_disk(Runti
auto& mutable_block = _shared_state->partitioned_build_blocks[partition_index];
DCHECK(mutable_block != nullptr);
- auto read_func = [this, state, &spilled_stream, &mutable_block] {
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+ auto read_func = [this, state, &spilled_stream, &mutable_block, execution_context] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return;
+ }
Defer defer([this] { --_spilling_task_count; });
(void)state; // avoid ut compile error
SCOPED_ATTACH_TASK(state);
@@ -363,7 +386,14 @@ Status PartitionedHashJoinProbeLocalState::recovery_probe_blocks_from_disk(Runti
auto& blocks = _probe_blocks[partition_index];
/// TODO: maybe recovery more blocks each time.
- auto read_func = [this, state, &spilled_stream, &blocks] {
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+ auto read_func = [this, execution_context, state, &spilled_stream, &blocks] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return;
+ }
Defer defer([this] { --_spilling_task_count; });
(void)state; // avoid ut compile error
SCOPED_ATTACH_TASK(state);
diff --git a/be/src/pipeline/exec/partitioned_hash_join_probe_operator.h b/be/src/pipeline/exec/partitioned_hash_join_probe_operator.h
index adbaf19314f..7337017fde6 100644
--- a/be/src/pipeline/exec/partitioned_hash_join_probe_operator.h
+++ b/be/src/pipeline/exec/partitioned_hash_join_probe_operator.h
@@ -80,6 +80,11 @@ private:
std::vector<std::unique_ptr<vectorized::MutableBlock>> _partitioned_blocks;
std::map<uint32_t, std::vector<vectorized::Block>> _probe_blocks;
+ /// Resources in shared state will be released when the operator is closed,
+ /// but there may be asynchronous spilling tasks at this time, which can lead to conflicts.
+ /// So, we need hold the pointer of shared state.
+ std::shared_ptr<PartitionedHashJoinSharedState> _shared_state_holder;
+
std::vector<vectorized::SpillStreamSPtr> _probe_spilling_streams;
std::unique_ptr<PartitionerType> _partitioner;
diff --git a/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp b/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
index 20b3531c0cd..a63f22c1329 100644
--- a/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
+++ b/be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
@@ -74,7 +74,14 @@ Status PartitionedHashJoinSinkLocalState::revoke_memory(RuntimeState* state) {
auto* spill_io_pool =
ExecEnv::GetInstance()->spill_stream_mgr()->get_async_task_thread_pool();
DCHECK(spill_io_pool != nullptr);
- auto st = spill_io_pool->submit_func([this, state, spilling_stream, i] {
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+ auto st = spill_io_pool->submit_func([this, execution_context, state, spilling_stream, i] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return;
+ }
(void)state; // avoid ut compile error
SCOPED_ATTACH_TASK(state);
_spill_to_disk(i, spilling_stream);
diff --git a/be/src/pipeline/exec/partitioned_hash_join_sink_operator.h b/be/src/pipeline/exec/partitioned_hash_join_sink_operator.h
index f2d5ca3e140..44081bb0caa 100644
--- a/be/src/pipeline/exec/partitioned_hash_join_sink_operator.h
+++ b/be/src/pipeline/exec/partitioned_hash_join_sink_operator.h
@@ -66,6 +66,11 @@ protected:
Status _spill_status;
std::mutex _spill_status_lock;
+ /// Resources in shared state will be released when the operator is closed,
+ /// but there may be asynchronous spilling tasks at this time, which can lead to conflicts.
+ /// So, we need hold the pointer of shared state.
+ std::shared_ptr<PartitionedHashJoinSharedState> _shared_state_holder;
+
std::unique_ptr<PartitionerType> _partitioner;
RuntimeProfile::Counter* _partition_timer = nullptr;
diff --git a/be/src/pipeline/exec/spill_sort_sink_operator.cpp b/be/src/pipeline/exec/spill_sort_sink_operator.cpp
index 15edcf3c8d1..6e70ba5fcaf 100644
--- a/be/src/pipeline/exec/spill_sort_sink_operator.cpp
+++ b/be/src/pipeline/exec/spill_sort_sink_operator.cpp
@@ -214,11 +214,20 @@ Status SpillSortSinkLocalState::revoke_memory(RuntimeState* state) {
if (!_eos) {
Base::_dependency->Dependency::block();
}
+
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
status =
ExecEnv::GetInstance()
->spill_stream_mgr()
->get_spill_io_thread_pool(_spilling_stream->get_spill_root_dir())
- ->submit_func([this, state, &parent] {
+ ->submit_func([this, state, &parent, execution_context] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return Status::OK();
+ }
+
SCOPED_ATTACH_TASK(state);
Defer defer {[&]() {
if (!_shared_state->sink_status.ok()) {
diff --git a/be/src/pipeline/exec/spill_sort_sink_operator.h b/be/src/pipeline/exec/spill_sort_sink_operator.h
index aba85abece3..ae5a3bcb8c7 100644
--- a/be/src/pipeline/exec/spill_sort_sink_operator.h
+++ b/be/src/pipeline/exec/spill_sort_sink_operator.h
@@ -48,6 +48,11 @@ private:
friend class SpillSortSinkOperatorX;
+ /// Resources in shared state will be released when the operator is closed,
+ /// but there may be asynchronous spilling tasks at this time, which can lead to conflicts.
+ /// So, we need hold the pointer of shared state.
+ std::shared_ptr<SpillSortSharedState> _shared_state_holder;
+
std::unique_ptr<RuntimeState> _runtime_state;
std::unique_ptr<RuntimeProfile> _internal_runtime_profile;
RuntimeProfile::Counter* _partial_sort_timer = nullptr;
diff --git a/be/src/pipeline/exec/spill_sort_source_operator.cpp b/be/src/pipeline/exec/spill_sort_source_operator.cpp
index f82c015a175..c772b6b2797 100644
--- a/be/src/pipeline/exec/spill_sort_source_operator.cpp
+++ b/be/src/pipeline/exec/spill_sort_source_operator.cpp
@@ -90,7 +90,15 @@ Status SpillSortLocalState::initiate_merge_sort_spill_streams(RuntimeState* stat
}
}};
- auto spill_func = [this, state, &parent] {
+ auto execution_context = state->get_task_execution_context();
+ _shared_state_holder = _shared_state->shared_from_this();
+ auto spill_func = [this, state, &parent, execution_context] {
+ auto execution_context_lock = execution_context.lock();
+ if (!execution_context_lock) {
+ LOG(INFO) << "execution_context released, maybe query was cancelled.";
+ return Status::OK();
+ }
+
SCOPED_TIMER(_spill_merge_sort_timer);
SCOPED_ATTACH_TASK(state);
Defer defer {[&]() {
diff --git a/be/src/pipeline/exec/spill_sort_source_operator.h b/be/src/pipeline/exec/spill_sort_source_operator.h
index ae76c4e1b58..8132dd5a56c 100644
--- a/be/src/pipeline/exec/spill_sort_source_operator.h
+++ b/be/src/pipeline/exec/spill_sort_source_operator.h
@@ -57,6 +57,11 @@ protected:
bool _opened = false;
Status _status;
+ /// Resources in shared state will be released when the operator is closed,
+ /// but there may be asynchronous spilling tasks at this time, which can lead to conflicts.
+ /// So, we need hold the pointer of shared state.
+ std::shared_ptr<SpillSortSharedState> _shared_state_holder;
+
int64_t _external_sort_bytes_threshold = 134217728; // 128M
std::vector<vectorized::SpillStreamSPtr> _current_merging_streams;
std::unique_ptr<vectorized::VSortedRunMerger> _merger;
diff --git a/be/src/pipeline/pipeline_fragment_context.cpp b/be/src/pipeline/pipeline_fragment_context.cpp
index 8600a98b32c..0e075b90ebc 100644
--- a/be/src/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_fragment_context.cpp
@@ -459,7 +459,11 @@ void PipelineFragmentContext::trigger_report_if_necessary() {
// _runtime_state->load_channel_profile()->compute_time_in_profile(); // TODO load channel profile add timer
_runtime_state->load_channel_profile()->pretty_print(&ss);
}
- VLOG_FILE << ss.str();
+
+ VLOG_FILE << "Query " << print_id(this->get_query_id()) << " fragment "
+ << this->get_fragment_id() << " instance "
+ << print_id(this->get_fragment_instance_id()) << " profile:\n"
+ << ss.str();
}
auto st = send_report(false);
if (!st.ok()) {
@@ -917,7 +921,9 @@ void PipelineFragmentContext::_close_fragment_instance() {
if (_runtime_state->load_channel_profile()) {
_runtime_state->load_channel_profile()->pretty_print(&ss);
}
- LOG(INFO) << ss.str();
+
+ LOG_INFO("Query {} fragment {} instance {} profile:\n {}", print_id(this->_query_id),
+ this->_fragment_id, print_id(this->get_fragment_instance_id()), ss.str());
}
// all submitted tasks done
_exec_env->fragment_mgr()->remove_pipeline_context(
diff --git a/be/src/pipeline/pipeline_x/dependency.h b/be/src/pipeline/pipeline_x/dependency.h
index fe95c1c4470..622c175edcb 100644
--- a/be/src/pipeline/pipeline_x/dependency.h
+++ b/be/src/pipeline/pipeline_x/dependency.h
@@ -57,6 +57,8 @@ static constexpr auto TIME_UNIT_DEPENDENCY_LOG = 30 * 1000L * 1000L * 1000L;
static_assert(TIME_UNIT_DEPENDENCY_LOG < SLOW_DEPENDENCY_THRESHOLD);
struct BasicSharedState {
+ ENABLE_FACTORY_CREATOR(BasicSharedState)
+
template <class TARGET>
TARGET* cast() {
DCHECK(dynamic_cast<TARGET*>(this))
@@ -184,7 +186,9 @@ protected:
std::mutex _always_ready_lock;
};
-struct FakeSharedState final : public BasicSharedState {};
+struct FakeSharedState final : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(FakeSharedState)
+};
struct FakeDependency final : public Dependency {
public:
@@ -325,6 +329,7 @@ protected:
};
struct AggSharedState : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(AggSharedState)
public:
AggSharedState() {
agg_data = std::make_unique<vectorized::AggregatedDataVariants>();
@@ -418,8 +423,10 @@ private:
};
struct AggSpillPartition;
-struct PartitionedAggSharedState : public BasicSharedState {
-public:
+struct PartitionedAggSharedState : public BasicSharedState,
+ public std::enable_shared_from_this<PartitionedAggSharedState> {
+ ENABLE_FACTORY_CREATOR(PartitionedAggSharedState)
+
PartitionedAggSharedState() = default;
~PartitionedAggSharedState() override = default;
@@ -480,11 +487,15 @@ struct AggSpillPartition {
};
using AggSpillPartitionSPtr = std::shared_ptr<AggSpillPartition>;
struct SortSharedState : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(SortSharedState)
public:
std::unique_ptr<vectorized::Sorter> sorter;
};
-struct SpillSortSharedState : public BasicSharedState {
+struct SpillSortSharedState : public BasicSharedState,
+ public std::enable_shared_from_this<SpillSortSharedState> {
+ ENABLE_FACTORY_CREATOR(SpillSortSharedState)
+
SpillSortSharedState() = default;
~SpillSortSharedState() override = default;
@@ -512,6 +523,8 @@ struct SpillSortSharedState : public BasicSharedState {
};
struct UnionSharedState : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(UnionSharedState)
+
public:
UnionSharedState(int child_count = 1) : data_queue(child_count), _child_count(child_count) {};
int child_count() const { return _child_count; }
@@ -527,6 +540,8 @@ public:
};
struct AnalyticSharedState : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(AnalyticSharedState)
+
public:
AnalyticSharedState() = default;
@@ -560,6 +575,7 @@ struct JoinSharedState : public BasicSharedState {
};
struct HashJoinSharedState : public JoinSharedState {
+ ENABLE_FACTORY_CREATOR(HashJoinSharedState)
// mark the join column whether support null eq
std::vector<bool> is_null_safe_eq_join;
// mark the build hash table whether it needs to store null value
@@ -576,13 +592,18 @@ struct HashJoinSharedState : public JoinSharedState {
bool probe_ignore_null = false;
};
-struct PartitionedHashJoinSharedState : public HashJoinSharedState {
+struct PartitionedHashJoinSharedState
+ : public HashJoinSharedState,
+ public std::enable_shared_from_this<PartitionedHashJoinSharedState> {
+ ENABLE_FACTORY_CREATOR(PartitionedHashJoinSharedState)
+
std::vector<std::unique_ptr<vectorized::MutableBlock>> partitioned_build_blocks;
std::vector<vectorized::SpillStreamSPtr> spilled_streams;
bool need_to_spill = false;
};
struct NestedLoopJoinSharedState : public JoinSharedState {
+ ENABLE_FACTORY_CREATOR(NestedLoopJoinSharedState)
// if true, left child has no more rows to process
bool left_side_eos = false;
// Visited flags for each row in build side.
@@ -592,6 +613,7 @@ struct NestedLoopJoinSharedState : public JoinSharedState {
};
struct PartitionSortNodeSharedState : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(PartitionSortNodeSharedState)
public:
std::queue<vectorized::Block> blocks_buffer;
std::mutex buffer_mutex;
@@ -610,6 +632,7 @@ public:
};
struct SetSharedState : public BasicSharedState {
+ ENABLE_FACTORY_CREATOR(SetSharedState)
public:
/// default init
vectorized::Block build_block; // build to source
diff --git a/be/src/pipeline/pipeline_x/operator.cpp b/be/src/pipeline/pipeline_x/operator.cpp
index 6ee9ccb13c4..989b1ee00a5 100644
--- a/be/src/pipeline/pipeline_x/operator.cpp
+++ b/be/src/pipeline/pipeline_x/operator.cpp
@@ -167,7 +167,7 @@ Status OperatorXBase::close(RuntimeState* state) {
}
void PipelineXLocalStateBase::clear_origin_block() {
- _origin_block.clear_column_data(_parent->_row_descriptor.num_materialized_slots());
+ _origin_block.clear_column_data(_parent->intermediate_row_desc().num_materialized_slots());
}
Status OperatorXBase::do_projections(RuntimeState* state, vectorized::Block* origin_block,
@@ -299,7 +299,7 @@ std::shared_ptr<BasicSharedState> DataSinkOperatorX<LocalStateType>::create_shar
return nullptr;
} else {
std::shared_ptr<BasicSharedState> ss = nullptr;
- ss.reset(new typename LocalStateType::SharedStateType());
+ ss = LocalStateType::SharedStateType::create_shared();
ss->id = operator_id();
for (auto& dest : dests_id()) {
ss->related_op_ids.insert(dest);
diff --git a/be/src/pipeline/pipeline_x/operator.h b/be/src/pipeline/pipeline_x/operator.h
index 56991d43105..c375efb924d 100644
--- a/be/src/pipeline/pipeline_x/operator.h
+++ b/be/src/pipeline/pipeline_x/operator.h
@@ -328,8 +328,8 @@ protected:
int _parallel_tasks = 0;
//_keep_origin is used to avoid copying during projection,
- // currently set to true only in the nestloop join.
- bool _keep_origin = false;
+ // currently set to false only in the nestloop join.
+ bool _keep_origin = true;
};
template <typename LocalStateType>
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
index 5ed5dc8d598..adab6408dde 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -990,7 +990,7 @@ Status PipelineXFragmentContext::_create_operator(ObjectPool* pool, const TPlanN
break;
}
case TPlanNodeType::AGGREGATION_NODE: {
- if (tnode.agg_node.aggregate_functions.empty() &&
+ if (tnode.agg_node.aggregate_functions.empty() && !_runtime_state->enable_agg_spill() &&
request.query_options.__isset.enable_distinct_streaming_aggregation &&
request.query_options.enable_distinct_streaming_aggregation) {
op.reset(new DistinctStreamingAggOperatorX(pool, next_operator_id(), tnode, descs));
@@ -1351,7 +1351,9 @@ void PipelineXFragmentContext::_close_fragment_instance() {
if (_runtime_state->load_channel_profile()) {
_runtime_state->load_channel_profile()->pretty_print(&ss);
}
- LOG(INFO) << ss.str();
+
+ LOG_INFO("Query {} fragment {} profile:\n {}", print_id(this->_query_id),
+ this->_fragment_id, ss.str());
}
// all submitted tasks done
_exec_env->fragment_mgr()->remove_pipeline_context(
diff --git a/be/src/runtime/exec_env.h b/be/src/runtime/exec_env.h
index 8a71ca5ba44..f3be4d80333 100644
--- a/be/src/runtime/exec_env.h
+++ b/be/src/runtime/exec_env.h
@@ -214,6 +214,9 @@ public:
MemTableMemoryLimiter* memtable_memory_limiter() { return _memtable_memory_limiter.get(); }
WalManager* wal_mgr() { return _wal_manager.get(); }
#ifdef BE_TEST
+ void set_tmp_file_dir(std::unique_ptr<segment_v2::TmpFileDirs> tmp_file_dirs) {
+ this->_tmp_file_dirs = std::move(tmp_file_dirs);
+ }
void set_ready() { this->_s_ready = true; }
void set_not_ready() { this->_s_ready = false; }
void set_memtable_memory_limiter(MemTableMemoryLimiter* limiter) {
diff --git a/be/src/service/doris_main.cpp b/be/src/service/doris_main.cpp
index 5960db99214..a218c7be6f1 100644
--- a/be/src/service/doris_main.cpp
+++ b/be/src/service/doris_main.cpp
@@ -619,6 +619,8 @@ int main(int argc, char** argv) {
LOG(INFO) << "Be server stopped";
brpc_service.reset(nullptr);
LOG(INFO) << "Brpc service stopped";
+ service.reset();
+ LOG(INFO) << "Backend Service stopped";
exec_env->destroy();
doris::ThreadLocalHandle::del_thread_local_if_count_is_zero();
LOG(INFO) << "Doris main exited.";
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index 6bc88d4f0a1..b7a046921b5 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -1208,7 +1208,7 @@ public:
}
}
- BitmapValue(BitmapValue&& other) {
+ BitmapValue(BitmapValue&& other) noexcept {
_type = other._type;
switch (other._type) {
case EMPTY:
@@ -1265,7 +1265,7 @@ public:
return buf;
}
- BitmapValue& operator=(BitmapValue&& other) {
+ BitmapValue& operator=(BitmapValue&& other) noexcept {
if (this == &other) {
return *this;
}
diff --git a/be/src/util/quantile_state.h b/be/src/util/quantile_state.h
index c3b8cf83001..e2b782cf268 100644
--- a/be/src/util/quantile_state.h
+++ b/be/src/util/quantile_state.h
@@ -46,6 +46,11 @@ public:
QuantileState();
explicit QuantileState(float compression);
explicit QuantileState(const Slice& slice);
+ QuantileState& operator=(const QuantileState& other) noexcept = default;
+ QuantileState(const QuantileState& other) noexcept = default;
+ QuantileState& operator=(QuantileState&& other) noexcept = default;
+ QuantileState(QuantileState&& other) noexcept = default;
+
void set_compression(float compression);
bool deserialize(const Slice& slice);
size_t serialize(uint8_t* dst) const;
diff --git a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
index c33b8b50609..2069e8ae8a4 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
+++ b/be/src/vec/aggregate_functions/aggregate_function_simple_factory.cpp
@@ -31,6 +31,7 @@ void register_aggregate_function_combinator_distinct(AggregateFunctionSimpleFact
void register_aggregate_function_combinator_foreach(AggregateFunctionSimpleFactory& factory);
void register_aggregate_function_sum(AggregateFunctionSimpleFactory& factory);
+void register_aggregate_function_sum0(AggregateFunctionSimpleFactory& factory);
void register_aggregate_function_minmax(AggregateFunctionSimpleFactory& factory);
void register_aggregate_function_min_by(AggregateFunctionSimpleFactory& factory);
void register_aggregate_function_max_by(AggregateFunctionSimpleFactory& factory);
@@ -70,6 +71,7 @@ AggregateFunctionSimpleFactory& AggregateFunctionSimpleFactory::instance() {
static AggregateFunctionSimpleFactory instance;
std::call_once(oc, [&]() {
register_aggregate_function_sum(instance);
+ register_aggregate_function_sum0(instance);
register_aggregate_function_minmax(instance);
register_aggregate_function_min_by(instance);
register_aggregate_function_max_by(instance);
diff --git a/be/src/vec/aggregate_functions/aggregate_function_sum.cpp b/be/src/vec/aggregate_functions/aggregate_function_sum.cpp
index 3ee7dc6ff48..e0676957d46 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_sum.cpp
+++ b/be/src/vec/aggregate_functions/aggregate_function_sum.cpp
@@ -31,4 +31,8 @@ void register_aggregate_function_sum(AggregateFunctionSimpleFactory& factory) {
"sum_decimal256", creator_with_type::creator<AggregateFunctionSumSimpleDecimal256>);
}
+void register_aggregate_function_sum0(AggregateFunctionSimpleFactory& factory) {
+ factory.register_function_both("sum0", creator_with_type::creator<AggregateFunctionSumSimple>);
+}
+
} // namespace doris::vectorized
diff --git a/be/src/vec/aggregate_functions/aggregate_function_uniq.h b/be/src/vec/aggregate_functions/aggregate_function_uniq.h
index 72be9e01833..2e8855134eb 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_uniq.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_uniq.h
@@ -36,7 +36,6 @@
#include "vec/common/assert_cast.h"
#include "vec/common/hash_table/hash.h"
#include "vec/common/hash_table/phmap_fwd_decl.h"
-#include "vec/common/sip_hash.h"
#include "vec/common/string_ref.h"
#include "vec/common/uint128.h"
#include "vec/core/types.h"
@@ -64,17 +63,14 @@ template <typename T>
struct AggregateFunctionUniqExactData {
static constexpr bool is_string_key = std::is_same_v<T, String>;
using Key = std::conditional_t<is_string_key, UInt128, T>;
- using Hash = std::conditional_t<is_string_key, UInt128TrivialHash, HashCRC32<Key>>;
+ using Hash = HashCRC32<Key>;
using Set = flat_hash_set<Key, Hash>;
// TODO: replace SipHash with xxhash to speed up
static UInt128 ALWAYS_INLINE get_key(const StringRef& value) {
- UInt128 key;
- SipHash hash;
- hash.update(value.data, value.size);
- hash.get128(key.low, key.high);
- return key;
+ auto hash_value = XXH_INLINE_XXH128(value.data, value.size, 0);
+ return UInt128 {hash_value.high64, hash_value.low64};
}
Set set;
diff --git a/be/src/vec/data_types/serde/data_type_object_serde.cpp b/be/src/vec/data_types/serde/data_type_object_serde.cpp
index 6d4f82e845b..99a301294dc 100644
--- a/be/src/vec/data_types/serde/data_type_object_serde.cpp
+++ b/be/src/vec/data_types/serde/data_type_object_serde.cpp
@@ -19,6 +19,7 @@
#include <rapidjson/stringbuffer.h>
+#include "common/exception.h"
#include "common/status.h"
#include "vec/columns/column.h"
#include "vec/columns/column_object.h"
@@ -95,6 +96,28 @@ void DataTypeObjectSerDe::read_one_cell_from_jsonb(IColumn& column, const JsonbV
variant.insert(field);
}
+void DataTypeObjectSerDe::write_column_to_arrow(const IColumn& column, const NullMap* null_map,
+ arrow::ArrayBuilder* array_builder, int start,
+ int end, const cctz::time_zone& ctz) const {
+ const auto* var = check_and_get_column<ColumnObject>(column);
+ auto& builder = assert_cast<arrow::StringBuilder&>(*array_builder);
+ for (size_t i = start; i < end; ++i) {
+ if (null_map && (*null_map)[i]) {
+ checkArrowStatus(builder.AppendNull(), column.get_name(),
+ array_builder->type()->name());
+ } else {
+ std::string serialized_value;
+ if (!var->serialize_one_row_to_string(i, &serialized_value)) {
+ throw doris::Exception(ErrorCode::INTERNAL_ERROR, "Failed to serialize variant {}",
+ var->dump_structure());
+ }
+ checkArrowStatus(builder.Append(serialized_value.data(),
+ static_cast<int>(serialized_value.size())),
+ column.get_name(), array_builder->type()->name());
+ }
+ }
+}
+
} // namespace vectorized
} // namespace doris
\ No newline at end of file
diff --git a/be/src/vec/data_types/serde/data_type_object_serde.h b/be/src/vec/data_types/serde/data_type_object_serde.h
index 4d78d67352d..c589a1af92a 100644
--- a/be/src/vec/data_types/serde/data_type_object_serde.h
+++ b/be/src/vec/data_types/serde/data_type_object_serde.h
@@ -73,10 +73,7 @@ public:
void write_column_to_arrow(const IColumn& column, const NullMap* null_map,
arrow::ArrayBuilder* array_builder, int start, int end,
- const cctz::time_zone& ctz) const override {
- throw doris::Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
- "write_column_to_arrow with type " + column.get_name());
- }
+ const cctz::time_zone& ctz) const override;
void read_column_from_arrow(IColumn& column, const arrow::Array* arrow_array, int start,
int end, const cctz::time_zone& ctz) const override {
throw doris::Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
diff --git a/be/src/vec/exec/join/vjoin_node_base.cpp b/be/src/vec/exec/join/vjoin_node_base.cpp
index 9b954811ee9..a9e25e7626b 100644
--- a/be/src/vec/exec/join/vjoin_node_base.cpp
+++ b/be/src/vec/exec/join/vjoin_node_base.cpp
@@ -185,8 +185,7 @@ Status VJoinNodeBase::_build_output_block(Block* origin_block, Block* output_blo
// and you could see a 'todo' in the Thrift definition.
// Here, we have refactored it, but considering upgrade compatibility, we still need to retain the old code.
if (!output_block->mem_reuse()) {
- MutableBlock tmp(VectorizedUtils::create_columns_with_type_and_name(row_desc()));
- output_block->swap(tmp.to_block());
+ output_block->swap(origin_block->clone_empty());
}
output_block->swap(*origin_block);
return Status::OK();
diff --git a/be/src/vec/functions/uuid_numeric.cpp b/be/src/vec/functions/uuid_numeric.cpp
index 012d0d488a1..221cbd7d34c 100644
--- a/be/src/vec/functions/uuid_numeric.cpp
+++ b/be/src/vec/functions/uuid_numeric.cpp
@@ -22,10 +22,7 @@
#include "common/status.h"
#include "runtime/large_int_value.h"
#include "vec/columns/column_vector.h"
-#include "vec/columns/columns_number.h"
#include "vec/common/hash_table/hash.h"
-#include "vec/common/sip_hash.h"
-#include "vec/common/uint128.h"
#include "vec/core/block.h"
#include "vec/core/types.h"
#include "vec/data_types/data_type_number.h"
@@ -100,7 +97,6 @@ public:
}
// TODO(zhiqiang): May be override open function?
-
Status execute_impl(FunctionContext* /*context*/, Block& block,
const ColumnNumbers& /*arguments*/, size_t result,
size_t input_rows_count) const override {
@@ -140,14 +136,16 @@ private:
UInt64 randomSeed() const {
struct timespec times {};
+ clock_gettime(CLOCK_MONOTONIC, ×);
/// Not cryptographically secure as time, pid and stack address can be predictable.
-
- SipHash hash;
- hash.update(times.tv_nsec);
- hash.update(times.tv_sec);
- hash.update((uintptr_t)pthread_self());
-
- return hash.get64();
+ auto ret = HashUtil::xxHash64WithSeed(reinterpret_cast<const char*>(×.tv_nsec),
+ sizeof(times.tv_nsec), 0);
+ ret = HashUtil::xxHash64WithSeed(reinterpret_cast<const char*>(×.tv_sec),
+ sizeof(times.tv_sec), ret);
+ ret = HashUtil::xxHash64WithSeed(reinterpret_cast<const char*>((uintptr_t)pthread_self()),
+ sizeof(pthread_t), ret);
+
+ return ret;
}
};
diff --git a/be/src/vec/olap/vgeneric_iterators.cpp b/be/src/vec/olap/vgeneric_iterators.cpp
index 2d45c884a78..c973a7131f9 100644
--- a/be/src/vec/olap/vgeneric_iterators.cpp
+++ b/be/src/vec/olap/vgeneric_iterators.cpp
@@ -422,9 +422,9 @@ Status VUnionIterator::current_block_row_locations(std::vector<RowLocation>* loc
RowwiseIteratorUPtr new_merge_iterator(std::vector<RowwiseIteratorUPtr>&& inputs,
int sequence_id_idx, bool is_unique, bool is_reverse,
uint64_t* merged_rows) {
- if (inputs.size() == 1) {
- return std::move(inputs[0]);
- }
+ // when the size of inputs is 1, we also need to use VMergeIterator, because the
+ // next_block_view function only be implemented in VMergeIterator. The reason why
+ // the size of inputs is 1 is that the segment was filtered out by zone map or others.
return std::make_unique<VMergeIterator>(std::move(inputs), sequence_id_idx, is_unique,
is_reverse, merged_rows);
}
diff --git a/be/test/olap/rowset/segment_v2/inverted_index_array_test.cpp b/be/test/olap/rowset/segment_v2/inverted_index_array_test.cpp
new file mode 100644
index 00000000000..74e9827db25
--- /dev/null
+++ b/be/test/olap/rowset/segment_v2/inverted_index_array_test.cpp
@@ -0,0 +1,225 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <CLucene.h>
+#include <CLucene/config/repl_wchar.h>
+#include <CLucene/index/IndexReader.h>
+#include <gtest/gtest-message.h>
+#include <gtest/gtest-test-part.h>
+#include <string.h>
+
+#include <memory>
+#include <string>
+
+#include "gtest/gtest_pred_impl.h"
+#include "io/fs/file_writer.h"
+#include "io/fs/local_file_system.h"
+#include "olap/rowset/segment_v2/inverted_index_compound_reader.h"
+#include "olap/rowset/segment_v2/inverted_index_file_writer.h"
+#include "olap/rowset/segment_v2/inverted_index_fs_directory.h"
+#include "olap/rowset/segment_v2/inverted_index_writer.h"
+#include "olap/rowset/segment_v2/zone_map_index.h"
+#include "olap/tablet_schema.h"
+#include "olap/tablet_schema_helper.h"
+#include "runtime/exec_env.h"
+#include "util/slice.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_string.h"
+#include "vec/common/pod_array_fwd.h"
+#include "vec/core/field.h"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_factory.hpp"
+#include "vec/olap/olap_data_convertor.h"
+
+using namespace lucene::index;
+using doris::segment_v2::InvertedIndexFileWriter;
+
+namespace doris {
+namespace segment_v2 {
+
+class InvertedIndexArrayTest : public testing::Test {
+public:
+ const std::string kTestDir = "./ut_dir/inverted_index_array_test";
+
+ void check_terms_stats(string dir_str, string file_str) {
+ auto fs = io::global_local_filesystem();
+ std::unique_ptr<DorisCompoundReader> reader = std::make_unique<DorisCompoundReader>(
+ DorisFSDirectoryFactory::getDirectory(fs, dir_str.c_str()), file_str.c_str(), 4096);
+ std::cout << "Term statistics for " << file_str << std::endl;
+ std::cout << "==================================" << std::endl;
+ lucene::store::Directory* dir = reader.get();
+
+ IndexReader* r = IndexReader::open(dir);
+
+ printf("Max Docs: %d\n", r->maxDoc());
+ printf("Num Docs: %d\n", r->numDocs());
+
+ TermEnum* te = r->terms();
+ int32_t nterms;
+ for (nterms = 0; te->next(); nterms++) {
+ /* empty */
+ std::string token =
+ lucene_wcstoutf8string(te->term(false)->text(), te->term(false)->textLength());
+
+ printf("Term: %s ", token.c_str());
+ printf("Freq: %d\n", te->docFreq());
+ }
+ printf("Term count: %d\n\n", nterms);
+ te->close();
+ _CLLDELETE(te);
+
+ r->close();
+ _CLLDELETE(r);
+ reader->close();
+ }
+
+ void SetUp() override {
+ auto st = io::global_local_filesystem()->delete_directory(kTestDir);
+ ASSERT_TRUE(st.ok()) << st;
+ st = io::global_local_filesystem()->create_directory(kTestDir);
+ ASSERT_TRUE(st.ok()) << st;
+ config::enable_write_index_searcher_cache = false;
+ std::vector<StorePath> paths;
+ paths.emplace_back(kTestDir, 1024);
+ auto tmp_file_dirs = std::make_unique<segment_v2::TmpFileDirs>(paths);
+ st = tmp_file_dirs->init();
+ if (!st.OK()) {
+ std::cout << "init tmp file dirs error:" << st.to_string() << std::endl;
+ return;
+ }
+ ExecEnv::GetInstance()->set_tmp_file_dir(std::move(tmp_file_dirs));
+ }
+ void TearDown() override {
+ EXPECT_TRUE(io::global_local_filesystem()->delete_directory(kTestDir).ok());
+ }
+
+ void test_string(std::string testname, Field* field) {
+ EXPECT_TRUE(field->type() == FieldType::OLAP_FIELD_TYPE_ARRAY);
+ std::string filename = kTestDir + "/" + testname;
+ auto fs = io::global_local_filesystem();
+
+ io::FileWriterPtr file_writer;
+ EXPECT_TRUE(fs->create_file(filename, &file_writer).ok());
+ auto index_meta_pb = std::make_unique<TabletIndexPB>();
+ index_meta_pb->set_index_type(IndexType::INVERTED);
+ index_meta_pb->set_index_id(26033);
+ index_meta_pb->set_index_name("index_inverted_arr1");
+ index_meta_pb->clear_col_unique_id();
+ index_meta_pb->add_col_unique_id(0);
+
+ TabletIndex idx_meta;
+ idx_meta.index_type();
+ idx_meta.init_from_pb(*index_meta_pb.get());
+ auto index_file_writer = std::make_unique<InvertedIndexFileWriter>(
+ fs, file_writer->path().parent_path(), file_writer->path().filename(),
+ InvertedIndexStorageFormatPB::V1);
+ std::unique_ptr<segment_v2::InvertedIndexColumnWriter> _inverted_index_builder = nullptr;
+ EXPECT_EQ(InvertedIndexColumnWriter::create(field, &_inverted_index_builder,
+ index_file_writer.get(), &idx_meta),
+ Status::OK());
+ vectorized::PaddedPODArray<Slice> _slice;
+ _slice.resize(5);
+
+ vectorized::Array a1, a2;
+ a1.push_back("amory");
+ a1.push_back("doris");
+ a2.push_back(vectorized::Null());
+ a2.push_back("amory");
+ a2.push_back("commiter");
+
+ vectorized::DataTypePtr s1 = std::make_shared<vectorized::DataTypeNullable>(
+ std::make_shared<vectorized::DataTypeString>());
+ vectorized::DataTypePtr au = std::make_shared<vectorized::DataTypeArray>(s1);
+ vectorized::MutableColumnPtr col = au->create_column();
+ col->insert(a1);
+ col->insert(a2);
+ vectorized::ColumnPtr column_array = std::move(col);
+ vectorized::ColumnWithTypeAndName type_and_name(column_array, au, "arr1");
+
+ vectorized::PaddedPODArray<vectorized::UInt64> _offsets;
+ _offsets.reserve(3);
+ _offsets.emplace_back(0);
+ _offsets.emplace_back(2);
+ _offsets.emplace_back(5);
+ const uint8_t* offsets_ptr = (const uint8_t*)(_offsets.data());
+
+ auto* col_arr = assert_cast<const vectorized::ColumnArray*>(column_array.get());
+ const vectorized::UInt8* nested_null_map =
+ assert_cast<const vectorized::ColumnNullable*>(col_arr->get_data_ptr().get())
+ ->get_null_map_data()
+ .data();
+ auto* col_arr_str = assert_cast<const vectorized::ColumnString*>(
+ assert_cast<const vectorized::ColumnNullable*>(col_arr->get_data_ptr().get())
+ ->get_nested_column_ptr()
+ .get());
+ const char* char_data = (const char*)(col_arr_str->get_chars().data());
+ const vectorized::ColumnString::Offset* offset_cur = col_arr_str->get_offsets().data();
+ const vectorized::ColumnString::Offset* offset_end = offset_cur + 5;
+
+ Slice* slice = _slice.data();
+ size_t string_offset = *(offset_cur - 1);
+ const vectorized::UInt8* nullmap_cur = nested_null_map;
+ while (offset_cur != offset_end) {
+ if (!*nullmap_cur) {
+ slice->data = const_cast<char*>(char_data + string_offset);
+ slice->size = *offset_cur - string_offset;
+ } else {
+ slice->data = nullptr;
+ slice->size = 0;
+ }
+ string_offset = *offset_cur;
+ ++nullmap_cur;
+ ++slice;
+ ++offset_cur;
+ }
+
+ auto field_size = field->get_sub_field(0)->size();
+ Status st = _inverted_index_builder->add_array_values(
+ field_size, reinterpret_cast<const void*>(_slice.data()),
+ reinterpret_cast<const uint8_t*>(nested_null_map), offsets_ptr, 2);
+ EXPECT_EQ(st, Status::OK());
+ EXPECT_EQ(_inverted_index_builder->finish(), Status::OK());
+ EXPECT_EQ(index_file_writer->close(), Status::OK());
+
+ {
+ std::cout << "dir: " << file_writer->path().parent_path().string() << std::endl;
+ string idx_file_name = file_writer->path().filename().string() + "_26033.idx";
+ std::cout << "file: " << file_writer->path().filename().string() << std::endl;
+ check_terms_stats(file_writer->path().parent_path().string(), idx_file_name);
+ }
+ }
+};
+
+TEST_F(InvertedIndexArrayTest, ArrayString) {
+ TabletColumn arrayTabletColumn;
+ arrayTabletColumn.set_unique_id(0);
+ arrayTabletColumn.set_name("arr1");
+ arrayTabletColumn.set_type(FieldType::OLAP_FIELD_TYPE_ARRAY);
+ TabletColumn arraySubColumn;
+ arraySubColumn.set_unique_id(1);
+ arraySubColumn.set_name("arr_sub_string");
+ arraySubColumn.set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ arrayTabletColumn.add_sub_column(arraySubColumn);
+ Field* field = FieldFactory::create(arrayTabletColumn);
+ test_string("InvertedIndexArray", field);
+ delete field;
+}
+
+} // namespace segment_v2
+} // namespace doris
diff --git a/cloud/src/meta-service/http_encode_key.cpp b/cloud/src/meta-service/http_encode_key.cpp
index aa0c3c4dac9..f70f2accb50 100644
--- a/cloud/src/meta-service/http_encode_key.cpp
+++ b/cloud/src/meta-service/http_encode_key.cpp
@@ -140,7 +140,8 @@ static std::unordered_map<std::string_view,
{"TxnInfoKey", {{"instance_id", "db_id", "txn_id"}, [](param_type& p) { return txn_info_key(KeyInfoSetter<TxnInfoKeyInfo>{p}.get()); } , parse<TxnInfoPB>}} ,
{"TxnIndexKey", {{"instance_id", "txn_id"}, [](param_type& p) { return txn_index_key(KeyInfoSetter<TxnIndexKeyInfo>{p}.get()); } , parse<TxnIndexPB>}} ,
{"TxnRunningKey", {{"instance_id", "db_id", "txn_id"}, [](param_type& p) { return txn_running_key(KeyInfoSetter<TxnRunningKeyInfo>{p}.get()); } , parse<TxnRunningPB>}} ,
- {"VersionKey", {{"instance_id", "db_id", "tbl_id", "partition_id"}, [](param_type& p) { return version_key(KeyInfoSetter<VersionKeyInfo>{p}.get()); } , parse<VersionPB>}} ,
+ {"PartitionVersionKey", {{"instance_id", "db_id", "tbl_id", "partition_id"}, [](param_type& p) { return partition_version_key(KeyInfoSetter<PartitionVersionKeyInfo>{p}.get()); } , parse<VersionPB>}} ,
+ {"TableVersionKey", {{"instance_id", "db_id", "tbl_id"}, [](param_type& p) { return table_version_key(KeyInfoSetter<TableVersionKeyInfo>{p}.get()); } , parse<VersionPB>}} ,
{"MetaRowsetKey", {{"instance_id", "tablet_id", "version"}, [](param_type& p) { return meta_rowset_key(KeyInfoSetter<MetaRowsetKeyInfo>{p}.get()); } , parse<doris::RowsetMetaCloudPB>}} ,
{"MetaRowsetTmpKey", {{"instance_id", "txn_id", "tablet_id"}, [](param_type& p) { return meta_rowset_tmp_key(KeyInfoSetter<MetaRowsetTmpKeyInfo>{p}.get()); } , parse<doris::RowsetMetaCloudPB>}} ,
{"MetaTabletKey", {{"instance_id", "table_id", "index_id", "part_id", "tablet_id"}, [](param_type& p) { return meta_tablet_key(KeyInfoSetter<MetaTabletKeyInfo>{p}.get()); } , parse<doris::TabletMetaCloudPB>}} ,
diff --git a/cloud/src/meta-service/keys.cpp b/cloud/src/meta-service/keys.cpp
index f3e89dfbc66..6e883681696 100644
--- a/cloud/src/meta-service/keys.cpp
+++ b/cloud/src/meta-service/keys.cpp
@@ -43,7 +43,8 @@ namespace doris::cloud {
[[maybe_unused]] static const char* TXN_KEY_INFIX_INDEX = "txn_index";
[[maybe_unused]] static const char* TXN_KEY_INFIX_RUNNING = "txn_running";
-[[maybe_unused]] static const char* VERSION_KEY_INFIX = "partition";
+[[maybe_unused]] static const char* PARTITION_VERSION_KEY_INFIX = "partition";
+[[maybe_unused]] static const char* TABLE_VERSION_KEY_INFIX = "table";
[[maybe_unused]] static const char* META_KEY_INFIX_ROWSET = "rowset";
[[maybe_unused]] static const char* META_KEY_INFIX_ROWSET_TMP = "rowset_tmp";
@@ -114,9 +115,9 @@ static void encode_prefix(const T& t, std::string* key) {
InstanceKeyInfo,
TxnLabelKeyInfo, TxnInfoKeyInfo, TxnIndexKeyInfo, TxnRunningKeyInfo,
MetaRowsetKeyInfo, MetaRowsetTmpKeyInfo, MetaTabletKeyInfo, MetaTabletIdxKeyInfo, MetaSchemaKeyInfo,
- MetaDeleteBitmapInfo, MetaDeleteBitmapUpdateLockInfo, MetaPendingDeleteBitmapInfo, VersionKeyInfo,
+ MetaDeleteBitmapInfo, MetaDeleteBitmapUpdateLockInfo, MetaPendingDeleteBitmapInfo, PartitionVersionKeyInfo,
RecycleIndexKeyInfo, RecyclePartKeyInfo, RecycleRowsetKeyInfo, RecycleTxnKeyInfo, RecycleStageKeyInfo,
- StatsTabletKeyInfo,
+ StatsTabletKeyInfo, TableVersionKeyInfo,
JobTabletKeyInfo, JobRecycleKeyInfo, RLJobProgressKeyInfo,
CopyJobKeyInfo, CopyFileKeyInfo, MetaRowsetSchemaKeyInfo, StorageVaultKeyInfo>);
@@ -139,7 +140,8 @@ static void encode_prefix(const T& t, std::string* key) {
|| std::is_same_v<T, MetaDeleteBitmapUpdateLockInfo>
|| std::is_same_v<T, MetaPendingDeleteBitmapInfo>) {
encode_bytes(META_KEY_PREFIX, key);
- } else if constexpr (std::is_same_v<T, VersionKeyInfo>) {
+ } else if constexpr (std::is_same_v<T, PartitionVersionKeyInfo>
+ || std::is_same_v<T, TableVersionKeyInfo>) {
encode_bytes(VERSION_KEY_PREFIX, key);
} else if constexpr (std::is_same_v<T, RecycleIndexKeyInfo>
|| std::is_same_v<T, RecyclePartKeyInfo>
@@ -217,9 +219,22 @@ void txn_running_key(const TxnRunningKeyInfo& in, std::string* out) {
// Version keys
//==============================================================================
-void version_key(const VersionKeyInfo& in, std::string* out) {
+std::string version_key_prefix(std::string_view instance_id) {
+ std::string out;
+ encode_prefix(TableVersionKeyInfo {instance_id, 0, 0}, &out);
+ return out;
+}
+
+void table_version_key(const TableVersionKeyInfo& in, std::string* out) {
+ encode_prefix(in, out); // 0x01 "version" ${instance_id}
+ encode_bytes(TABLE_VERSION_KEY_INFIX, out); // "table"
+ encode_int64(std::get<1>(in), out); // db_id
+ encode_int64(std::get<2>(in), out); // tbl_id
+}
+
+void partition_version_key(const PartitionVersionKeyInfo& in, std::string* out) {
encode_prefix(in, out); // 0x01 "version" ${instance_id}
- encode_bytes(VERSION_KEY_INFIX, out); // "partition"
+ encode_bytes(PARTITION_VERSION_KEY_INFIX, out); // "partition"
encode_int64(std::get<1>(in), out); // db_id
encode_int64(std::get<2>(in), out); // tbl_id
encode_int64(std::get<3>(in), out); // partition_id
diff --git a/cloud/src/meta-service/keys.h b/cloud/src/meta-service/keys.h
index edc9afcf4e8..483332a133c 100644
--- a/cloud/src/meta-service/keys.h
+++ b/cloud/src/meta-service/keys.h
@@ -34,6 +34,7 @@
// 0x01 "txn" ${instance_id} "txn_running" ${db_id} ${txn_id} -> TxnRunningPB
//
// 0x01 "version" ${instance_id} "partition" ${db_id} ${tbl_id} ${partition_id} -> VersionPB
+// 0x01 "version" ${instance_id} "table" ${db_id} ${tbl_id} -> int64
//
// 0x01 "meta" ${instance_id} "rowset" ${tablet_id} ${version} -> RowsetMetaCloudPB
// 0x01 "meta" ${instance_id} "rowset_tmp" ${txn_id} ${tablet_id} -> RowsetMetaCloudPB
@@ -115,7 +116,7 @@ using TxnIndexKeyInfo = BasicKeyInfo<3 , std::tuple<std::string, int64_t>>
using TxnRunningKeyInfo = BasicKeyInfo<5 , std::tuple<std::string, int64_t, int64_t>>;
// 0:instance_id 1:db_id 2:tbl_id 3:partition_id
-using VersionKeyInfo = BasicKeyInfo<6 , std::tuple<std::string, int64_t, int64_t, int64_t>>;
+using PartitionVersionKeyInfo = BasicKeyInfo<6 , std::tuple<std::string, int64_t, int64_t, int64_t>>;
// 0:instance_id 1:tablet_id 2:version
using MetaRowsetKeyInfo = BasicKeyInfo<7 , std::tuple<std::string, int64_t, int64_t>>;
@@ -181,6 +182,9 @@ using RLJobProgressKeyInfo = BasicKeyInfo<25, std::tuple<std::string, int64_t, i
// 0:instance_id 1:vault_id
using StorageVaultKeyInfo = BasicKeyInfo<26, std::tuple<std::string, std::string>>;
+// 0:instance_id 1:db_id 2:table_id
+using TableVersionKeyInfo = BasicKeyInfo<27, std::tuple<std::string, int64_t, int64_t>>;
+
void instance_key(const InstanceKeyInfo& in, std::string* out);
static inline std::string instance_key(const InstanceKeyInfo& in) { std::string s; instance_key(in, &s); return s; }
@@ -197,8 +201,11 @@ static inline std::string txn_info_key(const TxnInfoKeyInfo& in) { std::string s
static inline std::string txn_index_key(const TxnIndexKeyInfo& in) { std::string s; txn_index_key(in, &s); return s; }
static inline std::string txn_running_key(const TxnRunningKeyInfo& in) { std::string s; txn_running_key(in, &s); return s; }
-void version_key(const VersionKeyInfo& in, std::string* out);
-static inline std::string version_key(const VersionKeyInfo& in) { std::string s; version_key(in, &s); return s; }
+std::string version_key_prefix(std::string_view instance_id);
+void partition_version_key(const PartitionVersionKeyInfo& in, std::string* out);
+static inline std::string partition_version_key(const PartitionVersionKeyInfo& in) { std::string s; partition_version_key(in, &s); return s; }
+void table_version_key(const TableVersionKeyInfo& in, std::string* out);
+static inline std::string table_version_key(const TableVersionKeyInfo& in) { std::string s; table_version_key(in, &s); return s; }
std::string meta_key_prefix(std::string_view instance_id);
void meta_rowset_key(const MetaRowsetKeyInfo& in, std::string* out);
diff --git a/cloud/src/meta-service/meta_service.cpp b/cloud/src/meta-service/meta_service.cpp
index 5f2a0d39066..d9c9048793c 100644
--- a/cloud/src/meta-service/meta_service.cpp
+++ b/cloud/src/meta-service/meta_service.cpp
@@ -204,13 +204,19 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
cloud_unique_id = request->cloud_unique_id();
}
+ bool is_table_version = false;
+ if (request->has_is_table_version()) {
+ is_table_version = request->is_table_version();
+ }
+
int64_t db_id = request->has_db_id() ? request->db_id() : -1;
int64_t table_id = request->has_table_id() ? request->table_id() : -1;
int64_t partition_id = request->has_partition_id() ? request->partition_id() : -1;
- if (db_id == -1 || table_id == -1 || partition_id == -1) {
+ if (db_id == -1 || table_id == -1 || (!is_table_version && partition_id == -1)) {
msg = "params error, db_id=" + std::to_string(db_id) +
" table_id=" + std::to_string(table_id) +
- " partition_id=" + std::to_string(partition_id);
+ " partition_id=" + std::to_string(partition_id) +
+ " is_table_version=" + std::to_string(is_table_version);
code = MetaServiceCode::INVALID_ARGUMENT;
LOG(WARNING) << msg;
return;
@@ -224,9 +230,12 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
return;
}
RPC_RATE_LIMIT(get_version)
- VersionKeyInfo ver_key_info {instance_id, db_id, table_id, partition_id};
std::string ver_key;
- version_key(ver_key_info, &ver_key);
+ if (is_table_version) {
+ table_version_key({instance_id, db_id, table_id}, &ver_key);
+ } else {
+ partition_version_key({instance_id, db_id, table_id, partition_id}, &ver_key);
+ }
std::unique_ptr<Transaction> txn;
TxnErrorCode err = txn_kv_->create_txn(&txn);
@@ -237,17 +246,22 @@ void MetaServiceImpl::get_version(::google::protobuf::RpcController* controller,
}
std::string ver_val;
- VersionPB version_pb;
// 0 for success get a key, 1 for key not found, negative for error
err = txn->get(ver_key, &ver_val);
VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
if (err == TxnErrorCode::TXN_OK) {
- if (!version_pb.ParseFromString(ver_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed version value";
- return;
+ if (is_table_version) {
+ int64_t version = *reinterpret_cast<const int64_t*>(ver_val.data());
+ response->set_version(version);
+ } else {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed version value";
+ return;
+ }
+ response->set_version(version_pb.version());
}
- response->set_version(version_pb.version());
{ TEST_SYNC_POINT_CALLBACK("get_version_code", &code); }
return;
} else if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
@@ -270,12 +284,18 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
cloud_unique_id = request->cloud_unique_id();
}
+ bool is_table_version = false;
+ if (request->has_is_table_version()) {
+ is_table_version = request->is_table_version();
+ }
+
if (request->db_ids_size() == 0 || request->table_ids_size() == 0 ||
- request->table_ids_size() != request->partition_ids_size() ||
- request->db_ids_size() != request->partition_ids_size()) {
+ (!is_table_version && request->table_ids_size() != request->partition_ids_size()) ||
+ (!is_table_version && request->db_ids_size() != request->partition_ids_size())) {
msg = "param error, num db_ids=" + std::to_string(request->db_ids_size()) +
" num table_ids=" + std::to_string(request->table_ids_size()) +
- " num partition_ids=" + std::to_string(request->partition_ids_size());
+ " num partition_ids=" + std::to_string(request->partition_ids_size()) +
+ " is_table_version=" + std::to_string(request->is_table_version());
code = MetaServiceCode::INVALID_ARGUMENT;
LOG(WARNING) << msg;
return;
@@ -308,7 +328,12 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
int64_t db_id = request->db_ids(i);
int64_t table_id = request->table_ids(i);
int64_t partition_id = request->partition_ids(i);
- std::string ver_key = version_key({instance_id, db_id, table_id, partition_id});
+ std::string ver_key;
+ if (is_table_version) {
+ table_version_key({instance_id, db_id, table_id}, &ver_key);
+ } else {
+ partition_version_key({instance_id, db_id, table_id, partition_id}, &ver_key);
+ }
// TODO(walter) support batch get.
std::string ver_val;
@@ -316,13 +341,18 @@ void MetaServiceImpl::batch_get_version(::google::protobuf::RpcController* contr
TEST_SYNC_POINT_CALLBACK("batch_get_version_err", &err);
VLOG_DEBUG << "xxx get version_key=" << hex(ver_key);
if (err == TxnErrorCode::TXN_OK) {
- VersionPB version_pb;
- if (!version_pb.ParseFromString(ver_val)) {
- code = MetaServiceCode::PROTOBUF_PARSE_ERR;
- msg = "malformed version value";
- break;
+ if (is_table_version) {
+ int64_t version = *reinterpret_cast<const int64_t*>(ver_val.data());
+ response->add_versions(version);
+ } else {
+ VersionPB version_pb;
+ if (!version_pb.ParseFromString(ver_val)) {
+ code = MetaServiceCode::PROTOBUF_PARSE_ERR;
+ msg = "malformed version value";
+ break;
+ }
+ response->add_versions(version_pb.version());
}
- response->add_versions(version_pb.version());
} else if (err == TxnErrorCode::TXN_KEY_NOT_FOUND) {
// return -1 if the target version is not exists.
response->add_versions(-1);
diff --git a/cloud/src/meta-service/meta_service_partition.cpp b/cloud/src/meta-service/meta_service_partition.cpp
index 17c67fb22ff..c5a7db7d902 100644
--- a/cloud/src/meta-service/meta_service_partition.cpp
+++ b/cloud/src/meta-service/meta_service_partition.cpp
@@ -222,6 +222,16 @@ void MetaServiceImpl::commit_index(::google::protobuf::RpcController* controller
LOG_INFO("remove recycle index").tag("key", hex(key));
txn->remove(key);
}
+
+ if (request->has_db_id() && request->has_is_new_table() && request->is_new_table()) {
+ // init table version, for create and truncate table
+ std::string key = table_version_key({instance_id, request->db_id(), request->table_id()});
+ std::string val(sizeof(int64_t), 0);
+ *reinterpret_cast<int64_t*>(val.data()) = (int64_t)1;
+ txn->put(key, val);
+ LOG_INFO("put table version").tag("key", hex(key));
+ }
+
err = txn->commit();
if (err != TxnErrorCode::TXN_OK) {
code = cast_as<ErrCategory::COMMIT>(err);
@@ -493,6 +503,14 @@ void MetaServiceImpl::commit_partition(::google::protobuf::RpcController* contro
LOG_INFO("remove recycle partition").tag("key", hex(key));
txn->remove(key);
}
+
+ // update table versions
+ if (request->has_db_id()) {
+ std::string ver_key = table_version_key({instance_id, request->db_id(), request->table_id()});
+ txn->atomic_add(ver_key, 1);
+ LOG_INFO("update table version").tag("ver_key", hex(ver_key));
+ }
+
err = txn->commit();
if (err != TxnErrorCode::TXN_OK) {
code = cast_as<ErrCategory::COMMIT>(err);
@@ -513,8 +531,7 @@ void MetaServiceImpl::drop_partition(::google::protobuf::RpcController* controll
}
RPC_RATE_LIMIT(drop_partition)
- if (request->partition_ids().empty() || request->index_ids().empty() ||
- !request->has_table_id()) {
+ if (request->partition_ids().empty() || request->index_ids().empty() || !request->has_table_id()) {
code = MetaServiceCode::INVALID_ARGUMENT;
msg = "empty partition_ids or index_ids or table_id";
return;
@@ -579,6 +596,14 @@ void MetaServiceImpl::drop_partition(::google::protobuf::RpcController* controll
}
}
if (!need_commit) return;
+
+ // update table versions
+ if (request->has_db_id()) {
+ std::string ver_key = table_version_key({instance_id, request->db_id(), request->table_id()});
+ txn->atomic_add(ver_key, 1);
+ LOG_INFO("update table version").tag("ver_key", hex(ver_key));
+ }
+
err = txn->commit();
if (err != TxnErrorCode::TXN_OK) {
code = cast_as<ErrCategory::COMMIT>(err);
diff --git a/cloud/src/meta-service/meta_service_txn.cpp b/cloud/src/meta-service/meta_service_txn.cpp
index 03251ee0f07..0c7f57385b3 100644
--- a/cloud/src/meta-service/meta_service_txn.cpp
+++ b/cloud/src/meta-service/meta_service_txn.cpp
@@ -864,7 +864,7 @@ void MetaServiceImpl::commit_txn(::google::protobuf::RpcController* controller,
int64_t table_id = tablet_ids[tablet_id].table_id();
int64_t partition_id = i.partition_id();
- std::string ver_key = version_key({instance_id, db_id, table_id, partition_id});
+ std::string ver_key = partition_version_key({instance_id, db_id, table_id, partition_id});
int64_t version = -1;
std::string ver_val_str;
int64_t new_version = -1;
@@ -983,11 +983,11 @@ void MetaServiceImpl::commit_txn(::google::protobuf::RpcController* controller,
txn->put(i.first, ver_val);
put_size += i.first.size() + ver_val.size();
- LOG(INFO) << "xxx put version_key=" << hex(i.first) << " version:" << i.second
+ LOG(INFO) << "xxx put partition_version_key=" << hex(i.first) << " version:" << i.second
<< " txn_id=" << txn_id;
std::string_view ver_key = i.first;
- //VersionKeyInfo {instance_id, db_id, table_id, partition_id}
+ //PartitionVersionKeyInfo {instance_id, db_id, table_id, partition_id}
ver_key.remove_prefix(1); // Remove key space
std::vector<std::tuple<std::variant<int64_t, std::string>, int, int>> out;
int ret = decode_key(&ver_key, &out);
@@ -1009,6 +1009,15 @@ void MetaServiceImpl::commit_txn(::google::protobuf::RpcController* controller,
response->add_versions(i.second);
}
+ // Save table versions
+ num_put_keys += table_id_tablet_ids.size();
+ for (auto& i : table_id_tablet_ids) {
+ std::string ver_key = table_version_key({instance_id, db_id, i.first});
+ txn->atomic_add(ver_key, 1);
+ put_size += ver_key.size();
+ LOG(INFO) << "xxx atomic add table_version_key=" << hex(ver_key) << " txn_id=" << txn_id;
+ }
+
LOG(INFO) << " before update txn_info=" << txn_info.ShortDebugString();
// Update txn_info
diff --git a/cloud/src/recycler/meta_checker.cpp b/cloud/src/recycler/meta_checker.cpp
index 86a17019399..00a27aa9a29 100644
--- a/cloud/src/recycler/meta_checker.cpp
+++ b/cloud/src/recycler/meta_checker.cpp
@@ -171,7 +171,7 @@ bool MetaChecker::check_fdb_by_fe_meta(MYSQL* conn) {
int num_row = mysql_num_rows(result);
for (int i = 0; i < num_row; ++i) {
MYSQL_ROW row = mysql_fetch_row(result);
- TabletInfo tablet_info;
+ TabletInfo tablet_info = { 0 };
tablet_info.tablet_id = atoll(row[0]);
tablet_info.schema_version = atoll(row[4]);
tablets.push_back(std::move(tablet_info));
@@ -197,7 +197,7 @@ bool MetaChecker::check_fdb_by_fe_meta(MYSQL* conn) {
tablet_info.partition_id = atoll(row[6]);
tablet_info.index_id = atoll(row[7]);
- PartitionInfo partition_info;
+ PartitionInfo partition_info = { 0 };
partition_info.db_id = atoll(row[4]);
partition_info.table_id = atoll(row[5]);
partition_info.partition_id = atoll(row[6]);
diff --git a/cloud/src/recycler/recycler.cpp b/cloud/src/recycler/recycler.cpp
index 1ca1e05f741..cafcd2131d8 100644
--- a/cloud/src/recycler/recycler.cpp
+++ b/cloud/src/recycler/recycler.cpp
@@ -527,9 +527,8 @@ int InstanceRecycler::recycle_deleted_instance() {
std::string start_txn_key = txn_key_prefix(instance_id_);
std::string end_txn_key = txn_key_prefix(instance_id_ + '\x00');
txn->remove(start_txn_key, end_txn_key);
- // 0:instance_id 1:db_id 2:tbl_id 3:partition_id
- std::string start_version_key = version_key({instance_id_, 0, 0, 0});
- std::string end_version_key = version_key({instance_id_, INT64_MAX, 0, 0});
+ std::string start_version_key = version_key_prefix(instance_id_);
+ std::string end_version_key = version_key_prefix(instance_id_ + '\x00');
txn->remove(start_version_key, end_version_key);
std::string start_meta_key = meta_key_prefix(instance_id_);
std::string end_meta_key = meta_key_prefix(instance_id_ + '\x00');
@@ -758,7 +757,7 @@ int InstanceRecycler::recycle_partitions() {
// Elements in `partition_keys` has the same lifetime as `it` in `scan_and_recycle`
std::vector<std::string_view> partition_keys;
- std::vector<std::string> version_keys;
+ std::vector<std::string> partition_version_keys;
auto recycle_func = [&, this](std::string_view k, std::string_view v) -> int {
++num_scanned;
RecyclePartitionPB part_pb;
@@ -831,18 +830,18 @@ int InstanceRecycler::recycle_partitions() {
check_recycle_task(instance_id_, task_name, num_scanned, num_recycled, start_time);
partition_keys.push_back(k);
if (part_pb.db_id() > 0) {
- version_keys.push_back(version_key(
+ partition_version_keys.push_back(partition_version_key(
{instance_id_, part_pb.db_id(), part_pb.table_id(), partition_id}));
}
}
return ret;
};
- auto loop_done = [&partition_keys, &version_keys, this]() -> int {
+ auto loop_done = [&partition_keys, &partition_version_keys, this]() -> int {
if (partition_keys.empty()) return 0;
std::unique_ptr<int, std::function<void(int*)>> defer((int*)0x01, [&](int*) {
partition_keys.clear();
- version_keys.clear();
+ partition_version_keys.clear();
});
std::unique_ptr<Transaction> txn;
TxnErrorCode err = txn_kv_->create_txn(&txn);
@@ -853,7 +852,7 @@ int InstanceRecycler::recycle_partitions() {
for (auto& k : partition_keys) {
txn->remove(k);
}
- for (auto& k : version_keys) {
+ for (auto& k : partition_version_keys) {
txn->remove(k);
}
err = txn->commit();
@@ -872,24 +871,24 @@ int InstanceRecycler::recycle_versions() {
int num_scanned = 0;
int num_recycled = 0;
- LOG_INFO("begin to recycle partition versions").tag("instance_id", instance_id_);
+ LOG_INFO("begin to recycle table and partition versions").tag("instance_id", instance_id_);
auto start_time = steady_clock::now();
std::unique_ptr<int, std::function<void(int*)>> defer_log_statistics((int*)0x01, [&](int*) {
auto cost = duration<float>(steady_clock::now() - start_time).count();
- LOG_INFO("recycle partition versions finished, cost={}s", cost)
+ LOG_INFO("recycle table and partition versions finished, cost={}s", cost)
.tag("instance_id", instance_id_)
.tag("num_scanned", num_scanned)
.tag("num_recycled", num_recycled);
});
- auto version_key_begin = version_key({instance_id_, 0, 0, 0});
- auto version_key_end = version_key({instance_id_, INT64_MAX, 0, 0});
+ auto version_key_begin = partition_version_key({instance_id_, 0, 0, 0});
+ auto version_key_end = partition_version_key({instance_id_, INT64_MAX, 0, 0});
int64_t last_scanned_table_id = 0;
bool is_recycled = false; // Is last scanned kv recycled
auto recycle_func = [&num_scanned, &num_recycled, &last_scanned_table_id, &is_recycled, this](
- std::string_view k, std::string_view) {
+ std::string_view k, std::string_view) {
++num_scanned;
auto k1 = k;
k1.remove_prefix(1);
@@ -916,16 +915,20 @@ int InstanceRecycler::recycle_versions() {
if (err != TxnErrorCode::TXN_OK) {
return -1;
}
- if (iter->has_next()) { // Table is useful, should not recycle partiton versions
+ if (iter->has_next()) { // Table is useful, should not recycle table and partition versions
return 0;
}
- // Remove all version kvs of this table
auto db_id = std::get<int64_t>(std::get<0>(out[3]));
- auto table_version_key_begin = version_key({instance_id_, db_id, table_id, 0});
- auto table_version_key_end = version_key({instance_id_, db_id, table_id, INT64_MAX});
- txn->remove(table_version_key_begin, table_version_key_end);
- LOG(WARNING) << "remove version kv, begin=" << hex(table_version_key_begin)
- << " end=" << hex(table_version_key_end);
+ // 1. Remove all partition version kvs of this table
+ auto partition_version_key_begin = partition_version_key({instance_id_, db_id, table_id, 0});
+ auto partition_version_key_end = partition_version_key({instance_id_, db_id, table_id, INT64_MAX});
+ txn->remove(partition_version_key_begin, partition_version_key_end);
+ LOG(WARNING) << "remove partition version kv, begin=" << hex(partition_version_key_begin)
+ << " end=" << hex(partition_version_key_end);
+ // 2. Remove the table version kv of this table
+ auto tbl_version_key = table_version_key({instance_id_, db_id, table_id});
+ txn->remove(tbl_version_key);
+ LOG(WARNING) << "remove table version kv " << hex(tbl_version_key);
err = txn->commit();
if (err != TxnErrorCode::TXN_OK) {
return -1;
diff --git a/cloud/test/http_encode_key_test.cpp b/cloud/test/http_encode_key_test.cpp
index ac5eade9735..25182d54489 100644
--- a/cloud/test/http_encode_key_test.cpp
+++ b/cloud/test/http_encode_key_test.cpp
@@ -168,7 +168,7 @@ txn_id=126419752960)",
R"({"table_ids":["10001"]})",
},
Input {
- "VersionKey",
+ "PartitionVersionKey",
"instance_id=gavin-instance&db_id=10086&tbl_id=10010&partition_id=10000",
"011076657273696f6e000110676176696e2d696e7374616e6365000110706172746974696f6e000112000000000000276612000000000000271a120000000000002710",
[]() -> std::string {
@@ -178,6 +178,17 @@ txn_id=126419752960)",
},
R"({"version":"10"})",
},
+ Input {
+ "TableVersionKey",
+ "instance_id=gavin-instance&db_id=10086&tbl_id=10010",
+ "011076657273696f6e000110676176696e2d696e7374616e63650001107461626c65000112000000000000276612000000000000271a",
+ []() -> std::string {
+ VersionPB pb;
+ pb.set_version(10);
+ return pb.SerializeAsString();
+ },
+ R"({"version":"10"})",
+ },
Input {
"MetaRowsetKey",
"instance_id=gavin-instance&tablet_id=10086&version=10010",
diff --git a/cloud/test/keys_test.cpp b/cloud/test/keys_test.cpp
index 614a17374d0..ce7cbb0551e 100644
--- a/cloud/test/keys_test.cpp
+++ b/cloud/test/keys_test.cpp
@@ -298,14 +298,14 @@ TEST(KeysTest, VersionKeyTest) {
using namespace doris::cloud;
std::string instance_id = "instance_id_deadbeef";
- // 0x01 "version" ${instance_id} "version_id" ${db_id} ${tbl_id} ${partition_id} -> ${version}
+ // 0x01 "version" ${instance_id} "partition" ${db_id} ${tbl_id} ${partition_id} -> ${version}
{
int64_t db_id = 11111;
- int64_t tablet_id = 10086;
+ int64_t table_id = 10086;
int64_t partition_id = 9998;
- VersionKeyInfo v_key {instance_id, db_id, tablet_id, partition_id};
+ PartitionVersionKeyInfo v_key {instance_id, db_id, table_id, partition_id};
std::string encoded_version_key0;
- version_key(v_key, &encoded_version_key0);
+ partition_version_key(v_key, &encoded_version_key0);
std::cout << "version key after encode: " << hex(encoded_version_key0) << std::endl;
std::string dec_instance_id;
@@ -329,12 +329,50 @@ TEST(KeysTest, VersionKeyTest) {
EXPECT_EQ("partition", dec_version_infix);
EXPECT_EQ(instance_id, dec_instance_id);
EXPECT_EQ(db_id, dec_db_id);
- EXPECT_EQ(tablet_id, dec_table_id);
+ EXPECT_EQ(table_id, dec_table_id);
EXPECT_EQ(partition_id, dec_partition_id);
std::get<3>(v_key) = partition_id + 1;
std::string encoded_version_key1;
- version_key(v_key, &encoded_version_key1);
+ partition_version_key(v_key, &encoded_version_key1);
+ std::cout << "version key after encode: " << hex(encoded_version_key1) << std::endl;
+
+ ASSERT_GT(encoded_version_key1, encoded_version_key0);
+ }
+
+ // 0x01 "version" ${instance_id} "table" ${db_id} ${tbl_id} -> ${version}
+ {
+ int64_t db_id = 11111;
+ int64_t table_id = 10010;
+ TableVersionKeyInfo v_key {instance_id, db_id, table_id};
+ std::string encoded_version_key0;
+ table_version_key(v_key, &encoded_version_key0);
+ std::cout << "version key after encode: " << hex(encoded_version_key0) << std::endl;
+
+ std::string dec_instance_id;
+ int64_t dec_db_id = 0;
+ int64_t dec_table_id = 0;
+
+ std::string_view key_sv(encoded_version_key0);
+ std::string dec_version_prefix;
+ std::string dec_version_infix;
+ remove_user_space_prefix(&key_sv);
+ ASSERT_EQ(decode_bytes(&key_sv, &dec_version_prefix), 0);
+ ASSERT_EQ(decode_bytes(&key_sv, &dec_instance_id), 0);
+ ASSERT_EQ(decode_bytes(&key_sv, &dec_version_infix), 0);
+ ASSERT_EQ(decode_int64(&key_sv, &dec_db_id), 0) << hex(key_sv);
+ ASSERT_EQ(decode_int64(&key_sv, &dec_table_id), 0);
+ ASSERT_TRUE(key_sv.empty());
+
+ EXPECT_EQ("version", dec_version_prefix);
+ EXPECT_EQ("table", dec_version_infix);
+ EXPECT_EQ(instance_id, dec_instance_id);
+ EXPECT_EQ(db_id, dec_db_id);
+ EXPECT_EQ(table_id, dec_table_id);
+
+ std::get<2>(v_key) = table_id + 1;
+ std::string encoded_version_key1;
+ table_version_key(v_key, &encoded_version_key1);
std::cout << "version key after encode: " << hex(encoded_version_key1) << std::endl;
ASSERT_GT(encoded_version_key1, encoded_version_key0);
diff --git a/cloud/test/meta_service_test.cpp b/cloud/test/meta_service_test.cpp
index 2d8e0c89a5c..a497cb566f4 100644
--- a/cloud/test/meta_service_test.cpp
+++ b/cloud/test/meta_service_test.cpp
@@ -4247,6 +4247,8 @@ TEST(MetaServiceTest, IndexRequest) {
auto tablet_val = tablet_pb.SerializeAsString();
RecycleIndexPB index_pb;
auto index_key = recycle_index_key({instance_id, index_id});
+ int64_t val_int = 0;
+ auto tbl_version_key = table_version_key({instance_id, 1, table_id});
std::string val;
// ------------Test prepare index------------
@@ -4255,8 +4257,10 @@ TEST(MetaServiceTest, IndexRequest) {
IndexResponse res;
meta_service->prepare_index(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
+ req.set_db_id(1);
req.set_table_id(table_id);
req.add_index_ids(index_id);
+ req.set_is_new_table(true);
// Last state UNKNOWN
res.Clear();
meta_service->prepare_index(&ctrl, &req, &res, nullptr);
@@ -4312,19 +4316,26 @@ TEST(MetaServiceTest, IndexRequest) {
ASSERT_EQ(res.status().code(), MetaServiceCode::ALREADY_EXISTED);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ // Prepare index should not init table version
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// ------------Test commit index------------
reset_meta_service();
req.Clear();
meta_service->commit_index(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
+ req.set_db_id(1);
req.set_table_id(table_id);
req.add_index_ids(index_id);
+ req.set_is_new_table(true);
// Last state UNKNOWN
res.Clear();
meta_service->commit_index(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// Last state PREPARED
reset_meta_service();
index_pb.set_state(RecycleIndexPB::PREPARED);
@@ -4337,6 +4348,11 @@ TEST(MetaServiceTest, IndexRequest) {
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ // First commit index should init table version
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
// Last state DROPPED
reset_meta_service();
index_pb.set_state(RecycleIndexPB::DROPPED);
@@ -4351,6 +4367,8 @@ TEST(MetaServiceTest, IndexRequest) {
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(index_pb.ParseFromString(val));
ASSERT_EQ(index_pb.state(), RecycleIndexPB::DROPPED);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// Last state RECYCLING
reset_meta_service();
index_pb.set_state(RecycleIndexPB::RECYCLING);
@@ -4365,6 +4383,8 @@ TEST(MetaServiceTest, IndexRequest) {
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(index_pb.ParseFromString(val));
ASSERT_EQ(index_pb.state(), RecycleIndexPB::RECYCLING);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// Last state UNKNOWN but tablet meta existed
reset_meta_service();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4375,13 +4395,17 @@ TEST(MetaServiceTest, IndexRequest) {
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// ------------Test drop index------------
reset_meta_service();
req.Clear();
meta_service->drop_index(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
+ req.set_db_id(1);
req.set_table_id(table_id);
req.add_index_ids(index_id);
+ req.set_is_new_table(true);
// Last state UNKNOWN
res.Clear();
meta_service->drop_index(&ctrl, &req, &res, nullptr);
@@ -4432,6 +4456,9 @@ TEST(MetaServiceTest, IndexRequest) {
ASSERT_EQ(txn->get(index_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(index_pb.ParseFromString(val));
ASSERT_EQ(index_pb.state(), RecycleIndexPB::RECYCLING);
+ // Drop index should not init table version
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
}
TEST(MetaServiceTest, PartitionRequest) {
@@ -4460,6 +4487,8 @@ TEST(MetaServiceTest, PartitionRequest) {
auto tablet_val = tablet_pb.SerializeAsString();
RecyclePartitionPB partition_pb;
auto partition_key = recycle_partition_key({instance_id, partition_id});
+ int64_t val_int = 0;
+ auto tbl_version_key = table_version_key({instance_id, 1, table_id});
std::string val;
// ------------Test prepare partition------------
brpc::Controller ctrl;
@@ -4467,17 +4496,26 @@ TEST(MetaServiceTest, PartitionRequest) {
PartitionResponse res;
meta_service->prepare_partition(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
+ req.set_db_id(1);
req.set_table_id(table_id);
req.add_index_ids(index_id);
req.add_partition_ids(partition_id);
// Last state UNKNOWN
res.Clear();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
meta_service->prepare_partition(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::PREPARED);
+ // Prepare partition should not update table version
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
// Last state PREPARED
res.Clear();
meta_service->prepare_partition(&ctrl, &req, &res, nullptr);
@@ -4525,11 +4563,14 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(res.status().code(), MetaServiceCode::ALREADY_EXISTED);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// ------------Test commit partition------------
reset_meta_service();
req.Clear();
meta_service->commit_partition(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
+ req.set_db_id(1);
req.set_table_id(table_id);
req.add_index_ids(index_id);
req.add_partition_ids(partition_id);
@@ -4541,6 +4582,9 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
// Last state PREPARED
reset_meta_service();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
partition_pb.set_state(RecyclePartitionPB::PREPARED);
val = partition_pb.SerializeAsString();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4551,8 +4595,16 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ // Commit partition should update table version
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 2);
// Last state DROPPED
reset_meta_service();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
partition_pb.set_state(RecyclePartitionPB::DROPPED);
val = partition_pb.SerializeAsString();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4565,8 +4617,15 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::DROPPED);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
// Last state RECYCLING
reset_meta_service();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
partition_pb.set_state(RecyclePartitionPB::RECYCLING);
val = partition_pb.SerializeAsString();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4579,9 +4638,16 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::RECYCLING);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
// Last state UNKNOWN but tablet meta existed
reset_meta_service();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
txn->put(tablet_key, tablet_val);
ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
res.Clear();
@@ -4589,6 +4655,10 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_KEY_NOT_FOUND);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
// Last state UNKNOWN and tablet meta existed, but request has no index ids
reset_meta_service();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4606,19 +4676,31 @@ TEST(MetaServiceTest, PartitionRequest) {
req.Clear();
meta_service->drop_partition(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::INVALID_ARGUMENT);
+ req.set_db_id(1);
req.set_table_id(table_id);
req.add_index_ids(index_id);
req.add_partition_ids(partition_id);
// Last state UNKNOWN
res.Clear();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
meta_service->drop_partition(&ctrl, &req, &res, nullptr);
ASSERT_EQ(res.status().code(), MetaServiceCode::OK);
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::DROPPED);
+ // Drop partition should update table version
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 2);
// Last state PREPARED
reset_meta_service();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
partition_pb.set_state(RecyclePartitionPB::PREPARED);
val = partition_pb.SerializeAsString();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4631,8 +4713,15 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::DROPPED);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 2);
// Last state DROPPED
reset_meta_service();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
partition_pb.set_state(RecyclePartitionPB::DROPPED);
val = partition_pb.SerializeAsString();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4645,8 +4734,15 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::DROPPED);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
// Last state RECYCLING
reset_meta_service();
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ txn->atomic_add(tbl_version_key, 1);
+ ASSERT_EQ(txn->commit(), TxnErrorCode::TXN_OK);
partition_pb.set_state(RecyclePartitionPB::RECYCLING);
val = partition_pb.SerializeAsString();
ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
@@ -4659,6 +4755,10 @@ TEST(MetaServiceTest, PartitionRequest) {
ASSERT_EQ(txn->get(partition_key, &val), TxnErrorCode::TXN_OK);
ASSERT_TRUE(partition_pb.ParseFromString(val));
ASSERT_EQ(partition_pb.state(), RecyclePartitionPB::RECYCLING);
+ ASSERT_EQ(meta_service->txn_kv()->create_txn(&txn), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(tbl_version_key, &val), TxnErrorCode::TXN_OK);
+ val_int = *reinterpret_cast<const int64_t*>(val.data());
+ ASSERT_EQ(val_int, 1);
}
TEST(MetaServiceTxnStoreRetryableTest, MockGetVersion) {
diff --git a/cloud/test/recycler_test.cpp b/cloud/test/recycler_test.cpp
index 87c70833a30..de059d0df09 100644
--- a/cloud/test/recycler_test.cpp
+++ b/cloud/test/recycler_test.cpp
@@ -293,8 +293,8 @@ static int create_recycle_partiton(TxnKv* txn_kv, int64_t table_id, int64_t part
return 0;
}
-static int create_version_kv(TxnKv* txn_kv, int64_t table_id, int64_t partition_id) {
- auto key = version_key({instance_id, db_id, table_id, partition_id});
+static int create_partition_version_kv(TxnKv* txn_kv, int64_t table_id, int64_t partition_id) {
+ auto key = partition_version_key({instance_id, db_id, table_id, partition_id});
VersionPB version;
version.set_version(1);
auto val = version.SerializeAsString();
@@ -309,6 +309,21 @@ static int create_version_kv(TxnKv* txn_kv, int64_t table_id, int64_t partition_
return 0;
}
+static int create_table_version_kv(TxnKv* txn_kv, int64_t table_id) {
+ auto key = table_version_key({instance_id, db_id, table_id});
+ std::string val(sizeof(int64_t), 0);
+ *reinterpret_cast<int64_t*>(val.data()) = (int64_t) 1;
+ std::unique_ptr<Transaction> txn;
+ if (txn_kv->create_txn(&txn) != TxnErrorCode::TXN_OK) {
+ return -1;
+ }
+ txn->put(key, val);
+ if (txn->commit() != TxnErrorCode::TXN_OK) {
+ return -1;
+ }
+ return 0;
+}
+
static int create_txn_label_kv(TxnKv* txn_kv, std::string label, int64_t db_id) {
std::string txn_label_key_;
std::string txn_label_val;
@@ -1133,8 +1148,9 @@ TEST(RecyclerTest, recycle_versions) {
}
}
for (auto partition_id : partition_ids) {
- create_version_kv(txn_kv.get(), table_id, partition_id);
+ create_partition_version_kv(txn_kv.get(), table_id, partition_id);
}
+ create_table_version_kv(txn_kv.get(), table_id);
// Drop partitions
for (int i = 0; i < 5; ++i) {
create_recycle_partiton(txn_kv.get(), table_id, partition_ids[i], index_ids);
@@ -1147,16 +1163,23 @@ TEST(RecyclerTest, recycle_versions) {
// Recycle all partitions in table except 30006
ASSERT_EQ(recycler.recycle_partitions(), 0);
ASSERT_EQ(recycler.recycle_versions(), 0); // `recycle_versions` should do nothing
- // All version kvs except version of partition 30006 must have been deleted
+ // All partition version kvs except version of partition 30006 must have been deleted
std::unique_ptr<Transaction> txn;
ASSERT_EQ(txn_kv->create_txn(&txn), TxnErrorCode::TXN_OK);
- auto key_begin = version_key({instance_id, db_id, table_id, 0});
- auto key_end = version_key({instance_id, db_id, table_id, INT64_MAX});
+ auto partition_key_begin = partition_version_key({instance_id, db_id, table_id, 0});
+ auto partition_key_end = partition_version_key({instance_id, db_id, table_id, INT64_MAX});
std::unique_ptr<RangeGetIterator> iter;
- ASSERT_EQ(txn->get(key_begin, key_end, &iter), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(partition_key_begin, partition_key_end, &iter), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(iter->size(), 1);
+ auto [pk, pv] = iter->next();
+ EXPECT_EQ(pk, partition_version_key({instance_id, db_id, table_id, 30006}));
+ // Table 10000's table version must not be deleted
+ auto table_key_begin = table_version_key({instance_id, db_id, 0});
+ auto table_key_end = table_version_key({instance_id, db_id, INT64_MAX});
+ ASSERT_EQ(txn->get(table_key_begin, table_key_end, &iter), TxnErrorCode::TXN_OK);
ASSERT_EQ(iter->size(), 1);
- auto [k, v] = iter->next();
- EXPECT_EQ(k, version_key({instance_id, db_id, table_id, 30006}));
+ auto [tk, tv] = iter->next();
+ EXPECT_EQ(tk, table_version_key({instance_id, db_id, 10000}));
// Drop indexes
for (auto index_id : index_ids) {
@@ -1167,7 +1190,9 @@ TEST(RecyclerTest, recycle_versions) {
// `recycle_versions` should delete all version kvs of the dropped table
ASSERT_EQ(recycler.recycle_versions(), 0);
ASSERT_EQ(txn_kv->create_txn(&txn), TxnErrorCode::TXN_OK);
- ASSERT_EQ(txn->get(key_begin, key_end, &iter), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(txn->get(partition_key_begin, partition_key_end, &iter), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(iter->size(), 0);
+ ASSERT_EQ(txn->get(table_key_begin, table_key_end, &iter), TxnErrorCode::TXN_OK);
ASSERT_EQ(iter->size(), 0);
}
@@ -1790,9 +1815,13 @@ TEST(RecyclerTest, recycle_deleted_instance) {
for (size_t i = 0; i < 100; i++) {
ASSERT_EQ(0, create_txn_label_kv(txn_kv.get(), fmt::format("fake_label{}", i), i));
}
- // create version key
+ // create partition version key
+ for (size_t i = 101; i < 200; i += 2) {
+ ASSERT_EQ(0, create_partition_version_kv(txn_kv.get(), i, i + 1));
+ }
+ // create table version key
for (size_t i = 101; i < 200; i += 2) {
- ASSERT_EQ(0, create_version_kv(txn_kv.get(), i, i + 1));
+ ASSERT_EQ(0, create_table_version_kv(txn_kv.get(), i));
}
// create meta key
std::vector<doris::TabletSchemaCloudPB> schemas;
@@ -1851,8 +1880,18 @@ TEST(RecyclerTest, recycle_deleted_instance) {
ASSERT_EQ(txn->get(start_txn_key, end_txn_key, &it), TxnErrorCode::TXN_OK);
ASSERT_EQ(it->size(), 0);
- std::string start_version_key = version_key({instance_id, 0, 0, 0});
- std::string end_version_key = version_key({instance_id, INT64_MAX, 0, 0});
+ std::string start_partition_version_key = partition_version_key({instance_id, 0, 0, 0});
+ std::string end_partition_version_key = partition_version_key({instance_id, INT64_MAX, 0, 0});
+ ASSERT_EQ(txn->get(start_partition_version_key, end_partition_version_key, &it), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(it->size(), 0);
+
+ std::string start_table_version_key = table_version_key({instance_id, 0, 0});
+ std::string end_table_version_key = table_version_key({instance_id, INT64_MAX, 0});
+ ASSERT_EQ(txn->get(start_table_version_key, end_table_version_key, &it), TxnErrorCode::TXN_OK);
+ ASSERT_EQ(it->size(), 0);
+
+ std::string start_version_key = version_key_prefix(instance_id);
+ std::string end_version_key = version_key_prefix(instance_id + '\x00');
ASSERT_EQ(txn->get(start_version_key, end_version_key, &it), TxnErrorCode::TXN_OK);
ASSERT_EQ(it->size(), 0);
diff --git a/conf/fe.conf b/conf/fe.conf
index 8b8c3be747d..91f7d048cb1 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -30,7 +30,7 @@ LOG_DIR = ${DORIS_HOME}/log
JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:$DORIS_HOME/log/fe.gc.log.$CUR_DATE -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M -Dlog4j2.formatMsgNoLookups=true"
# For jdk 17, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M"
+JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens java.base/jdk.internal.ref=ALL-UNNAMED"
# Set your own JAVA_HOME
# JAVA_HOME=/path/to/jdk/
diff --git a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-VIEWS.md b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-VIEWS.md
new file mode 100644
index 00000000000..5622be47fa3
--- /dev/null
+++ b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-VIEWS.md
@@ -0,0 +1,78 @@
+---
+{
+ "title": "SHOW-VIEWS",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## SHOW-VIEWS
+
+### Name
+
+SHOW VIEWS
+
+### Description
+
+This statement is used to display all logical views under the current db
+
+grammar:
+
+```sql
+SHOW [FULL] VIEWS [LIKE]
+````
+
+illustrate:
+
+1. LIKE: Fuzzy query can be performed according to the table name
+
+### Example
+
+ 1. Desplay all views under DB
+
+ ```sql
+ MySQL [test]> show views;
+ +----------------+
+ | Tables_in_test |
+ +----------------+
+ | t1_view |
+ | t2_view |
+ +----------------+
+ 2 rows in set (0.00 sec)
+ ```
+
+2. Fuzzy query by view name
+
+ ```sql
+ MySQL [test]> show views like '%t1%';
+ +----------------+
+ | Tables_in_test |
+ +----------------+
+ | t1_view |
+ +----------------+
+ 1 row in set (0.01 sec)
+ ```
+
+### Keywords
+
+ SHOW, VIEWS
+
+### Best Practice
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-VIEWS.md b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-VIEWS.md
new file mode 100644
index 00000000000..41262dba693
--- /dev/null
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-VIEWS.md
@@ -0,0 +1,78 @@
+---
+{
+ "title": "SHOW-VIEWS",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## SHOW-VIEWS
+
+### Name
+
+SHOW VIEWS
+
+### Description
+
+该语句用于展示当前 db 下所有的 logical view
+
+语法：
+
+```sql
+SHOW [FULL] VIEWS [LIKE] | [WHERE where_condition]
+```
+
+说明:
+
+1. LIKE：可按照表名进行模糊查询
+
+### Example
+
+ 1. 查看DB下所有逻辑视图
+
+ ```sql
+ MySQL [test]> show views;
+ +----------------+
+ | Tables_in_test |
+ +----------------+
+ | t1_view |
+ | t2_view |
+ +----------------+
+ 2 rows in set (0.00 sec)
+ ```
+
+2. 按照VIEW名进行模糊查询
+
+ ```sql
+ MySQL [test]> show views like '%t1%';
+ +----------------+
+ | Tables_in_test |
+ +----------------+
+ | t1_view |
+ +----------------+
+ 1 row in set (0.01 sec)
+ ```
+
+### Keywords
+
+ SHOW, VIEWS
+
+### Best Practice
diff --git a/fe/be-java-extensions/hudi-scanner/src/main/resources/package.xml b/fe/be-java-extensions/hudi-scanner/src/main/resources/package.xml
index 206b7d3a7ca..4bbb2610603 100644
--- a/fe/be-java-extensions/hudi-scanner/src/main/resources/package.xml
+++ b/fe/be-java-extensions/hudi-scanner/src/main/resources/package.xml
@@ -25,13 +25,17 @@ under the License.
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
- <fileSets>
- <fileSet>
- <directory>${project.build.outputDirectory}</directory>
+ <dependencySets>
+ <dependencySet>
<outputDirectory>/</outputDirectory>
- <includes>
- <include>**/*.class</include>
- </includes>
- </fileSet>
- </fileSets>
+ <useProjectArtifact>true</useProjectArtifact>
+ <unpack>true</unpack>
+ <scope>runtime</scope>
+ <unpackOptions>
+ <excludes>
+ <exclude>**/Log4j2Plugins.dat</exclude>
+ </excludes>
+ </unpackOptions>
+ </dependencySet>
+ </dependencySets>
</assembly>
diff --git a/fe/be-java-extensions/java-common/src/main/resources/package.xml b/fe/be-java-extensions/java-common/src/main/resources/package.xml
index 206b7d3a7ca..4bbb2610603 100644
--- a/fe/be-java-extensions/java-common/src/main/resources/package.xml
+++ b/fe/be-java-extensions/java-common/src/main/resources/package.xml
@@ -25,13 +25,17 @@ under the License.
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
- <fileSets>
- <fileSet>
- <directory>${project.build.outputDirectory}</directory>
+ <dependencySets>
+ <dependencySet>
<outputDirectory>/</outputDirectory>
- <includes>
- <include>**/*.class</include>
- </includes>
- </fileSet>
- </fileSets>
+ <useProjectArtifact>true</useProjectArtifact>
+ <unpack>true</unpack>
+ <scope>runtime</scope>
+ <unpackOptions>
+ <excludes>
+ <exclude>**/Log4j2Plugins.dat</exclude>
+ </excludes>
+ </unpackOptions>
+ </dependencySet>
+ </dependencySets>
</assembly>
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/resources/package.xml b/fe/be-java-extensions/jdbc-scanner/src/main/resources/package.xml
index 206b7d3a7ca..4bbb2610603 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/resources/package.xml
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/resources/package.xml
@@ -25,13 +25,17 @@ under the License.
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
- <fileSets>
- <fileSet>
- <directory>${project.build.outputDirectory}</directory>
+ <dependencySets>
+ <dependencySet>
<outputDirectory>/</outputDirectory>
- <includes>
- <include>**/*.class</include>
- </includes>
- </fileSet>
- </fileSets>
+ <useProjectArtifact>true</useProjectArtifact>
+ <unpack>true</unpack>
+ <scope>runtime</scope>
+ <unpackOptions>
+ <excludes>
+ <exclude>**/Log4j2Plugins.dat</exclude>
+ </excludes>
+ </unpackOptions>
+ </dependencySet>
+ </dependencySets>
</assembly>
diff --git a/fe/be-java-extensions/max-compute-scanner/src/main/resources/package.xml b/fe/be-java-extensions/max-compute-scanner/src/main/resources/package.xml
index 206b7d3a7ca..4bbb2610603 100644
--- a/fe/be-java-extensions/max-compute-scanner/src/main/resources/package.xml
+++ b/fe/be-java-extensions/max-compute-scanner/src/main/resources/package.xml
@@ -25,13 +25,17 @@ under the License.
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
- <fileSets>
- <fileSet>
- <directory>${project.build.outputDirectory}</directory>
+ <dependencySets>
+ <dependencySet>
<outputDirectory>/</outputDirectory>
- <includes>
- <include>**/*.class</include>
- </includes>
- </fileSet>
- </fileSets>
+ <useProjectArtifact>true</useProjectArtifact>
+ <unpack>true</unpack>
+ <scope>runtime</scope>
+ <unpackOptions>
+ <excludes>
+ <exclude>**/Log4j2Plugins.dat</exclude>
+ </excludes>
+ </unpackOptions>
+ </dependencySet>
+ </dependencySets>
</assembly>
diff --git a/fe/be-java-extensions/paimon-scanner/src/main/resources/package.xml b/fe/be-java-extensions/paimon-scanner/src/main/resources/package.xml
index 206b7d3a7ca..4bbb2610603 100644
--- a/fe/be-java-extensions/paimon-scanner/src/main/resources/package.xml
+++ b/fe/be-java-extensions/paimon-scanner/src/main/resources/package.xml
@@ -25,13 +25,17 @@ under the License.
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
- <fileSets>
- <fileSet>
- <directory>${project.build.outputDirectory}</directory>
+ <dependencySets>
+ <dependencySet>
<outputDirectory>/</outputDirectory>
- <includes>
- <include>**/*.class</include>
- </includes>
- </fileSet>
- </fileSets>
+ <useProjectArtifact>true</useProjectArtifact>
+ <unpack>true</unpack>
+ <scope>runtime</scope>
+ <unpackOptions>
+ <excludes>
+ <exclude>**/Log4j2Plugins.dat</exclude>
+ </excludes>
+ </unpackOptions>
+ </dependencySet>
+ </dependencySets>
</assembly>
diff --git a/fe/be-java-extensions/preload-extensions/src/main/resources/package-deps.xml b/fe/be-java-extensions/preload-extensions/src/main/resources/package-deps.xml
index 3ace22a7870..1fe322c0bf3 100644
--- a/fe/be-java-extensions/preload-extensions/src/main/resources/package-deps.xml
+++ b/fe/be-java-extensions/preload-extensions/src/main/resources/package-deps.xml
@@ -31,6 +31,11 @@ under the License.
<useProjectArtifact>false</useProjectArtifact>
<unpack>true</unpack>
<scope>runtime</scope>
+ <unpackOptions>
+ <excludes>
+ <exclude>**/Log4j2Plugins.dat</exclude>
+ </excludes>
+ </unpackOptions>
</dependencySet>
</dependencySets>
</assembly>
diff --git a/fe/be-java-extensions/trino-connector-scanner/pom.xml b/fe/be-java-extensions/trino-connector-scanner/pom.xml
index c9c821820a2..2c8b91e0a6c 100644
--- a/fe/be-java-extensions/trino-connector-scanner/pom.xml
+++ b/fe/be-java-extensions/trino-connector-scanner/pom.xml
@@ -33,11 +33,6 @@ under the License.
</properties>
<dependencies>
- <dependency>
- <groupId>org.apache.doris</groupId>
- <artifactId>hudi-scanner</artifactId>
- <version>${project.version}</version>
- </dependency>
<dependency>
<groupId>org.apache.doris</groupId>
<artifactId>java-common</artifactId>
@@ -47,6 +42,11 @@ under the License.
<groupId>io.trino</groupId>
<artifactId>trino-main</artifactId>
</dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ <version>${commons-io.version}</version>
+ </dependency>
</dependencies>
<build>
diff --git a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/ProcessUtils.java b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/ProcessUtils.java
new file mode 100644
index 00000000000..429b268e01b
--- /dev/null
+++ b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/ProcessUtils.java
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.trinoconnector;
+
+import org.apache.commons.io.FileUtils;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.lang.management.ManagementFactory;
+import java.util.LinkedList;
+import java.util.List;
+
+/**
+ * Utils for handling processes
+ */
+public class ProcessUtils {
+ public static long getCurrentProcId() {
+ try {
+ return ManagementFactory.getRuntimeMXBean().getPid();
+ } catch (Exception e) {
+ throw new RuntimeException("Couldn't find PID of current JVM process.", e);
+ }
+ }
+
+ public static List<Long> getChildProcessIds(long pid) {
+ try {
+ Process pgrep = (new ProcessBuilder("pgrep", "-P", String.valueOf(pid))).start();
+ BufferedReader reader = new BufferedReader(new InputStreamReader(pgrep.getInputStream()));
+ List<Long> result = new LinkedList<>();
+ String line;
+ while ((line = reader.readLine()) != null) {
+ result.add(Long.valueOf(line.trim()));
+ }
+ pgrep.waitFor();
+ return result;
+ } catch (Exception e) {
+ throw new RuntimeException("Couldn't get child processes of PID " + pid, e);
+ }
+ }
+
+ public static String getCommandLine(long pid) {
+ try {
+ return FileUtils.readFileToString(new File(String.format("/proc/%d/cmdline", pid))).trim();
+ } catch (IOException e) {
+ return null;
+ }
+ }
+
+ public static void killProcess(long pid) {
+ try {
+ Process kill = (new ProcessBuilder("kill", "-9", String.valueOf(pid))).start();
+ kill.waitFor();
+ } catch (Exception e) {
+ throw new RuntimeException("Couldn't kill process PID " + pid, e);
+ }
+ }
+}
diff --git a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorCache.java b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorCache.java
index 0a076716b2f..e9d92a98f78 100644
--- a/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorCache.java
+++ b/fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorCache.java
@@ -17,8 +17,6 @@
package org.apache.doris.trinoconnector;
-import org.apache.doris.hudi.Utils;
-
import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
@@ -115,12 +113,12 @@ public class TrinoConnectorCache {
ScheduledExecutorService executorService = Executors.newScheduledThreadPool(1);
executorService.scheduleAtFixedRate(() -> {
if (!isKilled.get()) {
- List<Long> pids = Utils.getChildProcessIds(
- Utils.getCurrentProcId());
+ List<Long> pids = ProcessUtils.getChildProcessIds(
+ ProcessUtils.getCurrentProcId());
for (long pid : pids) {
- String cmd = Utils.getCommandLine(pid);
+ String cmd = ProcessUtils.getCommandLine(pid);
if (cmd != null && cmd.contains("org.openjdk.jol.vm.sa.AttachMain")) {
- Utils.killProcess(pid);
+ ProcessUtils.killProcess(pid);
isKilled.set(true);
}
}
diff --git a/fe/be-java-extensions/trino-connector-scanner/src/main/resources/package.xml b/fe/be-java-extensions/trino-connector-scanner/src/main/resources/package.xml
index 98fb269a547..c4f838e54d1 100644
--- a/fe/be-java-extensions/trino-connector-scanner/src/main/resources/package.xml
+++ b/fe/be-java-extensions/trino-connector-scanner/src/main/resources/package.xml
@@ -23,20 +23,11 @@ under the License.
<format>jar</format>
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
- <fileSets>
- <fileSet>
- <directory>${project.build.outputDirectory}</directory>
- <outputDirectory>/</outputDirectory>
- <includes>
- <include>**/*.class</include>
- </includes>
- </fileSet>
- </fileSets>
- <!--dependencySets>
+ <dependencySets>
<dependencySet>
<outputDirectory>/</outputDirectory>
<useProjectArtifact>true</useProjectArtifact>
- <unpack>false</unpack>
+ <unpack>true</unpack>
<scope>runtime</scope>
<unpackOptions>
<excludes>
@@ -44,5 +35,5 @@ under the License.
</excludes>
</unpackOptions>
</dependencySet>
- </dependencySets-->
-</assembly>
+ </dependencySets>
+</assembly>
\ No newline at end of file
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 8639f62af45..a0e7311af0b 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2341,10 +2341,16 @@ public class Config extends ConfigBase {
})
public static int autobucket_min_buckets = 1;
- @ConfField(description = {"Arrow Flight Server中所有用户token的缓存上限，超过后LRU淘汰，默认值为2000",
+ @ConfField(description = {"Arrow Flight Server中所有用户token的缓存上限，超过后LRU淘汰，默认值为512, "
+ + "并强制限制小于 qe_max_connection/2, 避免`Reach limit of connections`, "
+ + "因为arrow flight sql是无状态的协议，连接通常不会主动断开，"
+ + "bearer token 从 cache 淘汰的同时会 unregister Connection.",
"The cache limit of all user tokens in Arrow Flight Server. which will be eliminated by"
- + "LRU rules after exceeding the limit, the default value is 2000."})
- public static int arrow_flight_token_cache_size = 2000;
+ + "LRU rules after exceeding the limit, the default value is 512, the mandatory limit is "
+ + "less than qe_max_connection/2 to avoid `Reach limit of connections`, "
+ + "because arrow flight sql is a stateless protocol, the connection is usually not actively "
+ + "disconnected, bearer token is evict from the cache will unregister ConnectContext."})
+ public static int arrow_flight_token_cache_size = 512;
@ConfField(description = {"Arrow Flight Server中用户token的存活时间，自上次写入后过期时间，单位分钟，默认值为4320，即3天",
"The alive time of the user token in Arrow Flight Server, expire after write, unit minutes,"
diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index 21dd17d60b4..307f9046cc6 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -35,7 +35,7 @@ under the License.
<antlr4.version>4.13.1</antlr4.version>
<awssdk.version>2.20.131</awssdk.version>
<huaweiobs.version>3.1.1-hw-46</huaweiobs.version>
- <tencentcos.version>3.3.5</tencentcos.version>
+ <tencentcos.version>8.2.7</tencentcos.version>
</properties>
<profiles>
<profile>
@@ -418,9 +418,15 @@ under the License.
<artifactId>hadoop-aliyun</artifactId>
</dependency>
<dependency>
- <groupId>org.apache.hadoop</groupId>
+ <groupId>com.qcloud.cos</groupId>
<artifactId>hadoop-cos</artifactId>
<version>${tencentcos.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.json</groupId>
+ <artifactId>json</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>com.aliyun.odps</groupId>
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
index 256888c2fc1..33354199e67 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
@@ -548,6 +548,7 @@ VALUE: 'VALUE';
VALUES: 'VALUES';
VARCHAR: 'VARCHAR';
VARIABLES: 'VARIABLES';
+VARIANT: 'VARIANT';
VERBOSE: 'VERBOSE';
VERSION: 'VERSION';
VIEW: 'VIEW';
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 7aaf9fbde65..0b3a5b5c5b7 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -899,6 +899,7 @@ primitiveColType:
| type=DECIMALV3
| type=IPV4
| type=IPV6
+ | type=VARIANT
| type=ALL
;
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup b/fe/fe-core/src/main/cup/sql_parser.cup
index d18f00146b9..9738b361c6b 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -45,6 +45,7 @@ import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.MapType;
import org.apache.doris.catalog.StructField;
import org.apache.doris.catalog.StructType;
+import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.catalog.View;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.FeConstants;
@@ -659,6 +660,7 @@ terminal String
KW_VERBOSE,
KW_VERSION,
KW_VIEW,
+ KW_VIEWS,
KW_WARNINGS,
KW_WEEK,
KW_WHEN,
@@ -3992,6 +3994,16 @@ show_param ::=
{:
RESULT = new ShowTableStmt(db, ctl, parser.isVerbose, parser.wild, parser.where);
:}
+ /* show views */
+ | opt_full KW_VIEWS opt_db:db opt_wild_where
+ {:
+ RESULT = new ShowTableStmt(db, null, parser.isVerbose, TableType.VIEW, parser.wild, parser.where);
+ :}
+ /* show views */
+ | opt_full KW_VIEWS from_or_in ident:ctl DOT ident:db opt_wild_where
+ {:
+ RESULT = new ShowTableStmt(db, ctl, parser.isVerbose, TableType.VIEW, parser.wild, parser.where);
+ :}
/* show table id */
| KW_TABLE INTEGER_LITERAL:tableId
{:
@@ -8080,6 +8092,8 @@ keyword ::=
{: RESULT = id; :}
| KW_VIEW:id
{: RESULT = id; :}
+ | KW_VIEWS:id
+ {: RESULT = id; :}
| KW_WARNINGS:id
{: RESULT = id; :}
| KW_WORK:id
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/CloudRollupJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/CloudRollupJobV2.java
index 60980d4699f..71ba5ef35cf 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/CloudRollupJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/CloudRollupJobV2.java
@@ -92,7 +92,7 @@ public class CloudRollupJobV2 extends RollupJobV2 {
rollupIndexList.add(rollupIndexId);
try {
((CloudInternalCatalog) Env.getCurrentInternalCatalog())
- .commitMaterializedIndex(tableId, rollupIndexList);
+ .commitMaterializedIndex(dbId, tableId, rollupIndexList, false);
} catch (Exception e) {
LOG.warn("commitMaterializedIndex Exception:{}", e);
throw new AlterCancelException(e.getMessage());
@@ -110,7 +110,7 @@ public class CloudRollupJobV2 extends RollupJobV2 {
while (true) {
try {
((CloudInternalCatalog) Env.getCurrentInternalCatalog())
- .dropMaterializedIndex(tableId, rollupIndexList);
+ .dropMaterializedIndex(tableId, rollupIndexList, false);
break;
} catch (Exception e) {
LOG.warn("tryTimes:{}, onCancel exception:", tryTimes, e);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/CloudSchemaChangeJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/CloudSchemaChangeJobV2.java
index 0739da03642..8e3a198c2a4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/CloudSchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/CloudSchemaChangeJobV2.java
@@ -83,8 +83,7 @@ public class CloudSchemaChangeJobV2 extends SchemaChangeJobV2 {
indexIdMap.keySet().stream().collect(Collectors.toList());
try {
((CloudInternalCatalog) Env.getCurrentInternalCatalog())
- .commitMaterializedIndex(tableId,
- shadowIdxList);
+ .commitMaterializedIndex(dbId, tableId, shadowIdxList, false);
} catch (Exception e) {
LOG.warn("commitMaterializedIndex exception:", e);
throw new AlterCancelException(e.getMessage());
@@ -110,7 +109,7 @@ public class CloudSchemaChangeJobV2 extends SchemaChangeJobV2 {
while (true) {
try {
((CloudInternalCatalog) Env.getCurrentInternalCatalog())
- .dropMaterializedIndex(tableId, idxList);
+ .dropMaterializedIndex(tableId, idxList, false);
break;
} catch (Exception e) {
LOG.warn("tryTimes:{}, dropIndex exception:", tryTimes, e);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
index fccc517f7a9..2960d9a0744 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
@@ -459,7 +459,7 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
if (indexColumnMap.containsKey(SchemaChangeHandler.SHADOW_NAME_PREFIX + column.getName())) {
Column newColumn = indexColumnMap
.get(SchemaChangeHandler.SHADOW_NAME_PREFIX + column.getName());
- if (newColumn.getType() != column.getType()) {
+ if (!newColumn.getType().equals(column.getType())) {
try {
SlotRef slot = new SlotRef(destSlotDesc);
slot.setCol(column.getName());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
index e503e093787..57e00f5ab14 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SystemHandler.java
@@ -33,24 +33,34 @@ import org.apache.doris.analysis.ModifyBrokerClause;
import org.apache.doris.analysis.ModifyFrontendHostNameClause;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.MysqlCompatibleDatabase;
import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.ReplicaAllocation;
+import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TabletInvertedIndex;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.NetUtils;
import org.apache.doris.ha.FrontendNodeType;
+import org.apache.doris.resource.Tag;
import org.apache.doris.system.Backend;
import org.apache.doris.system.SystemInfoService;
import org.apache.doris.system.SystemInfoService.HostInfo;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
import org.apache.commons.lang3.NotImplementedException;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
/*
* SystemHandler is for
@@ -235,7 +245,8 @@ public class SystemHandler extends AlterHandler {
decommissionBackends.add(backend);
}
- // TODO(cmy): check if replication num can be met
+ checkDecommissionWithReplicaAllocation(decommissionBackends);
+
// TODO(cmy): check remaining space
return decommissionBackends;
@@ -258,12 +269,81 @@ public class SystemHandler extends AlterHandler {
decommissionBackends.add(backend);
}
- // TODO(cmy): check if replication num can be met
+ checkDecommissionWithReplicaAllocation(decommissionBackends);
+
// TODO(cmy): check remaining space
return decommissionBackends;
}
+ private static void checkDecommissionWithReplicaAllocation(List<Backend> decommissionBackends)
+ throws DdlException {
+ if (Config.isCloudMode() || decommissionBackends.isEmpty()
+ || DebugPointUtil.isEnable("SystemHandler.decommission_no_check_replica_num")) {
+ return;
+ }
+
+ Set<Tag> decommissionTags = decommissionBackends.stream().map(be -> be.getLocationTag())
+ .collect(Collectors.toSet());
+ Map<Tag, Integer> tagAvailBackendNums = Maps.newHashMap();
+ for (Backend backend : Env.getCurrentSystemInfo().getAllBackends()) {
+ long beId = backend.getId();
+ if (!backend.isScheduleAvailable()
+ || decommissionBackends.stream().anyMatch(be -> be.getId() == beId)) {
+ continue;
+ }
+
+ Tag tag = backend.getLocationTag();
+ if (tag != null) {
+ tagAvailBackendNums.put(tag, tagAvailBackendNums.getOrDefault(tag, 0) + 1);
+ }
+ }
+
+ Env env = Env.getCurrentEnv();
+ List<Long> dbIds = env.getInternalCatalog().getDbIds();
+ for (Long dbId : dbIds) {
+ Database db = env.getInternalCatalog().getDbNullable(dbId);
+ if (db == null) {
+ continue;
+ }
+
+ if (db instanceof MysqlCompatibleDatabase) {
+ continue;
+ }
+
+ for (Table table : db.getTables()) {
+ table.readLock();
+ try {
+ if (!table.needSchedule()) {
+ continue;
+ }
+
+ OlapTable tbl = (OlapTable) table;
+ for (Partition partition : tbl.getAllPartitions()) {
+ ReplicaAllocation replicaAlloc = tbl.getPartitionInfo().getReplicaAllocation(partition.getId());
+ for (Map.Entry<Tag, Short> entry : replicaAlloc.getAllocMap().entrySet()) {
+ Tag tag = entry.getKey();
+ if (!decommissionTags.contains(tag)) {
+ continue;
+ }
+ int replicaNum = (int) entry.getValue();
+ int backendNum = tagAvailBackendNums.getOrDefault(tag, 0);
+ if (replicaNum > backendNum) {
+ throw new DdlException("After decommission, partition " + partition.getName()
+ + " of table " + db.getName() + "." + tbl.getName()
+ + " 's replication allocation { " + replicaAlloc
+ + " } > available backend num " + backendNum + " on tag " + tag
+ + ", otherwise need to decrease the partition's replication num.");
+ }
+ }
+ }
+ } finally {
+ table.readUnlock();
+ }
+ }
+ }
+ }
+
@Override
public synchronized void cancel(CancelStmt stmt) throws DdlException {
CancelAlterSystemStmt cancelAlterSystemStmt = (CancelAlterSystemStmt) stmt;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
index 0a60a3060c3..9021402d48a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
@@ -18,6 +18,8 @@
package org.apache.doris.analysis;
import org.apache.doris.catalog.Env;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeNameFormat;
@@ -60,6 +62,11 @@ public class CreateRoleStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
+ if (Config.access_controller_type.equalsIgnoreCase("ranger-doris")) {
+ throw new AnalysisException("Create role is prohibited when Ranger is enabled.");
+ }
+
FeNameFormat.checkRoleName(role, false /* can not be admin */, "Can not create role");
// check if current user has GRANT priv on GLOBAL level.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateUserStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateUserStmt.java
index 9446f918c23..e64a5e2e7f8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateUserStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateUserStmt.java
@@ -19,9 +19,12 @@ package org.apache.doris.analysis;
import org.apache.doris.catalog.Env;
import org.apache.doris.cluster.ClusterNamespace;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeNameFormat;
+import org.apache.doris.common.LdapConfig;
import org.apache.doris.common.UserException;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.mysql.privilege.Role;
@@ -142,6 +145,11 @@ public class CreateUserStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
+ if (Config.access_controller_type.equalsIgnoreCase("ranger-doris") && LdapConfig.ldap_authentication_enabled) {
+ throw new AnalysisException("Create user is prohibited when Ranger and LDAP are enabled at same time.");
+ }
+
userIdent.analyze();
if (userIdent.isRootUser()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateWorkloadGroupStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateWorkloadGroupStmt.java
index 3d48cad5cd6..92a60a94e55 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateWorkloadGroupStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateWorkloadGroupStmt.java
@@ -26,6 +26,7 @@ import org.apache.doris.common.UserException;
import org.apache.doris.common.util.PrintableMap;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.resource.workloadgroup.WorkloadGroup;
import java.util.Map;
@@ -69,6 +70,11 @@ public class CreateWorkloadGroupStmt extends DdlStmt {
if (properties == null || properties.isEmpty()) {
throw new AnalysisException("Resource group properties can't be null");
}
+
+ String wgTag = properties.get(WorkloadGroup.TAG);
+ if (wgTag != null) {
+ FeNameFormat.checkCommonName("workload group tag", wgTag);
+ }
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java
index df087432a0a..468b86579f4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java
@@ -18,6 +18,8 @@
package org.apache.doris.analysis;
import org.apache.doris.catalog.Env;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeNameFormat;
@@ -50,6 +52,11 @@ public class DropRoleStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
+ if (Config.access_controller_type.equalsIgnoreCase("ranger-doris")) {
+ throw new AnalysisException("Drop role is prohibited when Ranger is enabled.");
+ }
+
FeNameFormat.checkRoleName(role, false /* can not be superuser */, "Can not drop role");
// check if current user has GRANT priv on GLOBAL level.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java
index 61b9fdc7ab5..4b8196ad638 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java
@@ -19,8 +19,10 @@ package org.apache.doris.analysis;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.LdapConfig;
import org.apache.doris.common.UserException;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
@@ -53,6 +55,11 @@ public class DropUserStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws AnalysisException, UserException {
super.analyze(analyzer);
+
+ if (Config.access_controller_type.equalsIgnoreCase("ranger-doris") && LdapConfig.ldap_authentication_enabled) {
+ throw new AnalysisException("Drop user is prohibited when Ranger and LDAP are enabled at same time.");
+ }
+
userIdent.analyze();
if (userIdent.isRootUser()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
index 7962eb822b8..ae033a6ff39 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/GrantStmt.java
@@ -20,6 +20,7 @@ package org.apache.doris.analysis;
import org.apache.doris.catalog.AccessPrivilegeWithCols;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeNameFormat;
@@ -137,6 +138,11 @@ public class GrantStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
+ if (Config.access_controller_type.equalsIgnoreCase("ranger-doris")) {
+ throw new AnalysisException("Grant is prohibited when Ranger is enabled.");
+ }
+
if (userIdent != null) {
userIdent.analyze();
} else {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ResourcePattern.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ResourcePattern.java
index 6afeae3f4b8..59c70f770a5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ResourcePattern.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ResourcePattern.java
@@ -22,6 +22,7 @@ import org.apache.doris.common.FeNameFormat;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.mysql.privilege.Auth.PrivLevel;
+import org.apache.doris.persist.gson.GsonPostProcessable;
import org.apache.doris.persist.gson.GsonUtils;
import com.google.common.base.Strings;
@@ -34,7 +35,7 @@ import java.io.IOException;
// only the following 2 formats are allowed
// *
// resource
-public class ResourcePattern implements Writable {
+public class ResourcePattern implements Writable, GsonPostProcessable {
@SerializedName(value = "resourceName")
private String resourceName;
@@ -48,9 +49,9 @@ public class ResourcePattern implements Writable {
public static ResourcePattern ALL_STAGE;
static {
- ALL_GENERAL = new ResourcePattern("*", ResourceTypeEnum.GENERAL);
- ALL_CLUSTER = new ResourcePattern("*", ResourceTypeEnum.CLUSTER);
- ALL_STAGE = new ResourcePattern("*", ResourceTypeEnum.STAGE);
+ ALL_GENERAL = new ResourcePattern("%", ResourceTypeEnum.GENERAL);
+ ALL_CLUSTER = new ResourcePattern("%", ResourceTypeEnum.CLUSTER);
+ ALL_STAGE = new ResourcePattern("%", ResourceTypeEnum.STAGE);
try {
ALL_GENERAL.analyze();
@@ -65,7 +66,11 @@ public class ResourcePattern implements Writable {
}
public ResourcePattern(String resourceName, ResourceTypeEnum type) {
- this.resourceName = Strings.isNullOrEmpty(resourceName) ? "*" : resourceName;
+ // To be compatible with previous syntax
+ if ("*".equals(resourceName)) {
+ resourceName = "%";
+ }
+ this.resourceName = Strings.isNullOrEmpty(resourceName) ? "%" : resourceName;
resourceType = type;
}
@@ -86,15 +91,11 @@ public class ResourcePattern implements Writable {
}
public PrivLevel getPrivLevel() {
- if (resourceName.equals("*")) {
- return PrivLevel.GLOBAL;
- } else {
- return PrivLevel.RESOURCE;
- }
+ return PrivLevel.RESOURCE;
}
public void analyze() throws AnalysisException {
- if (!resourceName.equals("*")) {
+ if (!resourceName.equals("%")) {
FeNameFormat.checkResourceName(resourceName, resourceType);
}
}
@@ -130,4 +131,12 @@ public class ResourcePattern implements Writable {
String json = Text.readString(in);
return GsonUtils.GSON.fromJson(json, ResourcePattern.class);
}
+
+ @Override
+ public void gsonPostProcess() throws IOException {
+ // // To be compatible with previous syntax
+ if ("*".equals(resourceName)) {
+ resourceName = "%";
+ }
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
index 70a44a4625c..cbd5e0a83ef 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/RevokeStmt.java
@@ -19,6 +19,7 @@ package org.apache.doris.analysis;
import org.apache.doris.catalog.AccessPrivilegeWithCols;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.FeNameFormat;
import org.apache.doris.mysql.privilege.ColPrivilegeKey;
import org.apache.doris.mysql.privilege.PrivBitSet;
@@ -121,6 +122,10 @@ public class RevokeStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws AnalysisException {
+ if (Config.access_controller_type.equalsIgnoreCase("ranger-doris")) {
+ throw new AnalysisException("Revoke is prohibited when Ranger is enabled.");
+ }
+
if (userIdent != null) {
userIdent.analyze();
} else {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStatsStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStatsStmt.java
index 37be76b20df..a4216f55661 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStatsStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowColumnStatsStmt.java
@@ -32,6 +32,7 @@ import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.ShowResultSet;
import org.apache.doris.qe.ShowResultSetMetaData;
+import org.apache.doris.statistics.AnalysisManager;
import org.apache.doris.statistics.ColStatsMeta;
import org.apache.doris.statistics.ColumnStatistic;
@@ -138,14 +139,15 @@ public class ShowColumnStatsStmt extends ShowStmt {
public ShowResultSet constructResultSet(List<Pair<Pair<String, String>, ColumnStatistic>> columnStatistics) {
List<List<String>> result = Lists.newArrayList();
+ AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
columnStatistics.forEach(p -> {
if (p.second.isUnKnown) {
return;
}
-
List<String> row = Lists.newArrayList();
- row.add(p.first.first);
+ // p data structure is Pair<Pair<IndexName, ColumnName>, ColumnStatistic>
row.add(p.first.second);
+ row.add(p.first.first);
row.add(String.valueOf(p.second.count));
row.add(String.valueOf(p.second.ndv));
row.add(String.valueOf(p.second.numNulls));
@@ -153,8 +155,7 @@ public class ShowColumnStatsStmt extends ShowStmt {
row.add(String.valueOf(p.second.avgSizeByte));
row.add(String.valueOf(p.second.minExpr == null ? "N/A" : p.second.minExpr.toSql()));
row.add(String.valueOf(p.second.maxExpr == null ? "N/A" : p.second.maxExpr.toSql()));
- ColStatsMeta colStatsMeta = Env.getCurrentEnv().getAnalysisManager().findColStatsMeta(table.getId(),
- p.first.first);
+ ColStatsMeta colStatsMeta = analysisManager.findColStatsMeta(table.getId(), p.first.first, p.first.second);
row.add(String.valueOf(colStatsMeta == null ? "N/A" : colStatsMeta.analysisMethod));
row.add(String.valueOf(colStatsMeta == null ? "N/A" : colStatsMeta.analysisType));
row.add(String.valueOf(colStatsMeta == null ? "N/A" : colStatsMeta.jobType));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataSkewStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataSkewStmt.java
index e0e3cfdc835..a1806278358 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataSkewStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowDataSkewStmt.java
@@ -20,7 +20,6 @@ package org.apache.doris.analysis;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.ScalarType;
-import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
@@ -34,7 +33,7 @@ import com.google.common.collect.ImmutableList;
// show data skew from tbl [partition(p1, p2, ...)]
public class ShowDataSkewStmt extends ShowStmt {
public static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>()
- .add("BucketIdx").add("AvgRowCount").add("AvgDataSize")
+ .add("PartitionName").add("BucketIdx").add("AvgRowCount").add("AvgDataSize")
.add("Graph").add("Percent")
.build();
@@ -58,10 +57,6 @@ public class ShowDataSkewStmt extends ShowStmt {
ConnectContext.get().getRemoteIP(),
tblRef.getName().getDb() + "." + tblRef.getName().getTbl());
}
- PartitionNames partitionNames = tblRef.getPartitionNames();
- if (partitionNames == null || partitionNames.getPartitionNames().size() != 1) {
- throw new AnalysisException("Should specify one and only one partition");
- }
}
public String getDbName() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java
index 3dc3fa24aae..f3e12f1aa92 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java
@@ -20,6 +20,7 @@ package org.apache.doris.analysis;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.InfoSchemaDb;
import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.ErrorCode;
@@ -40,8 +41,9 @@ public class ShowTableStmt extends ShowStmt {
private static final String INVERTED_INDEX_STORAGE_FORMAT_COL = "Inverted_index_storage_format";
private String db;
private String catalog;
- private boolean isVerbose;
- private String pattern;
+ private final boolean isVerbose;
+ private TableType type;
+ private final String pattern;
private Expr where;
private SelectStmt selectStmt;
@@ -61,6 +63,12 @@ public class ShowTableStmt extends ShowStmt {
this.catalog = catalog;
}
+ public ShowTableStmt(String db, String catalog, boolean isVerbose, TableType type, String pattern,
+ Expr where) {
+ this(db, catalog, isVerbose, pattern, where);
+ this.type = type;
+ }
+
public String getDb() {
return db;
}
@@ -73,6 +81,10 @@ public class ShowTableStmt extends ShowStmt {
return isVerbose;
}
+ public TableType getType() {
+ return type;
+ }
+
public String getPattern() {
return pattern;
}
@@ -120,6 +132,11 @@ public class ShowTableStmt extends ShowStmt {
selectList.addItem(item);
aliasMap.put(new SlotRef(null, TYPE_COL), item.getExpr().clone(null));
}
+ if (type != null) {
+ BinaryPredicate viewFilter = new BinaryPredicate(BinaryPredicate.Operator.EQ,
+ new SlotRef(tablesTableName, "ENGINE"), new StringLiteral(type.toEngineName()));
+ where = CompoundPredicate.createConjunction(viewFilter, where);
+ }
where = where.substitute(aliasMap);
selectStmt = new SelectStmt(selectList,
new FromClause(Lists.newArrayList(new TableRef(tablesTableName, null))),
@@ -137,7 +154,18 @@ public class ShowTableStmt extends ShowStmt {
if (isVerbose) {
sb.append(" FULL");
}
- sb.append(" TABLES");
+ if (type != null) {
+ switch (type) {
+ // todo(only show views from now)
+ case VIEW:
+ sb.append(" VIEWS");
+ break;
+ default:
+ sb.append(" TABLES");
+ }
+ } else {
+ sb.append(" TABLES");
+ }
if (!Strings.isNullOrEmpty(db)) {
if (!Strings.isNullOrEmpty(catalog)) {
sb.append(" FROM ").append(catalog);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java
index e6134cfc31f..49e6d66f50e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java
@@ -50,6 +50,7 @@ import org.apache.doris.nereids.trees.expressions.functions.agg.MinBy;
import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctCount;
import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctGroupConcat;
import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctSum;
+import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctSum0;
import org.apache.doris.nereids.trees.expressions.functions.agg.Ndv;
import org.apache.doris.nereids.trees.expressions.functions.agg.OrthogonalBitmapIntersect;
import org.apache.doris.nereids.trees.expressions.functions.agg.OrthogonalBitmapIntersectCount;
@@ -64,6 +65,7 @@ import org.apache.doris.nereids.trees.expressions.functions.agg.SequenceMatch;
import org.apache.doris.nereids.trees.expressions.functions.agg.Stddev;
import org.apache.doris.nereids.trees.expressions.functions.agg.StddevSamp;
import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum0;
import org.apache.doris.nereids.trees.expressions.functions.agg.TopN;
import org.apache.doris.nereids.trees.expressions.functions.agg.TopNArray;
import org.apache.doris.nereids.trees.expressions.functions.agg.TopNWeighted;
@@ -118,6 +120,7 @@ public class BuiltinAggregateFunctions implements FunctionHelper {
agg(MultiDistinctCount.class, "multi_distinct_count"),
agg(MultiDistinctGroupConcat.class, "multi_distinct_group_concat"),
agg(MultiDistinctSum.class, "multi_distinct_sum"),
+ agg(MultiDistinctSum0.class, "multi_distinct_sum0"),
agg(Ndv.class, "approx_count_distinct", "ndv"),
agg(OrthogonalBitmapIntersect.class, "orthogonal_bitmap_intersect"),
agg(OrthogonalBitmapIntersectCount.class, "orthogonal_bitmap_intersect_count"),
@@ -132,6 +135,7 @@ public class BuiltinAggregateFunctions implements FunctionHelper {
agg(Stddev.class, "stddev_pop", "stddev"),
agg(StddevSamp.class, "stddev_samp"),
agg(Sum.class, "sum"),
+ agg(Sum0.class, "sum0"),
agg(TopN.class, "topn"),
agg(TopNArray.class, "topn_array"),
agg(TopNWeighted.class, "topn_weighted"),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 1b479f36463..0c286cf6dd1 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -5518,9 +5518,14 @@ public class Env {
}
}
olapTable.replaceTempPartitions(partitionNames, tempPartitionNames, isStrictRange, useTempPartitionName);
- long version = olapTable.getNextVersion();
+ long version;
long versionTime = System.currentTimeMillis();
- olapTable.updateVisibleVersionAndTime(version, versionTime);
+ if (Config.isNotCloudMode()) {
+ version = olapTable.getNextVersion();
+ olapTable.updateVisibleVersionAndTime(version, versionTime);
+ } else {
+ version = olapTable.getVisibleVersion();
+ }
// write log
ReplacePartitionOperationLog info =
new ReplacePartitionOperationLog(db.getId(), db.getFullName(), olapTable.getId(), olapTable.getName(),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
index b69d52b2c11..29ff913b8aa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java
@@ -46,6 +46,7 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
import java.util.Set;
public class FunctionSet<T> {
@@ -924,6 +925,17 @@ public class FunctionSet<T> {
// sum in multi distinct
if (t.equals(Type.BIGINT) || t.equals(Type.LARGEINT) || t.equals(Type.DOUBLE)) {
addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum", Lists.newArrayList(t),
+ t,
+ t,
+ "",
+ "",
+ "",
+ "",
+ null,
+ null,
+ "",
+ false, true, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum0", Lists.newArrayList(t),
t,
t,
"",
@@ -937,6 +949,17 @@ public class FunctionSet<T> {
} else if (t.equals(Type.MAX_DECIMALV2_TYPE)) {
// vectorized
addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum", Lists.newArrayList(t),
+ MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
+ Type.MAX_DECIMALV2_TYPE,
+ "",
+ "",
+ "",
+ "",
+ null,
+ null,
+ "",
+ false, true, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum0", Lists.newArrayList(t),
MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
Type.MAX_DECIMALV2_TYPE,
"",
@@ -950,6 +973,17 @@ public class FunctionSet<T> {
} else if (t.equals(Type.DECIMAL32)) {
// vectorized
addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum", Lists.newArrayList(t),
+ MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
+ Type.DECIMAL32,
+ "",
+ "",
+ "",
+ "",
+ null,
+ null,
+ "",
+ false, true, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum0", Lists.newArrayList(t),
MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
Type.DECIMAL32,
"",
@@ -962,6 +996,17 @@ public class FunctionSet<T> {
false, true, true, true));
} else if (t.equals(Type.DECIMAL64)) {
addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum", Lists.newArrayList(t),
+ MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
+ Type.DECIMAL64,
+ "",
+ "",
+ "",
+ "",
+ null,
+ null,
+ "",
+ false, true, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum0", Lists.newArrayList(t),
MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
Type.DECIMAL64,
"",
@@ -974,6 +1019,17 @@ public class FunctionSet<T> {
false, true, true, true));
} else if (t.equals(Type.DECIMAL128)) {
addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum", Lists.newArrayList(t),
+ MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
+ Type.DECIMAL128,
+ "",
+ "",
+ "",
+ "",
+ null,
+ null,
+ "",
+ false, true, false, true));
+ addBuiltin(AggregateFunction.createBuiltin("multi_distinct_sum0", Lists.newArrayList(t),
MULTI_DISTINCT_SUM_RETURN_TYPE.get(t),
Type.DECIMAL128,
"",
@@ -1174,86 +1230,91 @@ public class FunctionSet<T> {
}
// Sum
- String []sumNames = {"sum", "sum_distinct"};
- for (String name : sumNames) {
- addBuiltin(AggregateFunction.createBuiltin(name,
+ // functionName(String) -> returnsNonNullOnEmpty(Boolean)
+ Map<String, Boolean> sumNames = ImmutableMap.of(
+ "sum", false,
+ "sum_distinct", false,
+ "sum0", true
+ );
+ for (Entry<String, Boolean> nameWithReturn : sumNames.entrySet()) {
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.BOOLEAN), Type.BIGINT, Type.BIGINT, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
+ null, false, true, nameWithReturn.getValue(), true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.TINYINT), Type.BIGINT, Type.BIGINT, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.SMALLINT), Type.BIGINT, Type.BIGINT, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.INT), Type.BIGINT, Type.BIGINT, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.BIGINT), Type.BIGINT, Type.BIGINT, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.DOUBLE), Type.DOUBLE, Type.DOUBLE, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.MAX_DECIMALV2_TYPE), Type.MAX_DECIMALV2_TYPE, Type.MAX_DECIMALV2_TYPE, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.DECIMAL32), ScalarType.DECIMAL128, Type.DECIMAL128, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.DECIMAL64), Type.DECIMAL128, Type.DECIMAL128, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.DECIMAL128), Type.DECIMAL128, Type.DECIMAL128, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
- addBuiltin(AggregateFunction.createBuiltin(name,
+ null, false, true, nameWithReturn.getValue(), true));
+ addBuiltin(AggregateFunction.createBuiltin(nameWithReturn.getKey(),
Lists.<Type>newArrayList(Type.LARGEINT), Type.LARGEINT, Type.LARGEINT, "",
"",
"",
null, null,
"",
- null, false, true, false, true));
+ null, false, true, nameWithReturn.getValue(), true));
}
Type[] types = {Type.SMALLINT, Type.TINYINT, Type.INT, Type.BIGINT, Type.FLOAT, Type.DOUBLE, Type.CHAR,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
index 33f0d9f9a57..333b7b146ac 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java
@@ -34,6 +34,7 @@ import com.google.common.collect.Maps;
import java.text.DecimalFormat;
import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -267,55 +268,70 @@ public class MetadataViewer {
List<List<String>> result = Lists.newArrayList();
Env env = Env.getCurrentEnv();
- if (partitionNames == null || partitionNames.getPartitionNames().size() != 1) {
- throw new DdlException("Should specify one and only one partitions");
- }
-
Database db = env.getInternalCatalog().getDbOrDdlException(dbName);
OlapTable olapTable = db.getOlapTableOrDdlException(tblName);
+ if (olapTable.getPartitionNames().isEmpty()) {
+ throw new DdlException("Can not find any partition from " + dbName + "." + tblName);
+ }
+
+ // patition -> isTmep
+ Map<String, Boolean> allPartionNames = new HashMap<>();
+ if (partitionNames == null) {
+ for (Partition p : olapTable.getPartitions()) {
+ allPartionNames.put(p.getName(), false);
+ }
+ for (Partition p : olapTable.getTempPartitions()) {
+ allPartionNames.put(p.getName(), true);
+ }
+ } else {
+ for (String name : partitionNames.getPartitionNames()) {
+ allPartionNames.put(name, partitionNames.isTemp());
+ }
+ }
+
olapTable.readLock();
try {
Partition partition = null;
// check partition
- for (String partName : partitionNames.getPartitionNames()) {
- partition = olapTable.getPartition(partName, partitionNames.isTemp());
+ for (Map.Entry<String, Boolean> partName : allPartionNames.entrySet()) {
+ partition = olapTable.getPartition(partName.getKey(), partName.getValue());
if (partition == null) {
throw new DdlException("Partition does not exist: " + partName);
}
- break;
- }
- DistributionInfo distributionInfo = partition.getDistributionInfo();
- List<Long> rowCountTabletInfos = Lists.newArrayListWithCapacity(distributionInfo.getBucketNum());
- List<Long> dataSizeTabletInfos = Lists.newArrayListWithCapacity(distributionInfo.getBucketNum());
- for (long i = 0; i < distributionInfo.getBucketNum(); i++) {
- rowCountTabletInfos.add(0L);
- dataSizeTabletInfos.add(0L);
- }
+ DistributionInfo distributionInfo = partition.getDistributionInfo();
+ List<Long> rowCountTabletInfos = Lists.newArrayListWithCapacity(distributionInfo.getBucketNum());
+ List<Long> dataSizeTabletInfos = Lists.newArrayListWithCapacity(distributionInfo.getBucketNum());
+ for (long i = 0; i < distributionInfo.getBucketNum(); i++) {
+ rowCountTabletInfos.add(0L);
+ dataSizeTabletInfos.add(0L);
+ }
- long totalSize = 0;
- for (MaterializedIndex mIndex : partition.getMaterializedIndices(IndexExtState.VISIBLE)) {
- List<Long> tabletIds = mIndex.getTabletIdsInOrder();
- for (int i = 0; i < tabletIds.size(); i++) {
- Tablet tablet = mIndex.getTablet(tabletIds.get(i));
- long rowCount = tablet.getRowCount(true);
- long dataSize = tablet.getDataSize(true);
- rowCountTabletInfos.set(i, rowCountTabletInfos.get(i) + rowCount);
- dataSizeTabletInfos.set(i, dataSizeTabletInfos.get(i) + dataSize);
- totalSize += dataSize;
+ long totalSize = 0;
+ for (MaterializedIndex mIndex : partition.getMaterializedIndices(IndexExtState.VISIBLE)) {
+ List<Long> tabletIds = mIndex.getTabletIdsInOrder();
+ for (int i = 0; i < tabletIds.size(); i++) {
+ Tablet tablet = mIndex.getTablet(tabletIds.get(i));
+ long rowCount = tablet.getRowCount(true);
+ long dataSize = tablet.getDataSize(true);
+ rowCountTabletInfos.set(i, rowCountTabletInfos.get(i) + rowCount);
+ dataSizeTabletInfos.set(i, dataSizeTabletInfos.get(i) + dataSize);
+ totalSize += dataSize;
+ }
}
- }
- // graph
- for (int i = 0; i < distributionInfo.getBucketNum(); i++) {
- List<String> row = Lists.newArrayList();
- row.add(String.valueOf(i));
- row.add(rowCountTabletInfos.get(i).toString());
- row.add(dataSizeTabletInfos.get(i).toString());
- row.add(graph(dataSizeTabletInfos.get(i), totalSize));
- row.add(totalSize == dataSizeTabletInfos.get(i) ? (totalSize == 0L ? "0.00%" : "100.00%") :
- df.format((double) dataSizeTabletInfos.get(i) / totalSize));
- result.add(row);
+ // graph
+ for (int i = 0; i < distributionInfo.getBucketNum(); i++) {
+ List<String> row = Lists.newArrayList();
+ row.add(partName.getKey());
+ row.add(String.valueOf(i));
+ row.add(rowCountTabletInfos.get(i).toString());
+ row.add(dataSizeTabletInfos.get(i).toString());
+ row.add(graph(dataSizeTabletInfos.get(i), totalSize));
+ row.add(totalSize == dataSizeTabletInfos.get(i) ? (totalSize == 0L ? "0.00%" : "100.00%") :
+ df.format((double) dataSizeTabletInfos.get(i) / totalSize));
+ result.add(row);
+ }
}
} finally {
olapTable.readUnlock();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index b6239f486cd..89bba35645d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -37,6 +37,8 @@ import org.apache.doris.catalog.Tablet.TabletStatus;
import org.apache.doris.clone.TabletSchedCtx;
import org.apache.doris.clone.TabletScheduler;
import org.apache.doris.cloud.catalog.CloudPartition;
+import org.apache.doris.cloud.proto.Cloud;
+import org.apache.doris.cloud.qe.SnapshotProxy;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
@@ -46,6 +48,7 @@ import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
import org.apache.doris.common.io.DeepCopy;
import org.apache.doris.common.io.Text;
+import org.apache.doris.common.profile.SummaryProfile;
import org.apache.doris.common.util.PropertyAnalyzer;
import org.apache.doris.common.util.Util;
import org.apache.doris.mtmv.MTMVRelatedTableIf;
@@ -53,7 +56,9 @@ import org.apache.doris.mtmv.MTMVSnapshotIf;
import org.apache.doris.mtmv.MTMVVersionSnapshot;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.OriginStatement;
+import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.resource.Tag;
+import org.apache.doris.rpc.RpcException;
import org.apache.doris.statistics.AnalysisInfo;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.BaseAnalysisTask;
@@ -1279,11 +1284,11 @@ public class OlapTable extends Table implements MTMVRelatedTableIf {
if (tblStats == null) {
return true;
}
- if (!tblStats.analyzeColumns().containsAll(getBaseSchema()
+ if (!tblStats.analyzeColumns().containsAll(getColumnIndexPairs(getSchemaAllIndexes(false)
.stream()
.filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
.map(Column::getName)
- .collect(Collectors.toSet()))) {
+ .collect(Collectors.toSet())))) {
return true;
}
long rowCount = getRowCount();
@@ -1296,34 +1301,20 @@ public class OlapTable extends Table implements MTMVRelatedTableIf {
}
@Override
- public Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- TableStatsMeta tableStats = Env.getCurrentEnv().getAnalysisManager().findTableStatsStatus(getId());
- Set<String> allPartitions = getPartitionNames().stream().map(this::getPartition)
- .filter(Partition::hasData).map(Partition::getName).collect(Collectors.toSet());
- if (tableStats == null) {
- Map<String, Set<String>> ret = Maps.newHashMap();
- for (Column col : getSchemaAllIndexes(false)) {
- if (StatisticsUtil.isUnsupportedType(col.getType())) {
+ public List<Pair<String, String>> getColumnIndexPairs(Set<String> columns) {
+ List<Pair<String, String>> ret = Lists.newArrayList();
+ // Check the schema of all indexes for each given column name,
+ // If the column name exists in the index, add the <IndexName, ColumnName> pair to return list.
+ for (String column : columns) {
+ for (MaterializedIndexMeta meta : indexIdToMeta.values()) {
+ Column col = meta.getColumnByName(column);
+ if (col == null || StatisticsUtil.isUnsupportedType(col.getType())) {
continue;
}
- ret.put(col.getName(), allPartitions);
+ ret.add(Pair.of(getIndexNameById(meta.getIndexId()), column));
}
- return ret;
}
- Map<String, Set<String>> colToPart = new HashMap<>();
- for (Column col : getSchemaAllIndexes(false)) {
- if (StatisticsUtil.isUnsupportedType(col.getType())) {
- continue;
- }
- long lastUpdateTime = tableStats.findColumnLastUpdateTime(col.getName());
- Set<String> partitions = getPartitionNames().stream()
- .map(this::getPartition)
- .filter(Partition::hasData)
- .filter(partition -> partition.getVisibleVersionTime() >= lastUpdateTime).map(Partition::getName)
- .collect(Collectors.toSet());
- colToPart.put(col.getName(), partitions);
- }
- return colToPart;
+ return ret;
}
@Override
@@ -2666,11 +2657,48 @@ public class OlapTable extends Table implements MTMVRelatedTableIf {
// During `getNextVersion` and `updateVisibleVersionAndTime` period,
// the write lock on the table should be held continuously
public long getNextVersion() {
- return tableAttributes.getNextVersion();
+ if (!Config.isCloudMode()) {
+ return tableAttributes.getNextVersion();
+ } else {
+ // cloud mode should not reach here
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("getNextVersion in Cloud mode in OlapTable {} ", getName());
+ }
+ return getVisibleVersion() + 1;
+ }
}
public long getVisibleVersion() {
- return tableAttributes.getVisibleVersion();
+ if (Config.isNotCloudMode()) {
+ return tableAttributes.getVisibleVersion();
+ }
+ // get version rpc
+ Cloud.GetVersionRequest request = Cloud.GetVersionRequest.newBuilder()
+ .setDbId(this.getDatabase().getId())
+ .setTableId(this.id)
+ .setBatchMode(false)
+ .setIsTableVersion(true)
+ .build();
+
+ try {
+ Cloud.GetVersionResponse resp = getVersionFromMeta(request);
+ long version = -1;
+ if (resp.getStatus().getCode() == Cloud.MetaServiceCode.OK) {
+ version = resp.getVersion();
+ } else {
+ assert resp.getStatus().getCode() == Cloud.MetaServiceCode.VERSION_NOT_FOUND;
+ version = 0;
+ }
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("get version from meta service, version: {}, table: {}", version, getId());
+ }
+ if (version == 0) {
+ version = 1;
+ }
+ return version;
+ } catch (RpcException e) {
+ throw new RuntimeException("get version from meta service failed");
+ }
}
public long getVisibleVersionTime() {
@@ -2719,6 +2747,19 @@ public class OlapTable extends Table implements MTMVRelatedTableIf {
}
}
+ private static Cloud.GetVersionResponse getVersionFromMeta(Cloud.GetVersionRequest req)
+ throws RpcException {
+ long startAt = System.nanoTime();
+ try {
+ return SnapshotProxy.getVisibleVersion(req);
+ } finally {
+ SummaryProfile profile = getSummaryProfile();
+ if (profile != null) {
+ profile.addGetTableVersionTime(System.nanoTime() - startAt);
+ }
+ }
+ }
+
@Override
public boolean needAutoRefresh() {
return true;
@@ -2726,6 +2767,17 @@ public class OlapTable extends Table implements MTMVRelatedTableIf {
@Override
public boolean isPartitionColumnAllowNull() {
- return false;
+ return true;
+ }
+
+ private static SummaryProfile getSummaryProfile() {
+ ConnectContext ctx = ConnectContext.get();
+ if (ctx != null) {
+ StmtExecutor executor = ctx.getExecutor();
+ if (executor != null) {
+ return executor.getSummaryProfile();
+ }
+ }
+ return null;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
index f06c05839f9..89d1c363c10 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java
@@ -486,6 +486,7 @@ public class SchemaTable extends Table {
.column("MIN_REMOTE_SCAN_THREAD_NUM", ScalarType.createType(PrimitiveType.BIGINT))
.column("SPILL_THRESHOLD_LOW_WATERMARK", ScalarType.createVarchar(256))
.column("SPILL_THRESHOLD_HIGH_WATERMARK", ScalarType.createVarchar(256))
+ .column("TAG", ScalarType.createVarchar(256))
.build()))
.put("processlist", new SchemaTable(SystemIdGenerator.getNextId(), "processlist", TableType.SCHEMA,
builder().column("ID", ScalarType.createType(PrimitiveType.LARGEINT))
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
index 35f5b14efc5..52655fa0649 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
@@ -24,6 +24,7 @@ import org.apache.doris.common.DdlException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.FeMetaVersion;
import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.util.QueryableReentrantReadWriteLock;
@@ -647,11 +648,6 @@ public abstract class Table extends MetaObject implements Writable, TableIf {
return true;
}
- @Override
- public Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- return Collections.emptyMap();
- }
-
@Override
public List<Long> getChunkSizes() {
throw new NotImplementedException("getChunkSized not implemented");
@@ -661,4 +657,9 @@ public abstract class Table extends MetaObject implements Writable, TableIf {
public long fetchRowCount() {
return 0;
}
+
+ @Override
+ public List<Pair<String, String>> getColumnIndexPairs(Set<String> columns) {
+ return Lists.newArrayList();
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index 484dd3bb6eb..f7c8b4b8325 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -25,6 +25,7 @@ import org.apache.doris.catalog.constraint.UniqueConstraint;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.Pair;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.persist.AlterConstraintLog;
import org.apache.doris.statistics.AnalysisInfo;
@@ -184,7 +185,11 @@ public interface TableIf {
boolean needReAnalyzeTable(TableStatsMeta tblStats);
- Map<String, Set<String>> findReAnalyzeNeededPartitions();
+ /**
+ * @param columns Set of column names.
+ * @return List of pairs. Each pair is <IndexName, ColumnName>. For external table, index name is table name.
+ */
+ List<Pair<String, String>> getColumnIndexPairs(Set<String> columns);
// Get all the chunk sizes of this table. Now, only HMS external table implemented this interface.
// For HMS external table, the return result is a list of all the files' size.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudClusterChecker.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudClusterChecker.java
index cc4a3c09d9c..801f3166861 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudClusterChecker.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudClusterChecker.java
@@ -94,19 +94,13 @@ public class CloudClusterChecker extends MasterDaemon {
LOG.debug("begin to add clusterId: {}", addId);
}
// Attach tag to BEs
- Map<String, String> newTagMap = Tag.DEFAULT_BACKEND_TAG.toMap();
String clusterName = remoteClusterIdToPB.get(addId).getClusterName();
String clusterId = remoteClusterIdToPB.get(addId).getClusterId();
String publicEndpoint = remoteClusterIdToPB.get(addId).getPublicEndpoint();
String privateEndpoint = remoteClusterIdToPB.get(addId).getPrivateEndpoint();
- newTagMap.put(Tag.CLOUD_CLUSTER_NAME, clusterName);
- newTagMap.put(Tag.CLOUD_CLUSTER_ID, clusterId);
- newTagMap.put(Tag.CLOUD_CLUSTER_PUBLIC_ENDPOINT, publicEndpoint);
- newTagMap.put(Tag.CLOUD_CLUSTER_PRIVATE_ENDPOINT, privateEndpoint);
// For old versions that do no have status field set
ClusterStatus clusterStatus = remoteClusterIdToPB.get(addId).hasClusterStatus()
? remoteClusterIdToPB.get(addId).getClusterStatus() : ClusterStatus.NORMAL;
- newTagMap.put(Tag.CLOUD_CLUSTER_STATUS, String.valueOf(clusterStatus));
MetricRepo.registerCloudMetrics(clusterId, clusterName);
//toAdd.forEach(i -> i.setTagMap(newTagMap));
List<Backend> toAdd = new ArrayList<>();
@@ -117,6 +111,12 @@ public class CloudClusterChecker extends MasterDaemon {
continue;
}
Backend b = new Backend(Env.getCurrentEnv().getNextId(), addr, node.getHeartbeatPort());
+ Map<String, String> newTagMap = Tag.DEFAULT_BACKEND_TAG.toMap();
+ newTagMap.put(Tag.CLOUD_CLUSTER_STATUS, String.valueOf(clusterStatus));
+ newTagMap.put(Tag.CLOUD_CLUSTER_NAME, clusterName);
+ newTagMap.put(Tag.CLOUD_CLUSTER_ID, clusterId);
+ newTagMap.put(Tag.CLOUD_CLUSTER_PUBLIC_ENDPOINT, publicEndpoint);
+ newTagMap.put(Tag.CLOUD_CLUSTER_PRIVATE_ENDPOINT, privateEndpoint);
newTagMap.put(Tag.CLOUD_UNIQUE_ID, node.getCloudUniqueId());
b.setTagMap(newTagMap);
toAdd.add(b);
@@ -250,13 +250,6 @@ public class CloudClusterChecker extends MasterDaemon {
updateStatus(currentBes, expectedBes);
- // Attach tag to BEs
- Map<String, String> newTagMap = Tag.DEFAULT_BACKEND_TAG.toMap();
- newTagMap.put(Tag.CLOUD_CLUSTER_NAME, remoteClusterIdToPB.get(cid).getClusterName());
- newTagMap.put(Tag.CLOUD_CLUSTER_ID, remoteClusterIdToPB.get(cid).getClusterId());
- newTagMap.put(Tag.CLOUD_CLUSTER_PUBLIC_ENDPOINT, remoteClusterIdToPB.get(cid).getPublicEndpoint());
- newTagMap.put(Tag.CLOUD_CLUSTER_PRIVATE_ENDPOINT, remoteClusterIdToPB.get(cid).getPrivateEndpoint());
-
diffNodes(toAdd, toDel, () -> {
Map<String, Backend> currentMap = new HashMap<>();
for (Backend be : currentBes) {
@@ -280,6 +273,14 @@ public class CloudClusterChecker extends MasterDaemon {
if (node.hasIsSmoothUpgrade()) {
b.setSmoothUpgradeDst(node.getIsSmoothUpgrade());
}
+
+ // Attach tag to BEs
+ Map<String, String> newTagMap = Tag.DEFAULT_BACKEND_TAG.toMap();
+ newTagMap.put(Tag.CLOUD_CLUSTER_NAME, remoteClusterIdToPB.get(cid).getClusterName());
+ newTagMap.put(Tag.CLOUD_CLUSTER_ID, remoteClusterIdToPB.get(cid).getClusterId());
+ newTagMap.put(Tag.CLOUD_CLUSTER_PUBLIC_ENDPOINT, remoteClusterIdToPB.get(cid).getPublicEndpoint());
+ newTagMap.put(Tag.CLOUD_CLUSTER_PRIVATE_ENDPOINT,
+ remoteClusterIdToPB.get(cid).getPrivateEndpoint());
newTagMap.put(Tag.CLOUD_UNIQUE_ID, node.getCloudUniqueId());
b.setTagMap(newTagMap);
nodeMap.put(endpoint, b);
@@ -350,8 +351,8 @@ public class CloudClusterChecker extends MasterDaemon {
}
private void getCloudObserverFes() {
- Cloud.GetClusterResponse response = CloudSystemInfoService
- .getCloudCluster(Config.cloud_sql_server_cluster_name, Config.cloud_sql_server_cluster_id, "");
+ Cloud.GetClusterResponse response = cloudSystemInfoService.getCloudCluster(
+ Config.cloud_sql_server_cluster_name, Config.cloud_sql_server_cluster_id, "");
if (!response.hasStatus() || !response.getStatus().hasCode()
|| response.getStatus().getCode() != Cloud.MetaServiceCode.OK) {
LOG.warn("failed to get cloud cluster due to incomplete response, "
@@ -416,7 +417,7 @@ public class CloudClusterChecker extends MasterDaemon {
return;
}
try {
- CloudSystemInfoService.updateFrontends(toAdd, toDel);
+ cloudSystemInfoService.updateFrontends(toAdd, toDel);
} catch (DdlException e) {
LOG.warn("update cloud frontends exception e: {}, msg: {}", e, e.getMessage());
}
@@ -426,7 +427,7 @@ public class CloudClusterChecker extends MasterDaemon {
Map<String, List<Backend>> clusterIdToBackend = cloudSystemInfoService.getCloudClusterIdToBackend();
//rpc to ms, to get mysql user can use cluster_id
// NOTE: rpc args all empty, use cluster_unique_id to get a instance's all cluster info.
- Cloud.GetClusterResponse response = CloudSystemInfoService.getCloudCluster("", "", "");
+ Cloud.GetClusterResponse response = cloudSystemInfoService.getCloudCluster("", "", "");
if (!response.hasStatus() || !response.getStatus().hasCode()
|| (response.getStatus().getCode() != Cloud.MetaServiceCode.OK
&& response.getStatus().getCode() != MetaServiceCode.CLUSTER_NOT_FOUND)) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java
index 613fef3be68..7c37f1dbcff 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnv.java
@@ -83,8 +83,8 @@ public class CloudEnv extends Env {
private Cloud.NodeInfoPB getLocalTypeFromMetaService() {
// get helperNodes from ms
- Cloud.GetClusterResponse response = CloudSystemInfoService.getCloudCluster(
- Config.cloud_sql_server_cluster_name, Config.cloud_sql_server_cluster_id, "");
+ Cloud.GetClusterResponse response = ((CloudSystemInfoService) Env.getCurrentSystemInfo())
+ .getCloudCluster(Config.cloud_sql_server_cluster_name, Config.cloud_sql_server_cluster_id, "");
if (!response.hasStatus() || !response.getStatus().hasCode()
|| response.getStatus().getCode() != Cloud.MetaServiceCode.OK) {
LOG.warn("failed to get cloud cluster due to incomplete response, "
@@ -392,7 +392,7 @@ public class CloudEnv extends Env {
public void changeCloudCluster(String clusterName, ConnectContext ctx) throws DdlException {
checkCloudClusterPriv(clusterName);
- CloudSystemInfoService.waitForAutoStart(clusterName);
+ ((CloudSystemInfoService) Env.getCurrentSystemInfo()).waitForAutoStart(clusterName);
try {
((CloudSystemInfoService) Env.getCurrentSystemInfo()).addCloudCluster(clusterName, "");
} catch (UserException e) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
index 01603823752..60e567ce9fe 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudPartition.java
@@ -22,8 +22,7 @@ import org.apache.doris.catalog.MaterializedIndex;
import org.apache.doris.catalog.Partition;
import org.apache.doris.cloud.proto.Cloud;
import org.apache.doris.cloud.proto.Cloud.MetaServiceCode;
-import org.apache.doris.cloud.rpc.MetaServiceProxy;
-import org.apache.doris.common.Config;
+import org.apache.doris.cloud.qe.SnapshotProxy;
import org.apache.doris.common.profile.SummaryProfile;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.StmtExecutor;
@@ -38,10 +37,6 @@ import java.io.DataOutput;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
-import java.util.concurrent.ExecutionException;
-import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
import java.util.concurrent.locks.ReentrantLock;
import java.util.stream.Collectors;
@@ -328,7 +323,7 @@ public class CloudPartition extends Partition {
throws RpcException {
long startAt = System.nanoTime();
try {
- return getVersionFromMetaInner(req);
+ return SnapshotProxy.getVisibleVersion(req);
} finally {
SummaryProfile profile = getSummaryProfile();
if (profile != null) {
@@ -337,55 +332,6 @@ public class CloudPartition extends Partition {
}
}
- private static Cloud.GetVersionResponse getVersionFromMetaInner(Cloud.GetVersionRequest req)
- throws RpcException {
- for (int retryTime = 0; retryTime < Config.cloud_meta_service_rpc_failed_retry_times; retryTime++) {
- try {
- long deadline = System.currentTimeMillis() + Config.default_get_version_from_ms_timeout_second * 1000L;
- Future<Cloud.GetVersionResponse> future =
- MetaServiceProxy.getInstance().getVisibleVersionAsync(req);
-
- Cloud.GetVersionResponse resp = null;
- while (resp == null) {
- try {
- resp = future.get(Math.max(0, deadline - System.currentTimeMillis()), TimeUnit.MILLISECONDS);
- } catch (InterruptedException e) {
- LOG.warn("get version from meta service: future get interrupted exception");
- }
- }
-
- if (resp.hasStatus() && (resp.getStatus().getCode() == MetaServiceCode.OK
- || resp.getStatus().getCode() == MetaServiceCode.VERSION_NOT_FOUND)) {
- if (LOG.isDebugEnabled()) {
- LOG.debug("get version from meta service, code: {}", resp.getStatus().getCode());
- }
- return resp;
- }
-
- LOG.warn("get version from meta service failed, status: {}, retry time: {}",
- resp.getStatus(), retryTime);
- } catch (RpcException | ExecutionException | TimeoutException | RuntimeException e) {
- LOG.warn("get version from meta service failed, retry times: {} exception: ", retryTime, e);
- }
-
- // sleep random millis [20, 200] ms, retry rpc failed
- int randomMillis = 20 + (int) (Math.random() * (200 - 20));
- if (retryTime > Config.cloud_meta_service_rpc_failed_retry_times / 2) {
- // sleep random millis [500, 1000] ms, retry rpc failed
- randomMillis = 500 + (int) (Math.random() * (1000 - 500));
- }
- try {
- Thread.sleep(randomMillis);
- } catch (InterruptedException e) {
- LOG.warn("get version from meta service: sleep get interrupted exception");
- }
- }
-
- LOG.warn("get version from meta service failed after retry {} times",
- Config.cloud_meta_service_rpc_failed_retry_times);
- throw new RpcException("get version from meta service", "failed after retry n times");
- }
-
private static boolean isEmptyPartitionPruneDisabled() {
ConnectContext ctx = ConnectContext.get();
if (ctx != null && ctx.getSessionVariable().getDisableEmptyPartitionPrune()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
index a51e332a784..aebc66128ed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudReplica.java
@@ -163,7 +163,7 @@ public class CloudReplica extends Replica {
// if cluster is SUSPENDED, wait
try {
- CloudSystemInfoService.waitForAutoStart(cluster);
+ ((CloudSystemInfoService) Env.getCurrentSystemInfo()).waitForAutoStart(cluster);
} catch (DdlException e) {
// this function cant throw exception. so just log it
LOG.warn("cant resume cluster {}", cluster);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java
index f48c38de698..b473f80e126 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/datasource/CloudInternalCatalog.java
@@ -361,12 +361,13 @@ public class CloudInternalCatalog extends InternalCatalog {
}
@Override
- protected void afterCreatePartitions(long tableId, List<Long> partitionIds, List<Long> indexIds)
+ protected void afterCreatePartitions(long dbId, long tableId, List<Long> partitionIds, List<Long> indexIds,
+ boolean isCreateTable)
throws DdlException {
if (partitionIds == null) {
- commitMaterializedIndex(tableId, indexIds);
+ commitMaterializedIndex(dbId, tableId, indexIds, isCreateTable);
} else {
- commitPartition(tableId, partitionIds, indexIds);
+ commitPartition(dbId, tableId, partitionIds, indexIds);
}
}
@@ -406,11 +407,13 @@ public class CloudInternalCatalog extends InternalCatalog {
}
}
- private void commitPartition(long tableId, List<Long> partitionIds, List<Long> indexIds) throws DdlException {
+ private void commitPartition(long dbId, long tableId, List<Long> partitionIds, List<Long> indexIds)
+ throws DdlException {
Cloud.PartitionRequest.Builder partitionRequestBuilder = Cloud.PartitionRequest.newBuilder();
partitionRequestBuilder.setCloudUniqueId(Config.cloud_unique_id);
partitionRequestBuilder.addAllPartitionIds(partitionIds);
partitionRequestBuilder.addAllIndexIds(indexIds);
+ partitionRequestBuilder.setDbId(dbId);
partitionRequestBuilder.setTableId(tableId);
final Cloud.PartitionRequest partitionRequest = partitionRequestBuilder.build();
@@ -469,11 +472,14 @@ public class CloudInternalCatalog extends InternalCatalog {
}
}
- public void commitMaterializedIndex(Long tableId, List<Long> indexIds) throws DdlException {
+ public void commitMaterializedIndex(long dbId, long tableId, List<Long> indexIds, boolean isCreateTable)
+ throws DdlException {
Cloud.IndexRequest.Builder indexRequestBuilder = Cloud.IndexRequest.newBuilder();
indexRequestBuilder.setCloudUniqueId(Config.cloud_unique_id);
indexRequestBuilder.addAllIndexIds(indexIds);
+ indexRequestBuilder.setDbId(dbId);
indexRequestBuilder.setTableId(tableId);
+ indexRequestBuilder.setIsNewTable(isCreateTable);
final Cloud.IndexRequest indexRequest = indexRequestBuilder.build();
Cloud.IndexResponse response = null;
@@ -562,7 +568,7 @@ public class CloudInternalCatalog extends InternalCatalog {
if (indexs.isEmpty()) {
break;
}
- dropMaterializedIndex(olapTable.getId(), indexs);
+ dropMaterializedIndex(olapTable.getId(), indexs, true);
} catch (Exception e) {
LOG.warn("failed to drop index {} of table {}, try cnt {}, execption {}",
indexs, olapTable.getId(), tryCnt, e);
@@ -657,7 +663,7 @@ public class CloudInternalCatalog extends InternalCatalog {
}
}
- public void dropMaterializedIndex(Long tableId, List<Long> indexIds) throws DdlException {
+ public void dropMaterializedIndex(long tableId, List<Long> indexIds, boolean dropTable) throws DdlException {
Cloud.IndexRequest.Builder indexRequestBuilder = Cloud.IndexRequest.newBuilder();
indexRequestBuilder.setCloudUniqueId(Config.cloud_unique_id);
indexRequestBuilder.addAllIndexIds(indexIds);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/load/CloudLoadManager.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/load/CloudLoadManager.java
index 1d0bfc23f6c..5caa2108c59 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/load/CloudLoadManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/load/CloudLoadManager.java
@@ -19,6 +19,7 @@ package org.apache.doris.cloud.load;
import org.apache.doris.analysis.InsertStmt;
import org.apache.doris.analysis.LoadStmt;
+import org.apache.doris.catalog.Env;
import org.apache.doris.cloud.system.CloudSystemInfoService;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.UserException;
@@ -33,14 +34,14 @@ public class CloudLoadManager extends LoadManager {
@Override
public long createLoadJobFromStmt(LoadStmt stmt) throws DdlException, UserException {
- CloudSystemInfoService.waitForAutoStartCurrentCluster();
+ ((CloudSystemInfoService) Env.getCurrentSystemInfo()).waitForAutoStartCurrentCluster();
return super.createLoadJobFromStmt(stmt);
}
@Override
public long createLoadJobFromStmt(InsertStmt stmt) throws DdlException {
- CloudSystemInfoService.waitForAutoStartCurrentCluster();
+ ((CloudSystemInfoService) Env.getCurrentSystemInfo()).waitForAutoStartCurrentCluster();
return super.createLoadJobFromStmt(stmt);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/qe/SnapshotProxy.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/qe/SnapshotProxy.java
new file mode 100644
index 00000000000..1f4b20ab1dd
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/qe/SnapshotProxy.java
@@ -0,0 +1,92 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.qe;
+
+import org.apache.doris.cloud.proto.Cloud;
+import org.apache.doris.cloud.rpc.MetaServiceProxy;
+import org.apache.doris.common.Config;
+import org.apache.doris.rpc.RpcException;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+public class SnapshotProxy {
+ private static final Logger LOG = LogManager.getLogger(SnapshotProxy.class);
+
+ public static Cloud.GetVersionResponse getVisibleVersion(Cloud.GetVersionRequest request) throws RpcException {
+ int tryTimes = 0;
+ while (tryTimes++ < Config.meta_service_rpc_retry_times) {
+ Cloud.GetVersionResponse resp = getVisibleVersionInternal(request,
+ Config.default_get_version_from_ms_timeout_second * 1000);
+ if (resp.hasStatus() && (resp.getStatus().getCode() == Cloud.MetaServiceCode.OK
+ || resp.getStatus().getCode() == Cloud.MetaServiceCode.VERSION_NOT_FOUND)) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("get version from meta service, code: {}", resp.getStatus().getCode());
+ }
+ return resp;
+ }
+
+ LOG.warn("get version from meta service failed, status: {}, retry time: {}",
+ resp.getStatus(), tryTimes);
+
+ // sleep random millis, retry rpc failed
+ if (tryTimes > Config.meta_service_rpc_retry_times / 2) {
+ sleepSeveralMs(500, 1000);
+ } else {
+ sleepSeveralMs(20, 200);
+ }
+ }
+
+ LOG.warn("get version from meta service failed after retry {} times", tryTimes);
+ throw new RpcException("get version from meta service", "failed after retry n times");
+ }
+
+ public static Cloud.GetVersionResponse getVisibleVersionInternal(Cloud.GetVersionRequest request, int timeoutMs) {
+ long deadline = System.currentTimeMillis() + timeoutMs;
+ Cloud.GetVersionResponse resp = null;
+ try {
+ Future<Cloud.GetVersionResponse> future =
+ MetaServiceProxy.getInstance().getVisibleVersionAsync(request);
+
+ while (resp == null) {
+ try {
+ resp = future.get(Math.max(0, deadline - System.currentTimeMillis()), TimeUnit.MILLISECONDS);
+ } catch (InterruptedException e) {
+ LOG.warn("get version from meta service: future get interrupted exception");
+ }
+ }
+ } catch (RpcException | ExecutionException | TimeoutException | RuntimeException e) {
+ LOG.warn("get version from meta service failed, exception: ", e);
+ }
+ return resp;
+ }
+
+ private static void sleepSeveralMs(int lowerMs, int upperMs) {
+ // sleep random millis [lowerMs, upperMs] ms
+ try {
+ Thread.sleep(lowerMs + (int) (Math.random() * (upperMs - lowerMs)));
+ } catch (InterruptedException e) {
+ LOG.warn("get snapshot from meta service: sleep get interrupted exception");
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java b/fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java
index 7171fbafd16..59ba7340dfc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java
@@ -89,7 +89,7 @@ public class CloudSystemInfoService extends SystemInfoService {
* @param clusterId cluster id
* @return
*/
- public static Cloud.GetClusterResponse getCloudCluster(String clusterName, String clusterId, String userName) {
+ public Cloud.GetClusterResponse getCloudCluster(String clusterName, String clusterId, String userName) {
Cloud.GetClusterRequest.Builder builder = Cloud.GetClusterRequest.newBuilder();
builder.setCloudUniqueId(Config.cloud_unique_id)
.setClusterName(clusterName).setClusterId(clusterId).setMysqlUserName(userName);
@@ -261,8 +261,8 @@ public class CloudSystemInfoService extends SystemInfoService {
}
- public static synchronized void updateFrontends(List<Frontend> toAdd,
- List<Frontend> toDel) throws DdlException {
+ public synchronized void updateFrontends(List<Frontend> toAdd, List<Frontend> toDel)
+ throws DdlException {
if (LOG.isDebugEnabled()) {
LOG.debug("updateCloudFrontends toAdd={} toDel={}", toAdd, toDel);
}
@@ -570,7 +570,7 @@ public class CloudSystemInfoService extends SystemInfoService {
this.instanceStatus = instanceStatus;
}
- public static void waitForAutoStartCurrentCluster() throws DdlException {
+ public void waitForAutoStartCurrentCluster() throws DdlException {
ConnectContext context = ConnectContext.get();
if (context != null) {
String cloudCluster = context.getCloudCluster();
@@ -580,7 +580,7 @@ public class CloudSystemInfoService extends SystemInfoService {
}
}
- public static String getClusterNameAutoStart(final String clusterName) {
+ public String getClusterNameAutoStart(final String clusterName) {
if (!Strings.isNullOrEmpty(clusterName)) {
return clusterName;
}
@@ -607,7 +607,7 @@ public class CloudSystemInfoService extends SystemInfoService {
return cloudClusterTypeAndName.clusterName;
}
- public static void waitForAutoStart(String clusterName) throws DdlException {
+ public void waitForAutoStart(String clusterName) throws DdlException {
if (Config.isNotCloudMode()) {
return;
}
@@ -616,7 +616,7 @@ public class CloudSystemInfoService extends SystemInfoService {
LOG.warn("auto start in cloud mode, but clusterName empty {}", clusterName);
return;
}
- String clusterStatus = ((CloudSystemInfoService) Env.getCurrentSystemInfo()).getCloudStatusByName(clusterName);
+ String clusterStatus = getCloudStatusByName(clusterName);
if (Strings.isNullOrEmpty(clusterStatus)) {
// for cluster rename or cluster dropped
LOG.warn("cant find clusterStatus in fe, clusterName {}", clusterName);
@@ -631,8 +631,7 @@ public class CloudSystemInfoService extends SystemInfoService {
builder.setOp(Cloud.AlterClusterRequest.Operation.SET_CLUSTER_STATUS);
Cloud.ClusterPB.Builder clusterBuilder = Cloud.ClusterPB.newBuilder();
- clusterBuilder.setClusterId(((CloudSystemInfoService)
- Env.getCurrentSystemInfo()).getCloudClusterIdByName(clusterName));
+ clusterBuilder.setClusterId(getCloudClusterIdByName(clusterName));
clusterBuilder.setClusterStatus(Cloud.ClusterStatus.TO_RESUME);
builder.setCluster(clusterBuilder);
@@ -671,7 +670,7 @@ public class CloudSystemInfoService extends SystemInfoService {
} catch (InterruptedException e) {
LOG.info("change cluster sleep wait InterruptedException: ", e);
}
- clusterStatus = ((CloudSystemInfoService) Env.getCurrentSystemInfo()).getCloudStatusByName(clusterName);
+ clusterStatus = getCloudStatusByName(clusterName);
}
if (retryTime >= retryTimes) {
// auto start timeout
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
index d4615dab820..7e9367d0abd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java
@@ -74,6 +74,8 @@ public class SummaryProfile {
public static final String GET_PARTITION_VERSION_TIME = "Get Partition Version Time";
public static final String GET_PARTITION_VERSION_COUNT = "Get Partition Version Count";
public static final String GET_PARTITION_VERSION_BY_HAS_DATA_COUNT = "Get Partition Version Count (hasData)";
+ public static final String GET_TABLE_VERSION_TIME = "Get Table Version Time";
+ public static final String GET_TABLE_VERSION_COUNT = "Get Table Version Count";
public static final String PARSE_SQL_TIME = "Parse SQL Time";
public static final String NEREIDS_ANALYSIS_TIME = "Nereids Analysis Time";
@@ -93,7 +95,8 @@ public class SummaryProfile {
PLAN_TIME, JOIN_REORDER_TIME, CREATE_SINGLE_NODE_TIME, QUERY_DISTRIBUTED_TIME,
INIT_SCAN_NODE_TIME, FINALIZE_SCAN_NODE_TIME, GET_SPLITS_TIME, GET_PARTITIONS_TIME,
GET_PARTITION_FILES_TIME, CREATE_SCAN_RANGE_TIME, GET_PARTITION_VERSION_TIME,
- GET_PARTITION_VERSION_BY_HAS_DATA_COUNT, GET_PARTITION_VERSION_COUNT, SCHEDULE_TIME, FETCH_RESULT_TIME,
+ GET_PARTITION_VERSION_BY_HAS_DATA_COUNT, GET_PARTITION_VERSION_COUNT, GET_TABLE_VERSION_TIME,
+ GET_TABLE_VERSION_COUNT, SCHEDULE_TIME, FETCH_RESULT_TIME,
WRITE_RESULT_TIME, WAIT_FETCH_RESULT_TIME, DORIS_VERSION, IS_NEREIDS, IS_PIPELINE,
IS_CACHED, TOTAL_INSTANCES_NUM, INSTANCES_NUM_PER_BE, PARALLEL_FRAGMENT_EXEC_INSTANCE, TRACE_ID);
@@ -115,6 +118,8 @@ public class SummaryProfile {
builder.put(GET_PARTITION_VERSION_TIME, 1);
builder.put(GET_PARTITION_VERSION_COUNT, 1);
builder.put(GET_PARTITION_VERSION_BY_HAS_DATA_COUNT, 1);
+ builder.put(GET_TABLE_VERSION_TIME, 1);
+ builder.put(GET_TABLE_VERSION_COUNT, 1);
EXECUTION_SUMMARY_KEYS_IDENTATION = builder.build();
}
@@ -158,6 +163,8 @@ public class SummaryProfile {
private long getPartitionVersionTime = 0;
private long getPartitionVersionCount = 0;
private long getPartitionVersionByHasDataCount = 0;
+ private long getTableVersionTime = 0;
+ private long getTableVersionCount = 0;
public SummaryProfile(RuntimeProfile rootProfile) {
summaryProfile = new RuntimeProfile(SUMMARY_PROFILE_NAME);
@@ -235,6 +242,8 @@ public class SummaryProfile {
executionSummaryProfile.addInfoString(GET_PARTITION_VERSION_COUNT, getPrettyGetPartitionVersionCount());
executionSummaryProfile.addInfoString(GET_PARTITION_VERSION_BY_HAS_DATA_COUNT,
getPrettyGetPartitionVersionByHasDataCount());
+ executionSummaryProfile.addInfoString(GET_TABLE_VERSION_TIME, getPrettyGetTableVersionTime());
+ executionSummaryProfile.addInfoString(GET_TABLE_VERSION_COUNT, getPrettyGetTableVersionCount());
}
}
@@ -347,6 +356,11 @@ public class SummaryProfile {
this.getPartitionVersionCount += 1;
}
+ public void addGetTableVersionTime(long ns) {
+ this.getTableVersionTime += ns;
+ this.getTableVersionCount += 1;
+ }
+
public void incGetPartitionVersionByHasDataCount() {
this.getPartitionVersionByHasDataCount += 1;
}
@@ -593,4 +607,15 @@ public class SummaryProfile {
private String getPrettyGetPartitionVersionCount() {
return RuntimeProfile.printCounter(getPartitionVersionCount, TUnit.UNIT);
}
+
+ private String getPrettyGetTableVersionTime() {
+ if (getTableVersionTime == 0) {
+ return "N/A";
+ }
+ return RuntimeProfile.printCounter(getTableVersionTime, TUnit.TIME_NS);
+ }
+
+ private String getPrettyGetTableVersionCount() {
+ return RuntimeProfile.printCounter(getTableVersionCount, TUnit.UNIT);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java
index 7f82d0d3876..82390b91656 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java
@@ -24,6 +24,7 @@ import org.apache.doris.catalog.TableAttributes;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.constraint.Constraint;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.util.Util;
@@ -36,7 +37,7 @@ import org.apache.doris.statistics.TableStatsMeta;
import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.thrift.TTableDescriptor;
-import com.google.common.collect.Sets;
+import com.google.common.collect.Lists;
import com.google.gson.annotations.SerializedName;
import lombok.Getter;
import org.apache.commons.lang3.NotImplementedException;
@@ -46,7 +47,6 @@ import org.apache.logging.log4j.Logger;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -322,11 +322,12 @@ public class ExternalTable implements TableIf, Writable, GsonPostProcessable {
if (tblStats == null) {
return true;
}
- if (!tblStats.analyzeColumns().containsAll(getBaseSchema()
+ if (!tblStats.analyzeColumns().containsAll(getColumnIndexPairs(
+ getBaseSchema()
.stream()
.filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
.map(Column::getName)
- .collect(Collectors.toSet()))) {
+ .collect(Collectors.toSet())))) {
return true;
}
return System.currentTimeMillis()
@@ -334,12 +335,17 @@ public class ExternalTable implements TableIf, Writable, GsonPostProcessable {
}
@Override
- public Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- HashSet<String> partitions = Sets.newHashSet();
- // TODO: Find a way to collect external table partitions that need to be analyzed.
- partitions.add("Dummy Partition");
- return getBaseSchema().stream().filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
- .collect(Collectors.toMap(Column::getName, k -> partitions));
+ public List<Pair<String, String>> getColumnIndexPairs(Set<String> columns) {
+ List<Pair<String, String>> ret = Lists.newArrayList();
+ for (String column : columns) {
+ Column col = getColumn(column);
+ if (col == null || StatisticsUtil.isUnsupportedType(col.getType())) {
+ continue;
+ }
+ // External table put table name as index name.
+ ret.add(Pair.of(String.valueOf(name), column));
+ }
+ return ret;
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index 91c919af3f8..31c882a05f0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -1612,7 +1612,7 @@ public class InternalCatalog implements CatalogIf<Database> {
singlePartitionDesc.getTabletType(),
storagePolicy, idGeneratorBuffer,
binlogConfig, dataProperty.isStorageMediumSpecified(), null);
- afterCreatePartitions(olapTable.getId(), partitionIds, indexIds);
+ afterCreatePartitions(db.getId(), olapTable.getId(), partitionIds, indexIds, false);
// TODO cluster key ids
// check again
@@ -2033,7 +2033,8 @@ public class InternalCatalog implements CatalogIf<Database> {
throws DdlException {
}
- protected void afterCreatePartitions(long tableId, List<Long> partitionIds, List<Long> indexIds)
+ protected void afterCreatePartitions(long dbId, long tableId, List<Long> partitionIds, List<Long> indexIds,
+ boolean isCreateTable)
throws DdlException {
}
@@ -2669,7 +2670,8 @@ public class InternalCatalog implements CatalogIf<Database> {
binlogConfigForTask,
partitionInfo.getDataProperty(partitionId).isStorageMediumSpecified(),
keysDesc.getClusterKeysColumnIds());
- afterCreatePartitions(olapTable.getId(), null, olapTable.getIndexIdList());
+ afterCreatePartitions(db.getId(), olapTable.getId(), null,
+ olapTable.getIndexIdList(), true);
olapTable.addPartition(partition);
} else if (partitionInfo.getType() == PartitionType.RANGE
|| partitionInfo.getType() == PartitionType.LIST) {
@@ -2767,7 +2769,8 @@ public class InternalCatalog implements CatalogIf<Database> {
olapTable.getPartitionInfo().getDataProperty(partition.getId())
.setStoragePolicy(partionStoragePolicy);
}
- afterCreatePartitions(olapTable.getId(), null, olapTable.getIndexIdList());
+ afterCreatePartitions(db.getId(), olapTable.getId(), null,
+ olapTable.getIndexIdList(), true);
} else {
throw new DdlException("Unsupported partition method: " + partitionInfo.getType().name());
}
@@ -3227,7 +3230,7 @@ public class InternalCatalog implements CatalogIf<Database> {
newPartitions.add(newPartition);
}
- afterCreatePartitions(copiedTbl.getId(), newPartitionIds, indexIds);
+ afterCreatePartitions(db.getId(), copiedTbl.getId(), newPartitionIds, indexIds, true);
} catch (DdlException e) {
// create partition failed, remove all newly created tablets
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java
index 2ccf1e03e28..221aed4cd8c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergHMSExternalCatalog.java
@@ -17,16 +17,10 @@
package org.apache.doris.datasource.iceberg;
-import org.apache.doris.common.Config;
-import org.apache.doris.common.security.authentication.AuthenticationConfig;
-import org.apache.doris.common.security.authentication.HadoopUGI;
import org.apache.doris.datasource.CatalogProperty;
-import org.apache.doris.datasource.hive.HMSCachedClient;
-import org.apache.doris.datasource.hive.HiveMetadataOps;
import org.apache.doris.datasource.property.PropertyConverter;
import org.apache.doris.datasource.property.constants.HMSProperties;
-import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.hive.HiveCatalog;
@@ -51,20 +45,8 @@ public class IcebergHMSExternalCatalog extends IcebergExternalCatalog {
Map<String, String> catalogProperties = new HashMap<>();
String metastoreUris = catalogProperty.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
catalogProperties.put(CatalogProperties.URI, metastoreUris);
- HiveConf hiveConf = new HiveConf();
- for (Map.Entry<String, String> kv : catalogProperty.getHadoopProperties().entrySet()) {
- hiveConf.set(kv.getKey(), kv.getValue());
- }
- hiveConf.set(HiveConf.ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT.name(),
- String.valueOf(Config.hive_metastore_client_timeout_second));
- HadoopUGI.tryKrbLogin(this.getName(), AuthenticationConfig.getKerberosConfig(hiveConf,
- AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL,
- AuthenticationConfig.HADOOP_KERBEROS_KEYTAB));
- initS3Param(hiveConf);
- HMSCachedClient cachedClient = HiveMetadataOps.createCachedClient(hiveConf, 1, null);
- String location = cachedClient.getCatalogLocation("hive");
- catalogProperties.put(CatalogProperties.WAREHOUSE_LOCATION, location);
hiveCatalog.initialize(icebergCatalogType, catalogProperties);
catalog = hiveCatalog;
}
}
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
index 24457abc748..b385fb838b3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
@@ -39,9 +39,9 @@ import com.aliyun.datalake.metastore.common.DataLakeConfig;
import com.amazonaws.glue.catalog.util.AWSGlueConfig;
import com.google.common.base.Strings;
import com.google.common.collect.Maps;
+import org.apache.hadoop.fs.CosFileSystem;
+import org.apache.hadoop.fs.CosNConfigKeys;
import org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem;
-import org.apache.hadoop.fs.cosn.CosNConfigKeys;
-import org.apache.hadoop.fs.cosn.CosNFileSystem;
import org.apache.hadoop.fs.obs.OBSConstants;
import org.apache.hadoop.fs.obs.OBSFileSystem;
import org.apache.hadoop.fs.s3a.Constants;
@@ -186,7 +186,7 @@ public class PropertyConverter {
} else if (fsScheme.equalsIgnoreCase("oss")) {
return AliyunOSSFileSystem.class.getName();
} else if (fsScheme.equalsIgnoreCase("cosn")) {
- return CosNFileSystem.class.getName();
+ return CosFileSystem.class.getName();
} else {
return S3AFileSystem.class.getName();
}
@@ -352,8 +352,8 @@ public class PropertyConverter {
cosProperties.put("fs.cosn.impl.disable.cache", "true");
cosProperties.put("fs.cosn.impl", getHadoopFSImplByScheme("cosn"));
if (credential.isWhole()) {
- cosProperties.put(CosNConfigKeys.COSN_SECRET_ID_KEY, credential.getAccessKey());
- cosProperties.put(CosNConfigKeys.COSN_SECRET_KEY_KEY, credential.getSecretKey());
+ cosProperties.put(CosNConfigKeys.COSN_USERINFO_SECRET_ID_KEY, credential.getAccessKey());
+ cosProperties.put(CosNConfigKeys.COSN_USERINFO_SECRET_KEY_KEY, credential.getSecretKey());
}
// session token is unsupported
for (Map.Entry<String, String> entry : props.entrySet()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java b/fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java
index 32d52d7e7af..e0c46dde920 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/insertoverwrite/InsertOverwriteManager.java
@@ -56,8 +56,10 @@ public class InsertOverwriteManager extends MasterDaemon implements Writable {
private Map<Long, List<Long>> taskGroups = Maps.newConcurrentMap();
// for one task group, there may be different requests about changing a partition to new.
// but we only change one time and save the relations in partitionPairs. they're protected by taskLocks
+ @SerializedName(value = "taskLocks")
private Map<Long, ReentrantLock> taskLocks = Maps.newConcurrentMap();
// <groupId, <oldPartId, newPartId>>
+ @SerializedName(value = "partitionPairs")
private Map<Long, Map<Long, Long>> partitionPairs = Maps.newConcurrentMap();
public InsertOverwriteManager() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
index e3333f32585..d23c7534f86 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java
@@ -95,9 +95,9 @@ public class MTMVPlanUtil {
TableCollectorContext collectorContext =
new TableCollector.TableCollectorContext(
com.google.common.collect.Sets
- .newHashSet(TableType.values()));
+ .newHashSet(TableType.values()), true);
plan.accept(TableCollector.INSTANCE, collectorContext);
- List<TableIf> collectedTables = collectorContext.getCollectedTables();
+ Set<TableIf> collectedTables = collectorContext.getCollectedTables();
return transferTableIfToInfo(collectedTables);
}
@@ -105,7 +105,7 @@ public class MTMVPlanUtil {
return Sets.newHashSet();
}
- private static Set<BaseTableInfo> transferTableIfToInfo(List<TableIf> tables) {
+ private static Set<BaseTableInfo> transferTableIfToInfo(Set<TableIf> tables) {
Set<BaseTableInfo> result = com.google.common.collect.Sets.newHashSet();
for (TableIf table : tables) {
result.add(new BaseTableInfo(table));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivEntry.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivEntry.java
index a41f3803088..65546e49ebc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivEntry.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivEntry.java
@@ -27,11 +27,8 @@ import java.io.DataInput;
import java.io.IOException;
public class ResourcePrivEntry extends PrivEntry {
- protected static final String ANY_RESOURCE = "*";
-
protected PatternMatcher resourcePattern;
protected String origResource;
- protected boolean isAnyResource;
protected ResourcePrivEntry() {
}
@@ -41,15 +38,12 @@ public class ResourcePrivEntry extends PrivEntry {
super(privSet);
this.resourcePattern = resourcePattern;
this.origResource = origResource;
- if (origResource.equals(ANY_RESOURCE)) {
- isAnyResource = true;
- }
}
public static ResourcePrivEntry create(String resourceName, PrivBitSet privs)
throws AnalysisException, PatternMatcherException {
PatternMatcher resourcePattern = PatternMatcher.createMysqlPattern(
- resourceName.equals(ANY_RESOURCE) ? "%" : resourceName,
+ resourceName,
CaseSensibility.RESOURCE.getCaseSensibility());
if (privs.containsNodePriv() || privs.containsDbTablePriv()) {
throw new AnalysisException("Resource privilege can not contains node or db table privileges: " + privs);
@@ -112,6 +106,5 @@ public class ResourcePrivEntry extends PrivEntry {
} catch (PatternMatcherException e) {
throw new IOException(e);
}
- isAnyResource = origResource.equals(ANY_RESOURCE);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivTable.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivTable.java
index 010d1026c95..93f8bb20eb9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/ResourcePrivTable.java
@@ -26,26 +26,15 @@ import org.apache.logging.log4j.Logger;
public class ResourcePrivTable extends PrivTable {
private static final Logger LOG = LogManager.getLogger(ResourcePrivTable.class);
- /*
- * Return first priv which match the user@host on resourceName The returned priv will be
- * saved in 'savedPrivs'.
- */
public void getPrivs(String resourceName, PrivBitSet savedPrivs) {
- ResourcePrivEntry matchedEntry = null;
+ // need check all entries, because may have 2 entries match resourceName,
+ // For example, if the resourceName is g1, there are two entry `%` and `g1` compound requirements
for (PrivEntry entry : entries) {
ResourcePrivEntry resourcePrivEntry = (ResourcePrivEntry) entry;
// check resource
- if (!resourcePrivEntry.getResourcePattern().match(resourceName)) {
- continue;
+ if (resourcePrivEntry.getResourcePattern().match(resourceName)) {
+ savedPrivs.or(resourcePrivEntry.getPrivSet());
}
-
- matchedEntry = resourcePrivEntry;
- break;
- }
- if (matchedEntry == null) {
- return;
}
-
- savedPrivs.or(matchedEntry.getPrivSet());
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
index b22849ea75e..e6090f87623 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/Role.java
@@ -471,8 +471,8 @@ public class Role implements Writable, GsonPostProcessable {
return true;
}
PrivBitSet savedPrivs = PrivBitSet.of();
- // Workload groups do not support global usage_priv, so only global admin_priv and usage_priv are checked.
- if (checkGlobalInternal(PrivPredicate.ADMIN, savedPrivs)
+ // usage priv not in global, but grant_priv may in global
+ if (checkGlobalInternal(wanted, savedPrivs)
|| checkWorkloadGroupInternal(workloadGroupName, wanted, savedPrivs)) {
return true;
}
@@ -573,22 +573,11 @@ public class Role implements Writable, GsonPostProcessable {
if (privs.isEmpty()) {
return;
}
- // grant privs to user
- switch (resourcePattern.getPrivLevel()) {
- case GLOBAL:
- grantGlobalPrivs(privs);
- break;
- case RESOURCE:
- if (resourcePattern.isClusterResource()) {
- grantCloudClusterPrivs(resourcePattern.getResourceName(), false, false, privs);
- } else {
- grantResourcePrivs(resourcePattern.getResourceName(), privs);
- }
- break;
- default:
- Preconditions.checkNotNull(null, resourcePattern.getPrivLevel());
+ if (resourcePattern.isClusterResource()) {
+ grantCloudClusterPrivs(resourcePattern.getResourceName(), false, false, privs);
+ } else {
+ grantResourcePrivs(resourcePattern.getResourceName(), privs);
}
-
}
private void grantPrivs(WorkloadGroupPattern workloadGroupPattern, PrivBitSet privs) throws DdlException {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivTable.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivTable.java
index 994118c7431..7b35b41f01f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/WorkloadGroupPrivTable.java
@@ -20,20 +20,13 @@ package org.apache.doris.mysql.privilege;
public class WorkloadGroupPrivTable extends PrivTable {
public void getPrivs(String workloadGroupName, PrivBitSet savedPrivs) {
- WorkloadGroupPrivEntry matchedEntry = null;
+ // need check all entries, because may have 2 entries match workloadGroupName,
+ // For example, if the workloadGroupName is g1, there are two entry `%` and `g1` compound requirements
for (PrivEntry entry : entries) {
WorkloadGroupPrivEntry workloadGroupPrivEntry = (WorkloadGroupPrivEntry) entry;
- if (!workloadGroupPrivEntry.getWorkloadGroupPattern().match(workloadGroupName)) {
- continue;
+ if (workloadGroupPrivEntry.getWorkloadGroupPattern().match(workloadGroupName)) {
+ savedPrivs.or(workloadGroupPrivEntry.getPrivSet());
}
-
- matchedEntry = workloadGroupPrivEntry;
- break;
- }
- if (matchedEntry == null) {
- return;
}
-
- savedPrivs.or(matchedEntry.getPrivSet());
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 2985664f8b8..a9ba3603f53 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -2625,7 +2625,7 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
String indexType = ctx.indexType != null ? ctx.indexType.getText().toUpperCase() : null;
String comment = ctx.comment != null ? ctx.comment.getText() : "";
// change BITMAP index to INVERTED index
- if (indexType.equalsIgnoreCase("BITMAP")) {
+ if ("BITMAP".equalsIgnoreCase(indexType)) {
indexType = "INVERTED";
}
return new IndexDefinition(indexName, indexCols, indexType, properties, comment);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
new file mode 100644
index 00000000000..ff362b7bc9e
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.processor.post;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.annotation.DependsRules;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalJoin;
+import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Prune column for Join-Cluster
+ */
+@DependsRules({
+ MergeProjectPostProcessor.class
+})
+public class ColumnPruningPostProcessor extends PlanPostProcessor {
+ @Override
+ public PhysicalProject visitPhysicalProject(PhysicalProject<? extends Plan> project, CascadesContext ctx) {
+ Plan child = project.child();
+ Plan newChild = child.accept(this, ctx);
+ if (newChild instanceof AbstractPhysicalJoin) {
+ AbstractPhysicalJoin<? extends Plan, ? extends Plan> join = (AbstractPhysicalJoin) newChild;
+ Plan left = join.left();
+ Plan right = join.right();
+ Set<Slot> leftOutput = left.getOutputSet();
+ Set<Slot> rightOutput = right.getOutputSet();
+
+ Set<Slot> usedSlots = project.getProjects().stream().flatMap(ne -> ne.getInputSlots().stream())
+ .collect(Collectors.toSet());
+ usedSlots.addAll(join.getConditionSlot());
+
+ List<NamedExpression> leftNewProjections = new ArrayList<>();
+ List<NamedExpression> rightNewProjections = new ArrayList<>();
+
+ for (Slot usedSlot : usedSlots) {
+ if (leftOutput.contains(usedSlot)) {
+ leftNewProjections.add(usedSlot);
+ } else if (rightOutput.contains(usedSlot)) {
+ rightNewProjections.add(usedSlot);
+ }
+ }
+
+ Plan newLeft;
+ if (left instanceof PhysicalDistribute) {
+ newLeft = leftNewProjections.size() != leftOutput.size() && !leftNewProjections.isEmpty()
+ ? left.withChildren(new PhysicalProject<>(leftNewProjections,
+ left.getLogicalProperties(), left.child(0)))
+ : left;
+ } else {
+ newLeft = leftNewProjections.size() != leftOutput.size() && !leftNewProjections.isEmpty()
+ ? new PhysicalProject<>(leftNewProjections, left.getLogicalProperties(),
+ left).copyStatsAndGroupIdFrom((AbstractPhysicalPlan) left)
+ : left;
+ }
+ Plan newRight;
+ if (right instanceof PhysicalDistribute) {
+ newRight = rightNewProjections.size() != rightOutput.size() && !rightNewProjections.isEmpty()
+ ? right.withChildren(new PhysicalProject<>(rightNewProjections,
+ right.getLogicalProperties(), right.child(0)))
+ : right;
+ } else {
+ newRight = rightNewProjections.size() != rightOutput.size() && !rightNewProjections.isEmpty()
+ ? new PhysicalProject<>(rightNewProjections, right.getLogicalProperties(),
+ right).copyStatsAndGroupIdFrom((AbstractPhysicalPlan) right)
+ : right;
+ }
+
+ if (newLeft != left || newRight != right) {
+ return (PhysicalProject) project.withChildren(join.withChildren(newLeft, newRight));
+ } else {
+ return project;
+ }
+ }
+ return project;
+ }
+}
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
index d89ef1acd26..60c1a74445e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PlanPostProcessors.java
@@ -59,6 +59,7 @@ public class PlanPostProcessors {
// add processor if we need
Builder<PlanPostProcessor> builder = ImmutableList.builder();
builder.add(new PushDownFilterThroughProject());
+ builder.add(new ColumnPruningPostProcessor());
builder.add(new MergeProjectPostProcessor());
builder.add(new RecomputeLogicalPropertiesProcessor());
builder.add(new AddOffsetIntoDistribute());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateGroupByConstant.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateGroupByConstant.java
index e683153e9a2..4408e64487c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateGroupByConstant.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateGroupByConstant.java
@@ -26,11 +26,14 @@ import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.util.ExpressionUtils;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Sets;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.Set;
/**
@@ -54,6 +57,7 @@ public class EliminateGroupByConstant extends OneRewriteRuleFactory {
List<Expression> groupByExprs = aggregate.getGroupByExpressions();
List<NamedExpression> outputExprs = aggregate.getOutputExpressions();
Set<Expression> slotGroupByExprs = Sets.newLinkedHashSet();
+ Map<Expression, Expression> constantExprsReplaceMap = new HashMap<>(groupByExprs.size());
Expression lit = null;
for (Expression expression : groupByExprs) {
// NOTICE: we should not use the expression after fold as new aggregate's output or group expr
@@ -64,13 +68,15 @@ public class EliminateGroupByConstant extends OneRewriteRuleFactory {
if (!foldExpression.isConstant()) {
slotGroupByExprs.add(expression);
} else {
+ constantExprsReplaceMap.put(expression, foldExpression);
lit = expression;
}
}
if (slotGroupByExprs.isEmpty() && lit != null) {
slotGroupByExprs.add(lit);
}
- return aggregate.withGroupByAndOutput(ImmutableList.copyOf(slotGroupByExprs), outputExprs);
+ return aggregate.withGroupByAndOutput(ImmutableList.copyOf(slotGroupByExprs),
+ ExpressionUtils.replaceNamedExpressions(outputExprs, constantExprsReplaceMap));
}).toRule(RuleType.ELIMINATE_GROUP_BY_CONSTANT);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
index 32e90a67efa..66852444cc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
@@ -58,7 +58,6 @@ import com.google.common.collect.Sets;
import java.util.ArrayList;
import java.util.Collection;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
@@ -69,8 +68,15 @@ import java.util.stream.Collectors;
* The abstract class for all materialized view rules
*/
public abstract class AbstractMaterializedViewRule implements ExplorationRuleFactory {
- public static final HashSet<JoinType> SUPPORTED_JOIN_TYPE_SET = Sets.newHashSet(JoinType.INNER_JOIN,
- JoinType.LEFT_OUTER_JOIN);
+ public static final Set<JoinType> SUPPORTED_JOIN_TYPE_SET = ImmutableSet.of(
+ JoinType.INNER_JOIN,
+ JoinType.LEFT_OUTER_JOIN,
+ JoinType.RIGHT_OUTER_JOIN,
+ JoinType.FULL_OUTER_JOIN,
+ JoinType.LEFT_SEMI_JOIN,
+ JoinType.RIGHT_SEMI_JOIN,
+ JoinType.LEFT_ANTI_JOIN,
+ JoinType.RIGHT_ANTI_JOIN);
/**
* The abstract template method for query rewrite, it contains the main logic, try to rewrite query by
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
index c60eb9f94d3..2ffdeb5718b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/ComparisonResult.java
@@ -17,6 +17,7 @@
package org.apache.doris.nereids.rules.exploration.mv;
+import org.apache.doris.common.Pair;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
@@ -27,6 +28,7 @@ import com.google.common.collect.ImmutableSet;
import java.util.Collection;
import java.util.List;
import java.util.Set;
+import java.util.stream.Collectors;
/**
* comparison result of view and query
@@ -111,8 +113,14 @@ public class ComparisonResult {
return this;
}
- public Builder addViewNoNullableSlot(Set<Slot> viewNoNullableSlot) {
- viewNoNullableSlotBuilder.add(ImmutableSet.copyOf(viewNoNullableSlot));
+ /**Add slots which should reject null slots in view*/
+ public Builder addViewNoNullableSlot(Pair<Set<Slot>, Set<Slot>> viewNoNullableSlotsPair) {
+ if (!viewNoNullableSlotsPair.first.isEmpty()) {
+ viewNoNullableSlotBuilder.add(viewNoNullableSlotsPair.first);
+ }
+ if (!viewNoNullableSlotsPair.second.isEmpty()) {
+ viewNoNullableSlotBuilder.add(viewNoNullableSlotsPair.second);
+ }
return this;
}
@@ -127,7 +135,9 @@ public class ComparisonResult {
public ComparisonResult build() {
Preconditions.checkArgument(valid, "Comparison result must be valid");
- return new ComparisonResult(queryBuilder.build(), queryAllPulledUpExpressionsBuilder.build(),
+ return new ComparisonResult(queryBuilder.build(),
+ queryAllPulledUpExpressionsBuilder.build().stream()
+ .filter(expr -> !expr.isInferred()).collect(Collectors.toList()),
viewBuilder.build(), viewNoNullableSlotBuilder.build(), valid, "");
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
index 341caa88094..956a974982c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
@@ -73,7 +73,11 @@ public class HyperGraphComparator {
private final Map<Edge, List<? extends Expression>> pullUpQueryExprWithEdge = new HashMap<>();
private final Map<Edge, List<? extends Expression>> pullUpViewExprWithEdge = new HashMap<>();
private final LogicalCompatibilityContext logicalCompatibilityContext;
- private final Map<JoinEdge, Pair<JoinType, Set<Slot>>> inferredViewEdgeWithCond = new HashMap<>();
+ // this records the slots which needs to reject null
+ // the key is the target join which should reject null, the value is a pair, the first value of the pair is the
+ // join type, the second value is also a pair which left represents the slots in the left of join that should
+ // reject null, right represents the slots in the right of join that should reject null.
+ private final Map<JoinEdge, Pair<JoinType, Pair<Set<Slot>, Set<Slot>>>> inferredViewEdgeWithCond = new HashMap<>();
private List<JoinEdge> viewJoinEdgesAfterInferring;
private List<FilterEdge> viewFilterEdgesAfterInferring;
private final long eliminateViewNodesMap;
@@ -263,7 +267,7 @@ public class HyperGraphComparator {
}
builder.addViewExpressions(rawFilter);
}
- for (Pair<JoinType, Set<Slot>> inferredCond : inferredViewEdgeWithCond.values()) {
+ for (Pair<JoinType, Pair<Set<Slot>, Set<Slot>>> inferredCond : inferredViewEdgeWithCond.values()) {
builder.addViewNoNullableSlot(inferredCond.second);
}
builder.addQueryAllPulledUpExpressions(
@@ -485,11 +489,14 @@ public class HyperGraphComparator {
if (noNullableChild == null) {
return false;
}
- Set<Slot> noNullableSlot = Sets.union(
- noNullableChild.first ? view.getJoin().left().getOutputSet() : ImmutableSet.of(),
- noNullableChild.second ? view.getJoin().right().getOutputSet() : ImmutableSet.of()
- );
- inferredViewEdgeWithCond.put(view, Pair.of(query.getJoinType(), noNullableSlot));
+ Pair<Set<Slot>, Set<Slot>> noNullableSlotSetPair = Pair.of(ImmutableSet.of(), ImmutableSet.of());
+ if (noNullableChild.first) {
+ noNullableSlotSetPair.first = view.getJoin().left().getOutputSet();
+ }
+ if (noNullableChild.second) {
+ noNullableSlotSetPair.second = view.getJoin().right().getOutputSet();
+ }
+ inferredViewEdgeWithCond.put(view, Pair.of(query.getJoinType(), noNullableSlotSetPair));
}
return true;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
index 8046639d1ba..914b2133613 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
@@ -66,9 +66,9 @@ public class InitMaterializationContextHook implements PlannerHook {
return;
}
Plan rewritePlan = cascadesContext.getRewritePlan();
- TableCollectorContext collectorContext = new TableCollectorContext(Sets.newHashSet());
+ TableCollectorContext collectorContext = new TableCollectorContext(Sets.newHashSet(), true);
rewritePlan.accept(TableCollector.INSTANCE, collectorContext);
- List<TableIf> collectedTables = collectorContext.getCollectedTables();
+ Set<TableIf> collectedTables = collectorContext.getCollectedTables();
if (collectedTables.isEmpty()) {
return;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/MultiDistinctSum0.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/MultiDistinctSum0.java
new file mode 100644
index 00000000000..89999258713
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/MultiDistinctSum0.java
@@ -0,0 +1,82 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.ComputePrecisionForSum;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DoubleType;
+import org.apache.doris.nereids.types.LargeIntType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/** MultiDistinctSum0 */
+public class MultiDistinctSum0 extends AggregateFunction implements UnaryExpression,
+ ExplicitlyCastableSignature, ComputePrecisionForSum, MultiDistinction, AlwaysNotNullable {
+
+ public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(BigIntType.INSTANCE).varArgs(BigIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).varArgs(DoubleType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).varArgs(LargeIntType.INSTANCE)
+ );
+
+ public MultiDistinctSum0(Expression arg0) {
+ super("multi_distinct_sum0", true, arg0);
+ }
+
+ public MultiDistinctSum0(boolean distinct, Expression arg0) {
+ super("multi_distinct_sum0", true, arg0);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (child().getDataType().isDateLikeType()) {
+ throw new AnalysisException("Sum0 in multi distinct functions do not support Date/Datetime type");
+ }
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return new Sum0(getArgument(0)).getSignatures();
+ }
+
+ @Override
+ public FunctionSignature searchSignature(List<FunctionSignature> signatures) {
+ return new Sum0(getArgument(0)).searchSignature(signatures);
+ }
+
+ @Override
+ public MultiDistinctSum0 withDistinctAndChildren(boolean distinct, List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new MultiDistinctSum0(distinct, children.get(0));
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitMultiDistinctSum0(this, context);
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum0.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum0.java
new file mode 100644
index 00000000000..b25dd5fe60e
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Sum0.java
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.ComputePrecisionForSum;
+import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.Function;
+import org.apache.doris.nereids.trees.expressions.functions.window.SupportWindowAnalytic;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DecimalV3Type;
+import org.apache.doris.nereids.types.DoubleType;
+import org.apache.doris.nereids.types.FloatType;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.types.LargeIntType;
+import org.apache.doris.nereids.types.SmallIntType;
+import org.apache.doris.nereids.types.TinyIntType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * AggregateFunction 'sum0'. sum0 returns the sum of the values which go into it like sum.
+ * It differs in that when no non null values are applied zero is returned instead of null.
+ */
+public class Sum0 extends AggregateFunction
+ implements UnaryExpression, AlwaysNotNullable, ExplicitlyCastableSignature, ComputePrecisionForSum,
+ SupportWindowAnalytic, CouldRollUp {
+
+ public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(BigIntType.INSTANCE).args(BooleanType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(TinyIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(SmallIntType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(IntegerType.INSTANCE),
+ FunctionSignature.ret(BigIntType.INSTANCE).args(BigIntType.INSTANCE),
+ FunctionSignature.ret(LargeIntType.INSTANCE).args(LargeIntType.INSTANCE),
+ FunctionSignature.ret(DecimalV3Type.WILDCARD).args(DecimalV3Type.WILDCARD),
+ FunctionSignature.ret(DoubleType.INSTANCE).args(DoubleType.INSTANCE)
+ );
+
+ /**
+ * constructor with 1 argument.
+ */
+ public Sum0(Expression arg) {
+ this(false, arg);
+ }
+
+ /**
+ * constructor with 2 argument.
+ */
+ public Sum0(boolean distinct, Expression arg) {
+ super("sum0", distinct, arg);
+ }
+
+ public MultiDistinctSum0 convertToMultiDistinct() {
+ Preconditions.checkArgument(distinct,
+ "can't convert to multi_distinct_sum because there is no distinct args");
+ return new MultiDistinctSum0(false, child());
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ DataType argType = child().getDataType();
+ if ((!argType.isNumericType() && !argType.isBooleanType() && !argType.isNullType())
+ || argType.isOnlyMetricType()) {
+ throw new AnalysisException("sum0 requires a numeric or boolean parameter: " + this.toSql());
+ }
+ }
+
+ /**
+ * withDistinctAndChildren.
+ */
+ @Override
+ public Sum0 withDistinctAndChildren(boolean distinct, List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new Sum0(distinct, children.get(0));
+ }
+
+ @Override
+ public <R, C> R accept(ExpressionVisitor<R, C> visitor, C context) {
+ return visitor.visitSum0(this, context);
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
+ }
+
+ @Override
+ public FunctionSignature searchSignature(List<FunctionSignature> signatures) {
+ if (getArgument(0).getDataType() instanceof FloatType) {
+ return FunctionSignature.ret(DoubleType.INSTANCE).args(FloatType.INSTANCE);
+ }
+ return ExplicitlyCastableSignature.super.searchSignature(signatures);
+ }
+
+ @Override
+ public Function constructRollUp(Expression param, Expression... varParams) {
+ return new Sum0(this.distinct, param);
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/AggregateFunctionVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/AggregateFunctionVisitor.java
index 594f9c75433..febe93974f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/AggregateFunctionVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/visitor/AggregateFunctionVisitor.java
@@ -51,6 +51,7 @@ import org.apache.doris.nereids.trees.expressions.functions.agg.MinBy;
import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctCount;
import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctGroupConcat;
import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctSum;
+import org.apache.doris.nereids.trees.expressions.functions.agg.MultiDistinctSum0;
import org.apache.doris.nereids.trees.expressions.functions.agg.Ndv;
import org.apache.doris.nereids.trees.expressions.functions.agg.NullableAggregateFunction;
import org.apache.doris.nereids.trees.expressions.functions.agg.OrthogonalBitmapIntersect;
@@ -66,6 +67,7 @@ import org.apache.doris.nereids.trees.expressions.functions.agg.SequenceMatch;
import org.apache.doris.nereids.trees.expressions.functions.agg.Stddev;
import org.apache.doris.nereids.trees.expressions.functions.agg.StddevSamp;
import org.apache.doris.nereids.trees.expressions.functions.agg.Sum;
+import org.apache.doris.nereids.trees.expressions.functions.agg.Sum0;
import org.apache.doris.nereids.trees.expressions.functions.agg.TopN;
import org.apache.doris.nereids.trees.expressions.functions.agg.TopNArray;
import org.apache.doris.nereids.trees.expressions.functions.agg.TopNWeighted;
@@ -162,6 +164,10 @@ public interface AggregateFunctionVisitor<R, C> {
return visitAggregateFunction(multiDistinctSum, context);
}
+ default R visitMultiDistinctSum0(MultiDistinctSum0 multiDistinctSum0, C context) {
+ return visitAggregateFunction(multiDistinctSum0, context);
+ }
+
default R visitGroupBitAnd(GroupBitAnd groupBitAnd, C context) {
return visitNullableAggregateFunction(groupBitAnd, context);
}
@@ -274,6 +280,10 @@ public interface AggregateFunctionVisitor<R, C> {
return visitNullableAggregateFunction(sum, context);
}
+ default R visitSum0(Sum0 sum0, C context) {
+ return visitAggregateFunction(sum0, context);
+ }
+
default R visitTopN(TopN topN, C context) {
return visitAggregateFunction(topN, context);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
index d7c502a9eb8..7e7f58db44d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateMTMVInfo.java
@@ -320,9 +320,9 @@ public class CreateMTMVInfo {
private void analyzeBaseTables(Plan plan) {
TableCollectorContext collectorContext =
- new TableCollector.TableCollectorContext(Sets.newHashSet(TableType.MATERIALIZED_VIEW));
+ new TableCollector.TableCollectorContext(Sets.newHashSet(TableType.MATERIALIZED_VIEW), true);
plan.accept(TableCollector.INSTANCE, collectorContext);
- List<TableIf> collectedTables = collectorContext.getCollectedTables();
+ Set<TableIf> collectedTables = collectorContext.getCollectedTables();
if (!CollectionUtils.isEmpty(collectedTables)) {
throw new AnalysisException("can not contain MATERIALIZED_VIEW");
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
index ba56cfa4a8b..09c670e1b8e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/IndexDefinition.java
@@ -105,8 +105,8 @@ public class IndexDefinition {
}
if (!(colType.isDateLikeType() || colType.isDecimalLikeType()
|| colType.isIntegralType() || colType.isStringLikeType()
- || colType.isBooleanType())) {
- // TODO add colType.isVariantType() and colType.isAggState()
+ || colType.isBooleanType() || colType.isVariantType())) {
+ // TODO add colType.isAggState()
throw new AnalysisException(colType + " is not supported in " + indexType.toString()
+ " index. " + "invalid index: " + name);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTestScan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTestScan.java
index 7d326ade3d6..94cae4b955d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTestScan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalTestScan.java
@@ -23,6 +23,7 @@ import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PlanType;
import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.nereids.util.Utils;
import java.util.List;
@@ -68,4 +69,9 @@ public class LogicalTestScan extends LogicalCatalogRelation {
Optional<LogicalProperties> logicalProperties, List<Plan> children) {
return new LogicalTestScan(relationId, table, qualifier, groupExpression, logicalProperties);
}
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitLogicalTestScan(this, context);
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
index 1e9135d600e..3ebf9390d84 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/AbstractPhysicalPlan.java
@@ -149,7 +149,7 @@ public abstract class AbstractPhysicalPlan extends AbstractPlan implements Physi
return this;
}
- public <T extends AbstractPhysicalPlan> T copyStatsAndGroupIdFrom(T from) {
+ public <T extends AbstractPhysicalPlan> AbstractPhysicalPlan copyStatsAndGroupIdFrom(T from) {
T newPlan = (T) withPhysicalPropertiesAndStats(
from.getPhysicalProperties(), from.getStats());
newPlan.setMutableState(MutableState.KEY_GROUP, from.getGroupIdAsString());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/RelationVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/RelationVisitor.java
index adfe4991665..5eaf0b72136 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/RelationVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/RelationVisitor.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.trees.plans.visitor;
import org.apache.doris.nereids.analyzer.UnboundOneRowRelation;
import org.apache.doris.nereids.analyzer.UnboundRelation;
import org.apache.doris.nereids.analyzer.UnboundTVFRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalDeferMaterializeOlapScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalEsScan;
@@ -31,6 +32,8 @@ import org.apache.doris.nereids.trees.plans.logical.LogicalOneRowRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
import org.apache.doris.nereids.trees.plans.logical.LogicalSchemaScan;
import org.apache.doris.nereids.trees.plans.logical.LogicalTVFRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTestScan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalCatalogRelation;
import org.apache.doris.nereids.trees.plans.physical.PhysicalDeferMaterializeOlapScan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalEmptyRelation;
import org.apache.doris.nereids.trees.plans.physical.PhysicalEsScan;
@@ -54,8 +57,16 @@ public interface RelationVisitor<R, C> {
R visitLogicalRelation(LogicalRelation logicalRelation, C context);
+ default R visitLogicalCatalogRelation(LogicalCatalogRelation catalogRelation, C context) {
+ return visitLogicalRelation(catalogRelation, context);
+ }
+
R visitPhysicalRelation(PhysicalRelation physicalRelation, C context);
+ default R visitPhysicalCatalogRelation(PhysicalCatalogRelation catalogRelation, C context) {
+ return visitPhysicalRelation(catalogRelation, context);
+ }
+
// *******************************
// unbound relations
// *******************************
@@ -81,28 +92,28 @@ public interface RelationVisitor<R, C> {
}
default R visitLogicalEsScan(LogicalEsScan esScan, C context) {
- return visitLogicalRelation(esScan, context);
+ return visitLogicalCatalogRelation(esScan, context);
}
default R visitLogicalFileScan(LogicalFileScan fileScan, C context) {
- return visitLogicalRelation(fileScan, context);
+ return visitLogicalCatalogRelation(fileScan, context);
}
default R visitLogicalJdbcScan(LogicalJdbcScan jdbcScan, C context) {
- return visitLogicalRelation(jdbcScan, context);
+ return visitLogicalCatalogRelation(jdbcScan, context);
}
default R visitLogicalOdbcScan(LogicalOdbcScan odbcScan, C context) {
- return visitLogicalRelation(odbcScan, context);
+ return visitLogicalCatalogRelation(odbcScan, context);
}
default R visitLogicalOlapScan(LogicalOlapScan olapScan, C context) {
- return visitLogicalRelation(olapScan, context);
+ return visitLogicalCatalogRelation(olapScan, context);
}
default R visitLogicalDeferMaterializeOlapScan(
LogicalDeferMaterializeOlapScan deferMaterializeOlapScan, C context) {
- return visitLogicalRelation(deferMaterializeOlapScan, context);
+ return visitLogicalCatalogRelation(deferMaterializeOlapScan, context);
}
default R visitLogicalOneRowRelation(LogicalOneRowRelation oneRowRelation, C context) {
@@ -110,13 +121,17 @@ public interface RelationVisitor<R, C> {
}
default R visitLogicalSchemaScan(LogicalSchemaScan schemaScan, C context) {
- return visitLogicalRelation(schemaScan, context);
+ return visitLogicalCatalogRelation(schemaScan, context);
}
default R visitLogicalTVFRelation(LogicalTVFRelation tvfRelation, C context) {
return visitLogicalRelation(tvfRelation, context);
}
+ default R visitLogicalTestScan(LogicalTestScan testScan, C context) {
+ return visitLogicalCatalogRelation(testScan, context);
+ }
+
// *******************************
// physical relations
// *******************************
@@ -126,28 +141,28 @@ public interface RelationVisitor<R, C> {
}
default R visitPhysicalEsScan(PhysicalEsScan esScan, C context) {
- return visitPhysicalRelation(esScan, context);
+ return visitPhysicalCatalogRelation(esScan, context);
}
default R visitPhysicalFileScan(PhysicalFileScan fileScan, C context) {
- return visitPhysicalRelation(fileScan, context);
+ return visitPhysicalCatalogRelation(fileScan, context);
}
default R visitPhysicalJdbcScan(PhysicalJdbcScan jdbcScan, C context) {
- return visitPhysicalRelation(jdbcScan, context);
+ return visitPhysicalCatalogRelation(jdbcScan, context);
}
default R visitPhysicalOdbcScan(PhysicalOdbcScan odbcScan, C context) {
- return visitPhysicalRelation(odbcScan, context);
+ return visitPhysicalCatalogRelation(odbcScan, context);
}
default R visitPhysicalOlapScan(PhysicalOlapScan olapScan, C context) {
- return visitPhysicalRelation(olapScan, context);
+ return visitPhysicalCatalogRelation(olapScan, context);
}
default R visitPhysicalDeferMaterializeOlapScan(
PhysicalDeferMaterializeOlapScan deferMaterializeOlapScan, C context) {
- return visitPhysicalRelation(deferMaterializeOlapScan, context);
+ return visitPhysicalCatalogRelation(deferMaterializeOlapScan, context);
}
default R visitPhysicalOneRowRelation(PhysicalOneRowRelation oneRowRelation, C context) {
@@ -155,7 +170,7 @@ public interface RelationVisitor<R, C> {
}
default R visitPhysicalSchemaScan(PhysicalSchemaScan schemaScan, C context) {
- return visitPhysicalRelation(schemaScan, context);
+ return visitPhysicalCatalogRelation(schemaScan, context);
}
default R visitPhysicalTVFRelation(PhysicalTVFRelation tvfRelation, C context) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/TableCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/TableCollector.java
index a3c874f6370..3736ab60b0f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/TableCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/TableCollector.java
@@ -17,33 +17,73 @@
package org.apache.doris.nereids.trees.plans.visitor;
+import org.apache.doris.catalog.MTMV;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.TableIf.TableType;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.mtmv.MTMVCache;
+import org.apache.doris.mtmv.MTMVPlanUtil;
import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalCatalogRelation;
import org.apache.doris.nereids.trees.plans.visitor.TableCollector.TableCollectorContext;
-import java.util.ArrayList;
-import java.util.List;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.HashSet;
import java.util.Set;
/**
* Collect the table in plan
* Note: will not get table if table is eliminated by EmptyRelation in rewrite.
+ * View expand is in RBO, if call this method with the plan after RBO, this will get base tables in view, or will not.
+ * Materialized view is extended or not can be controlled by the field expand
*/
-public class TableCollector extends DefaultPlanVisitor<Void, TableCollectorContext> {
+public class TableCollector extends DefaultPlanVisitor<Plan, TableCollectorContext> {
public static final TableCollector INSTANCE = new TableCollector();
+ private static final Logger LOG = LogManager.getLogger(TableCollector.class);
@Override
- public Void visit(Plan plan, TableCollectorContext context) {
- if (plan instanceof CatalogRelation) {
- TableIf table = ((CatalogRelation) plan).getTable();
- if (context.getTargetTableTypes().isEmpty() || context.getTargetTableTypes().contains(table.getType())) {
- context.getCollectedTables().add(table);
- }
+ public Plan visitLogicalCatalogRelation(LogicalCatalogRelation catalogRelation, TableCollectorContext context) {
+ TableIf table = catalogRelation.getTable();
+ if (context.getTargetTableTypes().isEmpty() || context.getTargetTableTypes().contains(table.getType())) {
+ context.getCollectedTables().add(table);
+ }
+ if (table instanceof MTMV) {
+ expandMvAndCollect((MTMV) table, context);
+ }
+ return catalogRelation;
+ }
+
+ @Override
+ public Plan visitPhysicalCatalogRelation(PhysicalCatalogRelation catalogRelation, TableCollectorContext context) {
+ TableIf table = catalogRelation.getTable();
+ if (context.getTargetTableTypes().isEmpty() || context.getTargetTableTypes().contains(table.getType())) {
+ context.getCollectedTables().add(table);
+ }
+ if (table instanceof MTMV) {
+ expandMvAndCollect((MTMV) table, context);
+ }
+ return catalogRelation;
+ }
+
+ private void expandMvAndCollect(MTMV mtmv, TableCollectorContext context) {
+ if (!context.isExpand()) {
+ return;
+ }
+ try {
+ MTMVCache expandedMv = MTMVCache.from(mtmv, MTMVPlanUtil.createMTMVContext(mtmv));
+ expandedMv.getLogicalPlan().accept(this, context);
+ } catch (AnalysisException e) {
+ LOG.error(String.format(
+ "table collector expand fail, mtmv name is %s, targetTableTypes is %s",
+ mtmv.getName(), context.targetTableTypes), e);
+ throw new org.apache.doris.nereids.exceptions.AnalysisException(
+ String.format("expand mv and collect table fail, mv name is %s, mv sql is %s",
+ mtmv.getName(), mtmv.getQuerySql()), e);
}
- return super.visit(plan, context);
}
/**
@@ -51,19 +91,26 @@ public class TableCollector extends DefaultPlanVisitor<Void, TableCollectorConte
* and the result of collect.
*/
public static final class TableCollectorContext {
- private final List<TableIf> collectedTables = new ArrayList<>();
+ private final Set<TableIf> collectedTables = new HashSet<>();
private final Set<TableType> targetTableTypes;
+ // if expand the mv or not
+ private final boolean expand;
- public TableCollectorContext(Set<TableType> targetTableTypes) {
+ public TableCollectorContext(Set<TableType> targetTableTypes, boolean expand) {
this.targetTableTypes = targetTableTypes;
+ this.expand = expand;
}
- public List<TableIf> getCollectedTables() {
+ public Set<TableIf> getCollectedTables() {
return collectedTables;
}
public Set<TableType> getTargetTableTypes() {
return targetTableTypes;
}
+
+ public boolean isExpand() {
+ return expand;
+ }
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
index 470eee8153b..e9905a44a41 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java
@@ -297,6 +297,9 @@ public abstract class DataType {
case "ipv6":
dataType = IPv6Type.INSTANCE;
break;
+ case "variant":
+ dataType = VariantType.INSTANCE;
+ break;
default:
throw new AnalysisException("Nereids do not support type: " + type);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/plsql/metastore/PlsqlManager.java b/fe/fe-core/src/main/java/org/apache/doris/plsql/metastore/PlsqlManager.java
index 87cbd7b58a3..dc91d344f63 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/plsql/metastore/PlsqlManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/plsql/metastore/PlsqlManager.java
@@ -22,6 +22,7 @@ import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.gson.GsonUtils;
+import com.google.common.collect.ImmutableList;
import com.google.common.collect.Maps;
import com.google.gson.annotations.SerializedName;
import org.apache.logging.log4j.LogManager;
@@ -34,7 +35,20 @@ import java.util.Map;
public class PlsqlManager implements Writable {
private static final Logger LOG = LogManager.getLogger(PlsqlManager.class);
-
+ public static final ImmutableList<String> ROUTINE_INFO_TITLE_NAMES = new ImmutableList.Builder<String>()
+ .add("SPECIFIC_NAME").add("ROUTINE_CATALOG").add("ROUTINE_SCHEMA").add("ROUTINE_NAME")
+ .add("ROUTINE_TYPE")
+ .add("DTD_IDENTIFIER").add("ROUTINE_BODY")
+ .add("ROUTINE_DEFINITION").add("EXTERNAL_NAME")
+ .add("EXTERNAL_LANGUAGE").add("PARAMETER_STYLE")
+ .add("IS_DETERMINISTIC")
+ .add("SQL_DATA_ACCESS").add("SQL_PATH")
+ .add("SECURITY_TYPE").add("CREATED")
+ .add("LAST_ALTERED").add("SQL_MODE")
+ .add("ROUTINE_COMMENT")
+ .add("DEFINER").add("CHARACTER_SET_CLIENT")
+ .add("COLLATION_CONNECTION").add("DATABASE_COLLATION")
+ .build();
@SerializedName(value = "nameToStoredProcedures")
Map<PlsqlProcedureKey, PlsqlStoredProcedure> nameToStoredProcedures = Maps.newConcurrentMap();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index 61beeb57ce6..a0d369eafde 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -909,6 +909,9 @@ public class ShowExecutor {
if (tbl.getName().startsWith(FeConstants.TEMP_MATERIZLIZE_DVIEW_PREFIX)) {
continue;
}
+ if (showTableStmt.getType() != null && tbl.getType() != showTableStmt.getType()) {
+ continue;
+ }
if (matcher != null && !matcher.match(tbl.getName())) {
continue;
}
@@ -2532,16 +2535,18 @@ public class ShowExecutor {
private void getStatsForAllColumns(List<Pair<Pair<String, String>, ColumnStatistic>> columnStatistics,
TableIf tableIf) throws AnalysisException {
List<ResultRow> resultRows = StatisticsRepository.queryColumnStatisticsForTable(tableIf.getId());
+ // row[4] is index id, row[5] is column name.
for (ResultRow row : resultRows) {
- String indexName = "N/A";
+ String indexName = tableIf.getName();
long indexId = Long.parseLong(row.get(4));
- if (indexId != -1) {
- indexName = ((OlapTable) tableIf).getIndexNameById(indexId);
- if (indexName == null) {
- continue;
- }
+ if (tableIf instanceof OlapTable) {
+ OlapTable olapTable = (OlapTable) tableIf;
+ indexName = olapTable.getIndexNameById(indexId == -1 ? olapTable.getBaseIndexId() : indexId);
+ }
+ if (indexName == null) {
+ continue;
}
- columnStatistics.add(Pair.of(Pair.of(row.get(5), indexName), ColumnStatistic.fromResultRow(row)));
+ columnStatistics.add(Pair.of(Pair.of(indexName, row.get(5)), ColumnStatistic.fromResultRow(row)));
}
}
@@ -2558,28 +2563,29 @@ public class ShowExecutor {
indexIds.add(-1L);
}
for (long indexId : indexIds) {
- String indexName = "N/A";
- if (indexId != -1) {
- indexName = ((OlapTable) tableIf).getIndexNameById(indexId);
- if (indexName == null) {
- continue;
- }
+ String indexName = tableIf.getName();
+ if (tableIf instanceof OlapTable) {
+ OlapTable olapTable = (OlapTable) tableIf;
+ indexName = olapTable.getIndexNameById(indexId == -1 ? olapTable.getBaseIndexId() : indexId);
+ }
+ if (indexName == null) {
+ continue;
}
// Show column statistics in columnStatisticsCache.
if (showCache) {
ColumnStatistic columnStatistic = Env.getCurrentEnv().getStatisticsCache().getColumnStatistics(
tableIf.getDatabase().getCatalog().getId(),
tableIf.getDatabase().getId(), tableIf.getId(), indexId, colName);
- columnStatistics.add(Pair.of(Pair.of(colName, indexName), columnStatistic));
+ columnStatistics.add(Pair.of(Pair.of(indexName, colName), columnStatistic));
} else if (partitionNames == null) {
ColumnStatistic columnStatistic =
StatisticsRepository.queryColumnStatisticsByName(tableIf.getId(), indexId, colName);
- columnStatistics.add(Pair.of(Pair.of(colName, indexName), columnStatistic));
+ columnStatistics.add(Pair.of(Pair.of(indexName, colName), columnStatistic));
} else {
String finalIndexName = indexName;
columnStatistics.addAll(StatisticsRepository.queryColumnStatisticsByPartitions(tableName,
colName, partitionNames.getPartitionNames())
- .stream().map(s -> Pair.of(Pair.of(colName, finalIndexName), s))
+ .stream().map(s -> Pair.of(Pair.of(finalIndexName, colName), s))
.collect(Collectors.toList()));
}
}
@@ -2983,7 +2989,7 @@ public class ShowExecutor {
if (table instanceof OlapTable && analysisInfo.indexId != -1) {
row.add(((OlapTable) table).getIndexNameById(analysisInfo.indexId));
} else {
- row.add("N/A");
+ row.add(table.getName());
}
row.add(analysisInfo.message);
row.add(TimeUtils.DATETIME_FORMAT.format(
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index 40af03d9237..783323b03fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -820,7 +820,8 @@ public class StmtExecutor {
deadCloudClusterStatus);
if (Strings.isNullOrEmpty(deadCloudClusterStatus)
|| ClusterStatus.valueOf(deadCloudClusterStatus) != ClusterStatus.NORMAL) {
- CloudSystemInfoService.waitForAutoStart(deadCloudClusterClusterName);
+ ((CloudSystemInfoService) Env.getCurrentSystemInfo())
+ .waitForAutoStart(deadCloudClusterClusterName);
}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java
index 4a220252afe..482d2f6f11a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java
@@ -71,6 +71,8 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
public static final String SPILL_THRESHOLD_HIGH_WATERMARK = "spill_threshold_high_watermark";
+ public static final String TAG = "tag";
+
// NOTE(wb): all property is not required, some properties default value is set in be
// default value is as followed
// cpu_share=1024, memory_limit=0%(0 means not limit), enable_memory_overcommit=true
@@ -78,7 +80,8 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
.add(CPU_SHARE).add(MEMORY_LIMIT).add(ENABLE_MEMORY_OVERCOMMIT).add(MAX_CONCURRENCY)
.add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM)
.add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM)
- .add(SPILL_THRESHOLD_LOW_WATERMARK).add(SPILL_THRESHOLD_HIGH_WATERMARK).build();
+ .add(SPILL_THRESHOLD_LOW_WATERMARK).add(SPILL_THRESHOLD_HIGH_WATERMARK)
+ .add(TAG).build();
public static final int SPILL_LOW_WATERMARK_DEFAULT_VALUE = 50;
public static final int SPILL_HIGH_WATERMARK_DEFAULT_VALUE = 80;
@@ -142,6 +145,9 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
}
this.properties.put(SPILL_THRESHOLD_HIGH_WATERMARK, highWatermarkStr);
}
+ if (properties.containsKey(TAG)) {
+ this.properties.put(TAG, properties.get(TAG).toLowerCase());
+ }
resetQueueProperty(properties);
}
@@ -206,11 +212,25 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
if (properties.containsKey(CPU_HARD_LIMIT)) {
String cpuHardLimit = properties.get(CPU_HARD_LIMIT);
- if (cpuHardLimit.endsWith("%")) {
- cpuHardLimit = cpuHardLimit.substring(0, cpuHardLimit.length() - 1);
- }
- if (!StringUtils.isNumeric(cpuHardLimit) || Long.parseLong(cpuHardLimit) <= 0) {
- throw new DdlException(CPU_HARD_LIMIT + " " + cpuHardLimit + " requires a positive integer.");
+ String originValue = cpuHardLimit;
+ try {
+ boolean endWithSign = false;
+ if (cpuHardLimit.endsWith("%")) {
+ cpuHardLimit = cpuHardLimit.substring(0, cpuHardLimit.length() - 1);
+ endWithSign = true;
+ }
+
+ int intVal = Integer.parseInt(cpuHardLimit);
+ if (endWithSign && intVal == -1) {
+ throw new NumberFormatException();
+ }
+ if (!(intVal >= 1 && intVal <= 100) && -1 != intVal) {
+ throw new NumberFormatException();
+ }
+ } catch (NumberFormatException e) {
+ throw new DdlException(
+ "workload group's " + WorkloadGroup.CPU_HARD_LIMIT
+ + " must be a positive integer[1,100] or -1, but input value is " + originValue);
}
}
@@ -395,7 +415,9 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
if (CPU_HARD_LIMIT.equals(key)) {
String val = properties.get(key);
if (StringUtils.isEmpty(val)) { // cpu_hard_limit is not required
- row.add("0%");
+ row.add("-1");
+ } else if ("-1".equals(val)) {
+ row.add(val);
} else {
row.add(val + "%");
}
@@ -431,6 +453,13 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
row.add(qq == null ? "0" : String.valueOf(qq.getCurrentRunningQueryNum()));
} else if (QueryQueue.WAITING_QUERY_NUM.equals(key)) {
row.add(qq == null ? "0" : String.valueOf(qq.getCurrentWaitingQueryNum()));
+ } else if (TAG.equals(key)) {
+ String val = properties.get(key);
+ if (StringUtils.isEmpty(val)) {
+ row.add("");
+ } else {
+ row.add(val);
+ }
} else {
row.add(properties.get(key));
}
@@ -442,6 +471,10 @@ public class WorkloadGroup implements Writable, GsonPostProcessable {
return cpuHardLimit;
}
+ public String getTag() {
+ return properties.get(TAG);
+ }
+
@Override
public String toString() {
return GsonUtils.GSON.toJson(this);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
index 08de0ce338a..53269b15049 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java
@@ -48,6 +48,7 @@ import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.gson.annotations.SerializedName;
+import org.apache.commons.lang3.StringUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -59,7 +60,6 @@ import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
-import java.util.Objects;
import java.util.concurrent.locks.ReentrantReadWriteLock;
public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
@@ -73,6 +73,7 @@ public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
.add(WorkloadGroup.SCAN_THREAD_NUM).add(WorkloadGroup.MAX_REMOTE_SCAN_THREAD_NUM)
.add(WorkloadGroup.MIN_REMOTE_SCAN_THREAD_NUM)
.add(WorkloadGroup.SPILL_THRESHOLD_LOW_WATERMARK).add(WorkloadGroup.SPILL_THRESHOLD_HIGH_WATERMARK)
+ .add(WorkloadGroup.TAG)
.add(QueryQueue.RUNNING_QUERY_NUM).add(QueryQueue.WAITING_QUERY_NUM)
.build();
@@ -301,38 +302,44 @@ public class WorkloadGroupMgr implements Writable, GsonPostProcessable {
LOG.info("Create workload group success: {}", workloadGroup);
}
- private void checkGlobalUnlock(WorkloadGroup workloadGroup, WorkloadGroup old) throws DdlException {
- double totalMemoryLimit = idToWorkloadGroup.values().stream().mapToDouble(WorkloadGroup::getMemoryLimitPercent)
- .sum() + workloadGroup.getMemoryLimitPercent();
- if (!Objects.isNull(old)) {
- totalMemoryLimit -= old.getMemoryLimitPercent();
- }
- if (totalMemoryLimit > 100.0 + 1e-6) {
- throw new DdlException(
- "The sum of all workload group " + WorkloadGroup.MEMORY_LIMIT + " cannot be greater than 100.0%.");
- }
-
- // 1, check new group
- int newGroupCpuHardLimit = workloadGroup.getCpuHardLimit();
- if (newGroupCpuHardLimit > 100 || newGroupCpuHardLimit < 0) {
- throw new DdlException(
- "new group's " + WorkloadGroup.CPU_HARD_LIMIT
- + " value can not be greater than 100% or less than or equal 0%");
- }
-
- // 2, check sum of all cpu hard limit
+ // NOTE: used for checking sum value of 100% for cpu_hard_limit and memory_limit
+ // when create/alter workload group with same tag.
+ // when oldWg is null it means caller is an alter stmt.
+ private void checkGlobalUnlock(WorkloadGroup newWg, WorkloadGroup oldWg) throws DdlException {
+ String wgTag = newWg.getTag();
+ double sumOfAllMemLimit = 0;
int sumOfAllCpuHardLimit = 0;
for (Map.Entry<Long, WorkloadGroup> entry : idToWorkloadGroup.entrySet()) {
- if (old != null && entry.getKey() == old.getId()) {
+ WorkloadGroup wg = entry.getValue();
+ if (!StringUtils.equals(wgTag, wg.getTag())) {
+ continue;
+ }
+
+ if (oldWg != null && entry.getKey() == oldWg.getId()) {
continue;
}
- sumOfAllCpuHardLimit += entry.getValue().getCpuHardLimit();
+
+ if (wg.getCpuHardLimit() > 0) {
+ sumOfAllCpuHardLimit += wg.getCpuHardLimit();
+ }
+ if (wg.getMemoryLimitPercent() > 0) {
+ sumOfAllMemLimit += wg.getMemoryLimitPercent();
+ }
+ }
+
+ sumOfAllMemLimit += newWg.getMemoryLimitPercent();
+ sumOfAllCpuHardLimit += newWg.getCpuHardLimit();
+
+ if (sumOfAllMemLimit > 100.0 + 1e-6) {
+ throw new DdlException(
+ "The sum of all workload group " + WorkloadGroup.MEMORY_LIMIT + " within tag " + wgTag
+ + " cannot be greater than 100.0%.");
}
- sumOfAllCpuHardLimit += newGroupCpuHardLimit;
if (sumOfAllCpuHardLimit > 100) {
- throw new DdlException("sum of all workload group " + WorkloadGroup.CPU_HARD_LIMIT
- + " can not be greater than 100% ");
+ throw new DdlException(
+ "sum of all workload group " + WorkloadGroup.CPU_HARD_LIMIT + " within tag "
+ + wgTag + " can not be greater than 100% ");
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index 992c512e402..60ae26df757 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -285,9 +285,9 @@ public class FrontendServiceImpl implements FrontendService.Iface {
private MasterImpl masterImpl;
private ExecuteEnv exeEnv;
- // key is txn id,value is index of plan fragment instance, it's used by multi table request plan
- private ConcurrentHashMap<Long, Integer> multiTableFragmentInstanceIdIndexMap =
- new ConcurrentHashMap<>(64);
+ // key is txn id,value is index of plan fragment instance, it's used by multi
+ // table request plan
+ private ConcurrentHashMap<Long, Integer> multiTableFragmentInstanceIdIndexMap = new ConcurrentHashMap<>(64);
private final Map<TUniqueId, ConnectContext> proxyQueryIdToConnCtx =
new ConcurrentHashMap<>(64);
@@ -352,8 +352,9 @@ public class FrontendServiceImpl implements FrontendService.Iface {
}
// check cooldownMetaId of all replicas are the same
List<Replica> replicas = Env.getCurrentEnv().getTabletInvertedIndex().getReplicas(info.tablet_id);
- // FIXME(plat1ko): We only delete remote files when tablet is under a stable state: enough replicas and
- // all replicas are alive. Are these conditions really sufficient or necessary?
+ // FIXME(plat1ko): We only delete remote files when tablet is under a stable
+ // state: enough replicas and
+ // all replicas are alive. Are these conditions really sufficient or necessary?
if (replicas.size() < replicaNum) {
LOG.info("num replicas are not enough, tablet={}", info.tablet_id);
return;
@@ -669,7 +670,6 @@ public class FrontendServiceImpl implements FrontendService.Iface {
}
PatternMatcher finalMatcher = matcher;
-
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<?> future = executor.submit(() -> {
@@ -1013,7 +1013,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
LOG.debug("receive forwarded stmt {} from FE: {}", params.getStmtId(), params.getClientNodeHost());
}
ConnectContext context = new ConnectContext(null, true);
- // Set current connected FE to the client address, so that we can know where this request come from.
+ // Set current connected FE to the client address, so that we can know where
+ // this request come from.
context.setCurrentConnectedFEIp(params.getClientNodeHost());
if (Config.isCloudMode() && !Strings.isNullOrEmpty(params.getCloudCluster())) {
context.setCloudCluster(params.getCloudCluster());
@@ -1359,7 +1360,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
OlapTable table = (OlapTable) db.getTableOrMetaException(tbl, TableType.OLAP);
tables.add(table);
}
- // if it has multi table, use multi table and update multi table running transaction table ids
+ // if it has multi table, use multi table and update multi table running
+ // transaction table ids
if (CollectionUtils.isNotEmpty(request.getTbls())) {
List<Long> multiTableIds = tables.stream().map(Table::getId).collect(Collectors.toList());
Env.getCurrentGlobalTransactionMgr()
@@ -1555,7 +1557,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
return result;
}
- // return true if commit success and publish success, return false if publish timeout
+ // return true if commit success and publish success, return false if publish
+ // timeout
private boolean loadTxnCommitImpl(TLoadTxnCommitRequest request) throws UserException {
if (request.isSetAuthCode()) {
// TODO(cmy): find a way to check
@@ -1633,7 +1636,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
return result;
}
- // return true if commit success and publish success, return false if publish timeout
+ // return true if commit success and publish success, return false if publish
+ // timeout
private boolean commitTxnImpl(TCommitTxnRequest request) throws UserException {
/// Check required arg: user, passwd, db, txn_id, commit_infos
if (!request.isSetUser()) {
@@ -1698,7 +1702,6 @@ public class FrontendServiceImpl implements FrontendService.Iface {
// Step 4: get timeout
long timeoutMs = request.isSetThriftRpcTimeoutMs() ? request.getThriftRpcTimeoutMs() / 2 : 5000;
-
// Step 5: commit and publish
return Env.getCurrentGlobalTransactionMgr()
.commitAndPublishTransaction(db, tableList,
@@ -1949,7 +1952,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
}
/**
- * For first-class multi-table scenarios, we should store the mapping between Txn and data source type in a common
+ * For first-class multi-table scenarios, we should store the mapping between
+ * Txn and data source type in a common
* place. Since there is only Kafka now, we should do this first.
*/
private void buildMultiTableStreamLoadTask(StreamLoadTask baseTaskInfo, long txnId) {
@@ -2002,7 +2006,7 @@ public class FrontendServiceImpl implements FrontendService.Iface {
ctx.setCurrentUserIdentity(currentUser.get(0));
}
LOG.info("one stream multi table load use cloud cluster {}", request.getCloudCluster());
- //ctx.setCloudCluster();
+ // ctx.setCloudCluster();
if (!Strings.isNullOrEmpty(request.getCloudCluster())) {
if (Strings.isNullOrEmpty(request.getUser())) {
ctx.setCloudCluster(request.getCloudCluster());
@@ -2055,7 +2059,7 @@ public class FrontendServiceImpl implements FrontendService.Iface {
RoutineLoadJob routineLoadJob = Env.getCurrentEnv().getRoutineLoadManager()
.getRoutineLoadJobByMultiLoadTaskTxnId(request.getTxnId());
routineLoadJob.updateState(JobState.PAUSED, new ErrorReason(InternalErrorCode.CANNOT_RESUME_ERR,
- "failed to get stream load plan, " + exception.getMessage()), false);
+ "failed to get stream load plan, " + exception.getMessage()), false);
} catch (UserException e) {
LOG.warn("catch update routine load job error.", e);
}
@@ -2112,9 +2116,9 @@ public class FrontendServiceImpl implements FrontendService.Iface {
coord.setQueryType(TQueryType.LOAD);
TableIf table = httpStreamParams.getTable();
if (table instanceof OlapTable) {
- boolean isEnableMemtableOnSinkNode =
- ((OlapTable) table).getTableProperty().getUseSchemaLightChange()
- ? coord.getQueryOptions().isEnableMemtableOnSinkNode() : false;
+ boolean isEnableMemtableOnSinkNode = ((OlapTable) table).getTableProperty().getUseSchemaLightChange()
+ ? coord.getQueryOptions().isEnableMemtableOnSinkNode()
+ : false;
coord.getQueryOptions().setEnableMemtableOnSinkNode(isEnableMemtableOnSinkNode);
}
httpStreamParams.setParams(coord.getStreamLoadPlan());
@@ -2239,7 +2243,7 @@ public class FrontendServiceImpl implements FrontendService.Iface {
try {
if (!((OlapTable) table).getTableProperty().getUseSchemaLightChange()
&& (request.getGroupCommitMode() != null
- && !request.getGroupCommitMode().equals("off_mode"))) {
+ && !request.getGroupCommitMode().equals("off_mode"))) {
throw new UserException(
"table light_schema_change is false, can't do stream load with group commit mode");
}
@@ -2906,7 +2910,6 @@ public class FrontendServiceImpl implements FrontendService.Iface {
throw new UserException("prev_commit_seq is not set");
}
-
// step 1: check auth
if (Strings.isNullOrEmpty(request.getToken())) {
checkSingleTablePasswordAndPrivs(request.getUser(), request.getPasswd(), request.getDb(),
@@ -2999,7 +3002,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
// getSnapshotImpl
private TGetSnapshotResult getSnapshotImpl(TGetSnapshotRequest request, String clientIp)
throws UserException {
- // Step 1: Check all required arg: user, passwd, db, label_name, snapshot_name, snapshot_type
+ // Step 1: Check all required arg: user, passwd, db, label_name, snapshot_name,
+ // snapshot_type
if (!request.isSetUser()) {
throw new UserException("user is not set");
}
@@ -3080,7 +3084,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
// restoreSnapshotImpl
private TRestoreSnapshotResult restoreSnapshotImpl(TRestoreSnapshotRequest request, String clientIp)
throws UserException {
- // Step 1: Check all required arg: user, passwd, db, label_name, repo_name, meta, info
+ // Step 1: Check all required arg: user, passwd, db, label_name, repo_name,
+ // meta, info
if (!request.isSetUser()) {
throw new UserException("user is not set");
}
@@ -3347,7 +3352,6 @@ public class FrontendServiceImpl implements FrontendService.Iface {
throw new UserException("prev_commit_seq is not set");
}
-
// step 1: check auth
if (Strings.isNullOrEmpty(request.getToken())) {
checkSingleTablePasswordAndPrivs(request.getUser(), request.getPasswd(), request.getDb(),
@@ -3597,7 +3601,7 @@ public class FrontendServiceImpl implements FrontendService.Iface {
@Override
public TReplacePartitionResult replacePartition(TReplacePartitionRequest request) throws TException {
- LOG.info("Receive create partition request: {}", request);
+ LOG.info("Receive replace partition request: {}", request);
long dbId = request.getDbId();
long tableId = request.getTableId();
List<Long> partitionIds = request.getPartitionIds();
@@ -3642,8 +3646,10 @@ public class FrontendServiceImpl implements FrontendService.Iface {
List<String> allReqPartNames; // all request partitions
try {
taskLock.lock();
- // we dont lock the table. other thread in this txn will be controled by taskLock.
- // if we have already replaced. dont do it again, but acquire the recorded new partition directly.
+ // we dont lock the table. other thread in this txn will be controled by
+ // taskLock.
+ // if we have already replaced. dont do it again, but acquire the recorded new
+ // partition directly.
// if not by this txn, just let it fail naturally is ok.
List<Long> replacedPartIds = overwriteManager.tryReplacePartitionIds(taskGroupId, partitionIds);
// here if replacedPartIds still have null. this will throw exception.
@@ -3653,7 +3659,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
.filter(i -> partitionIds.get(i) == replacedPartIds.get(i)) // equal means not replaced
.mapToObj(partitionIds::get)
.collect(Collectors.toList());
- // from here we ONLY deal the pending partitions. not include the dealed(by others).
+ // from here we ONLY deal the pending partitions. not include the dealed(by
+ // others).
if (!pendingPartitionIds.isEmpty()) {
// below two must have same order inner.
List<String> pendingPartitionNames = olapTable.uncheckedGetPartNamesById(pendingPartitionIds);
@@ -3664,7 +3671,8 @@ public class FrontendServiceImpl implements FrontendService.Iface {
overwriteManager.registerTaskInGroup(taskGroupId, taskId);
InsertOverwriteUtil.addTempPartitions(olapTable, pendingPartitionNames, tempPartitionNames);
InsertOverwriteUtil.replacePartition(olapTable, pendingPartitionNames, tempPartitionNames);
- // now temp partitions are bumped up and use new names. we get their ids and record them.
+ // now temp partitions are bumped up and use new names. we get their ids and
+ // record them.
List<Long> newPartitionIds = new ArrayList<Long>();
for (String newPartName : pendingPartitionNames) {
newPartitionIds.add(olapTable.getPartition(newPartName).getId());
@@ -3686,8 +3694,10 @@ public class FrontendServiceImpl implements FrontendService.Iface {
taskLock.unlock();
}
- // build partition & tablets. now all partitions in allReqPartNames are replaced an recorded.
- // so they won't be changed again. if other transaction changing it. just let it fail.
+ // build partition & tablets. now all partitions in allReqPartNames are replaced
+ // an recorded.
+ // so they won't be changed again. if other transaction changing it. just let it
+ // fail.
List<TOlapTablePartition> partitions = Lists.newArrayList();
List<TTabletLocation> tablets = Lists.newArrayList();
PartitionInfo partitionInfo = olapTable.getPartitionInfo();
@@ -3796,7 +3806,7 @@ public class FrontendServiceImpl implements FrontendService.Iface {
private TGetMetaResult getMetaImpl(TGetMetaRequest request, String clientIp)
throws Exception {
- // Step 1: check fields
+ // Step 1: check fields
if (!request.isSetUser()) {
throw new UserException("user is not set");
}
@@ -3869,7 +3879,6 @@ public class FrontendServiceImpl implements FrontendService.Iface {
}
}
-
@Override
public TGetColumnInfoResult getColumnInfo(TGetColumnInfoRequest request) {
TGetColumnInfoResult result = new TGetColumnInfoResult();
@@ -3990,12 +3999,12 @@ public class FrontendServiceImpl implements FrontendService.Iface {
// FE only has one master, this should not be a problem
if (!Env.getCurrentEnv().isMaster()) {
LOG.error("failed to handle load stats report: not master, backend:{}",
- clientAddr);
+ clientAddr);
return new TStatus(TStatusCode.NOT_MASTER);
}
LOG.info("receive load stats report request: {}, backend: {}, dbId: {}, txnId: {}, label: {}",
- request, clientAddr, request.getDbId(), request.getTxnId(), request.getLabel());
+ request, clientAddr, request.getDbId(), request.getTxnId(), request.getLabel());
try {
byte[] receivedProtobufBytes = request.getPayload();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java
index 20ea5836483..2c7aaae4f2a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java
@@ -238,16 +238,16 @@ public class DorisFlightSqlProducer implements FlightSqlProducer, AutoCloseable
connectContext.getResultFlightServerAddr().port);
List<FlightEndpoint> endpoints = Collections.singletonList(new FlightEndpoint(ticket, location));
// TODO Set in BE callback after query end, Client will not callback.
- connectContext.setCommand(MysqlCommand.COM_SLEEP);
return new FlightInfo(schema, descriptor, endpoints, -1, -1);
}
} catch (Exception e) {
- connectContext.setCommand(MysqlCommand.COM_SLEEP);
String errMsg = "get flight info statement failed, " + e.getMessage() + ", " + Util.getRootCauseMessage(e)
+ ", error code: " + connectContext.getState().getErrorCode() + ", error msg: "
+ connectContext.getState().getErrorMessage();
LOG.warn(errMsg, e);
throw CallStatus.INTERNAL.withDescription(errMsg).withCause(e).toRuntimeException();
+ } finally {
+ connectContext.setCommand(MysqlCommand.COM_SLEEP);
}
}
@@ -306,6 +306,7 @@ public class DorisFlightSqlProducer implements FlightSqlProducer, AutoCloseable
executorService.submit(() -> {
ConnectContext connectContext = flightSessionsManager.getConnectContext(context.peerIdentity());
try {
+ connectContext.setCommand(MysqlCommand.COM_QUERY);
final String query = request.getQuery();
String preparedStatementId = UUID.randomUUID().toString();
final ByteString handle = ByteString.copyFromUtf8(context.peerIdentity() + ":" + preparedStatementId);
@@ -323,7 +324,6 @@ public class DorisFlightSqlProducer implements FlightSqlProducer, AutoCloseable
Any.pack(buildCreatePreparedStatementResult(handle, parameterSchema, metaData))
.toByteArray()));
} catch (Exception e) {
- connectContext.setCommand(MysqlCommand.COM_SLEEP);
String errMsg = "create prepared statement failed, " + e.getMessage() + ", "
+ Util.getRootCauseMessage(e) + ", error code: " + connectContext.getState().getErrorCode()
+ ", error msg: " + connectContext.getState().getErrorMessage();
@@ -333,6 +333,8 @@ public class DorisFlightSqlProducer implements FlightSqlProducer, AutoCloseable
} catch (final Throwable t) {
listener.onError(CallStatus.INTERNAL.withDescription("Unknown error: " + t).toRuntimeException());
return;
+ } finally {
+ connectContext.setCommand(MysqlCommand.COM_SLEEP);
}
listener.onCompleted();
});
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlService.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlService.java
index fe5a60f0cc2..a8ad9a05c93 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlService.java
@@ -47,7 +47,11 @@ public class DorisFlightSqlService {
public DorisFlightSqlService(int port) {
BufferAllocator allocator = new RootAllocator();
Location location = Location.forGrpcInsecure(FrontendOptions.getLocalHostAddress(), port);
- this.flightTokenManager = new FlightTokenManagerImpl(Config.arrow_flight_token_cache_size,
+ // arrow_flight_token_cache_size less than qe_max_connection to avoid `Reach limit of connections`.
+ // arrow flight sql is a stateless protocol, connection is usually not actively disconnected.
+ // bearer token is evict from the cache will unregister ConnectContext.
+ this.flightTokenManager = new FlightTokenManagerImpl(
+ Math.min(Config.arrow_flight_token_cache_size, Config.qe_max_connection / 2),
Config.arrow_flight_token_alive_time);
this.flightSessionsManager = new FlightSessionsWithTokenManager(flightTokenManager);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/auth2/FlightBearerTokenAuthenticator.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/auth2/FlightBearerTokenAuthenticator.java
index ef6e28b034d..9f4479c6bcd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/auth2/FlightBearerTokenAuthenticator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/auth2/FlightBearerTokenAuthenticator.java
@@ -86,7 +86,7 @@ public class FlightBearerTokenAuthenticator implements CallHeaderAuthenticator {
return createAuthResultWithBearerToken(token);
} catch (IllegalArgumentException e) {
LOG.error("Bearer token validation failed.", e);
- throw CallStatus.UNAUTHENTICATED.toRuntimeException();
+ throw CallStatus.UNAUTHENTICATED.withCause(e).withDescription(e.getMessage()).toRuntimeException();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSessionsWithTokenManager.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSessionsWithTokenManager.java
index 26a48f0cfd2..b7e5ffa4646 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSessionsWithTokenManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/sessions/FlightSessionsWithTokenManager.java
@@ -47,42 +47,36 @@ public class FlightSessionsWithTokenManager implements FlightSessionsManager {
ConnectContext connectContext = ExecuteEnv.getInstance().getScheduler().getContext(peerIdentity);
if (null == connectContext) {
connectContext = createConnectContext(peerIdentity);
- if (null == connectContext) {
- flightTokenManager.invalidateToken(peerIdentity);
- String err = "UserSession expire after access, need reauthorize.";
- LOG.error(err);
- throw CallStatus.UNAUTHENTICATED.withDescription(err).toRuntimeException();
- }
return connectContext;
}
return connectContext;
} catch (Exception e) {
- LOG.warn("getConnectContext failed, " + e.getMessage(), e);
+ LOG.warn("get ConnectContext failed, " + e.getMessage(), e);
throw CallStatus.INTERNAL.withDescription(Util.getRootCauseMessage(e)).withCause(e).toRuntimeException();
}
}
@Override
public ConnectContext createConnectContext(String peerIdentity) {
- try {
- final FlightTokenDetails flightTokenDetails = flightTokenManager.validateToken(peerIdentity);
- if (flightTokenDetails.getCreatedSession()) {
- return null;
- }
- flightTokenDetails.setCreatedSession(true);
- ConnectContext connectContext = FlightSessionsManager.buildConnectContext(peerIdentity,
- flightTokenDetails.getUserIdentity(), flightTokenDetails.getRemoteIp());
- connectContext.setConnectionId(nextConnectionId.getAndAdd(1));
- connectContext.resetLoginTime();
- if (!ExecuteEnv.getInstance().getScheduler().registerConnection(connectContext)) {
- connectContext.getState()
- .setError(ErrorCode.ERR_TOO_MANY_USER_CONNECTIONS, "Reach limit of connections");
- throw CallStatus.UNAUTHENTICATED.withDescription("Reach limit of connections").toRuntimeException();
- }
- return connectContext;
- } catch (IllegalArgumentException e) {
- LOG.error("Bearer token validation failed.", e);
- throw CallStatus.UNAUTHENTICATED.toRuntimeException();
+ final FlightTokenDetails flightTokenDetails = flightTokenManager.validateToken(peerIdentity);
+ if (flightTokenDetails.getCreatedSession()) {
+ flightTokenManager.invalidateToken(peerIdentity);
+ throw new IllegalArgumentException("UserSession expire after access, try reconnect, bearer token: "
+ + peerIdentity + ", a peerIdentity(bearer token) can only create a ConnectContext once. "
+ + "if ConnectContext is deleted without operation for a long time, it needs to be reconnected "
+ + "(at the same time obtain a new bearer token).");
+ }
+ flightTokenDetails.setCreatedSession(true);
+ ConnectContext connectContext = FlightSessionsManager.buildConnectContext(peerIdentity,
+ flightTokenDetails.getUserIdentity(), flightTokenDetails.getRemoteIp());
+ connectContext.setConnectionId(nextConnectionId.getAndAdd(1));
+ connectContext.resetLoginTime();
+ if (!ExecuteEnv.getInstance().getScheduler().registerConnection(connectContext)) {
+ String err = "Reach limit of connections, increase `qe_max_connection` in fe.conf, or decrease "
+ + "`arrow_flight_token_cache_size` to evict unused bearer tokens and it connections faster";
+ connectContext.getState().setError(ErrorCode.ERR_TOO_MANY_USER_CONNECTIONS, err);
+ throw new IllegalArgumentException(err);
}
+ return connectContext;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/tokens/FlightTokenManagerImpl.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/tokens/FlightTokenManagerImpl.java
index 54e53e931dd..cd1b492de06 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/tokens/FlightTokenManagerImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/tokens/FlightTokenManagerImpl.java
@@ -19,12 +19,16 @@
package org.apache.doris.service.arrowflight.tokens;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.service.ExecuteEnv;
import org.apache.doris.service.arrowflight.auth2.FlightAuthResult;
import com.google.common.base.Preconditions;
import com.google.common.cache.CacheBuilder;
import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalListener;
+import com.google.common.cache.RemovalNotification;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -39,17 +43,32 @@ public class FlightTokenManagerImpl implements FlightTokenManager {
private static final Logger LOG = LogManager.getLogger(FlightTokenManagerImpl.class);
private final SecureRandom generator = new SecureRandom();
+ private final int cacheSize;
private final int cacheExpiration;
private LoadingCache<String, FlightTokenDetails> tokenCache;
public FlightTokenManagerImpl(final int cacheSize, final int cacheExpiration) {
+ this.cacheSize = cacheSize;
this.cacheExpiration = cacheExpiration;
- this.tokenCache = CacheBuilder.newBuilder()
- .maximumSize(cacheSize)
+ this.tokenCache = CacheBuilder.newBuilder().maximumSize(cacheSize)
.expireAfterWrite(cacheExpiration, TimeUnit.MINUTES)
- .build(new CacheLoader<String, FlightTokenDetails>() {
+ .removalListener(new RemovalListener<String, FlightTokenDetails>() {
+ @Override
+ public void onRemoval(RemovalNotification<String, FlightTokenDetails> notification) {
+ // TODO: broadcast this message to other FE
+ LOG.info("evict bearer token: " + notification.getKey() + ", reason: "
+ + notification.getCause());
+ ConnectContext context = ExecuteEnv.getInstance().getScheduler()
+ .getContext(notification.getKey());
+ if (context != null) {
+ ExecuteEnv.getInstance().getScheduler().unregisterConnection(context);
+ LOG.info("unregister flight connect context after evict bearer token: "
+ + notification.getKey());
+ }
+ }
+ }).build(new CacheLoader<String, FlightTokenDetails>() {
@Override
public FlightTokenDetails load(String key) {
return new FlightTokenDetails();
@@ -77,26 +96,32 @@ public class FlightTokenManagerImpl implements FlightTokenManager {
flightAuthResult.getUserIdentity(), flightAuthResult.getRemoteIp());
tokenCache.put(token, flightTokenDetails);
- LOG.trace("Created flight token for user: {}", username);
+ LOG.info("Created flight token for user: {}, token: {}", username, token);
return flightTokenDetails;
}
@Override
public FlightTokenDetails validateToken(final String token) throws IllegalArgumentException {
final FlightTokenDetails value = getTokenDetails(token);
+ if (value.getToken().equals("")) {
+ throw new IllegalArgumentException("invalid bearer token: " + token
+ + ", try reconnect, bearer token may not be created, or may have been evict, search for this "
+ + "token in fe.log to see the evict reason. currently in fe.conf, `arrow_flight_token_cache_size`="
+ + this.cacheSize + ", `arrow_flight_token_alive_time`=" + this.cacheExpiration);
+ }
if (System.currentTimeMillis() >= value.getExpiresAt()) {
- tokenCache.invalidate(token); // removes from the store as well
- throw new IllegalArgumentException("token expired");
+ tokenCache.invalidate(token);
+ throw new IllegalArgumentException("bearer token expired: " + token + ", try reconnect, "
+ + "currently in fe.conf, `arrow_flight_token_alive_time`=" + this.cacheExpiration);
}
-
- LOG.trace("Validated flight token for user: {}", value.getUsername());
+ LOG.info("Validated bearer token for user: {}", value.getUsername());
return value;
}
@Override
public void invalidateToken(final String token) {
- LOG.trace("Invalidate flight token, {}", token);
- tokenCache.invalidate(token); // removes from the store as well
+ LOG.info("Invalidate bearer token, {}", token);
+ tokenCache.invalidate(token);
}
private FlightTokenDetails getTokenDetails(final String token) {
@@ -105,7 +130,7 @@ public class FlightTokenManagerImpl implements FlightTokenManager {
try {
value = tokenCache.getUnchecked(token);
} catch (CacheLoader.InvalidCacheLoadException ignored) {
- throw new IllegalArgumentException("invalid token");
+ throw new IllegalArgumentException("InvalidCacheLoadException, invalid bearer token: " + token);
}
return value;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java
index c707107e0e0..c167db2228d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java
@@ -18,6 +18,7 @@
package org.apache.doris.statistics;
import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.gson.GsonUtils;
@@ -35,7 +36,6 @@ import java.io.DataOutput;
import java.io.IOException;
import java.text.ParseException;
import java.util.List;
-import java.util.Map;
import java.util.Set;
import java.util.StringJoiner;
@@ -95,8 +95,8 @@ public class AnalysisInfo implements Writable {
@SerializedName("tblId")
public final long tblId;
- // TODO: Map here is wired, List is enough
- public final Map<String, Set<String>> colToPartitions;
+ // Pair<IndexName, ColumnName>
+ public final List<Pair<String, String>> jobColumns;
public final Set<String> partitionNames;
@@ -200,7 +200,7 @@ public class AnalysisInfo implements Writable {
public final boolean userInject;
public AnalysisInfo(long jobId, long taskId, List<Long> taskIds, long catalogId, long dbId, long tblId,
- Map<String, Set<String>> colToPartitions, Set<String> partitionNames, String colName, Long indexId,
+ List<Pair<String, String>> jobColumns, Set<String> partitionNames, String colName, Long indexId,
JobType jobType, AnalysisMode analysisMode, AnalysisMethod analysisMethod, AnalysisType analysisType,
int samplePercent, long sampleRows, int maxBucketNum, long periodTimeInMs, String message,
long lastExecTimeInMs, long timeCostInMs, AnalysisState state, ScheduleType scheduleType,
@@ -213,7 +213,7 @@ public class AnalysisInfo implements Writable {
this.catalogId = catalogId;
this.dbId = dbId;
this.tblId = tblId;
- this.colToPartitions = colToPartitions;
+ this.jobColumns = jobColumns;
this.partitionNames = partitionNames;
this.colName = colName;
this.indexId = indexId;
@@ -268,8 +268,8 @@ public class AnalysisInfo implements Writable {
if (maxBucketNum > 0) {
sj.add("MaxBucketNum: " + maxBucketNum);
}
- if (colToPartitions != null) {
- sj.add("colToPartitions: " + getColToPartitionStr());
+ if (jobColumns != null) {
+ sj.add("jobColumns: " + getJobColumns());
}
if (lastExecTimeInMs > 0) {
sj.add("LastExecTime: " + StatisticsUtil.getReadableTime(lastExecTimeInMs));
@@ -301,12 +301,12 @@ public class AnalysisInfo implements Writable {
taskIds.add(taskId);
}
- public String getColToPartitionStr() {
- if (colToPartitions == null || colToPartitions.isEmpty()) {
+ public String getJobColumns() {
+ if (jobColumns == null || jobColumns.isEmpty()) {
return "";
}
Gson gson = new Gson();
- return gson.toJson(colToPartitions);
+ return gson.toJson(jobColumns);
}
@Override
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java
index 22f3d22b3ce..00cf9f7b1bc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java
@@ -17,6 +17,7 @@
package org.apache.doris.statistics;
+import org.apache.doris.common.Pair;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMode;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
@@ -26,7 +27,6 @@ import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
import org.apache.logging.log4j.core.util.CronExpression;
import java.util.List;
-import java.util.Map;
import java.util.Set;
public class AnalysisInfoBuilder {
@@ -36,7 +36,7 @@ public class AnalysisInfoBuilder {
private long catalogId;
private long dbId;
private long tblId;
- private Map<String, Set<String>> colToPartitions;
+ private List<Pair<String, String>> jobColumns;
private Set<String> partitionNames;
private String colName;
private long indexId = -1L;
@@ -75,7 +75,7 @@ public class AnalysisInfoBuilder {
catalogId = info.catalogId;
dbId = info.dbId;
tblId = info.tblId;
- colToPartitions = info.colToPartitions;
+ jobColumns = info.jobColumns;
partitionNames = info.partitionNames;
colName = info.colName;
indexId = info.indexId;
@@ -135,8 +135,8 @@ public class AnalysisInfoBuilder {
return this;
}
- public AnalysisInfoBuilder setColToPartitions(Map<String, Set<String>> colToPartitions) {
- this.colToPartitions = colToPartitions;
+ public AnalysisInfoBuilder setJobColumns(List<Pair<String, String>> jobColumns) {
+ this.jobColumns = jobColumns;
return this;
}
@@ -276,7 +276,7 @@ public class AnalysisInfoBuilder {
}
public AnalysisInfo build() {
- return new AnalysisInfo(jobId, taskId, taskIds, catalogId, dbId, tblId, colToPartitions, partitionNames,
+ return new AnalysisInfo(jobId, taskId, taskIds, catalogId, dbId, tblId, jobColumns, partitionNames,
colName, indexId, jobType, analysisMode, analysisMethod, analysisType, samplePercent,
sampleRows, maxBucketNum, periodTimeInMs, message, lastExecTimeInMs, timeCostInMs, state, scheduleType,
externalTableLevelTask, partitionOnly, samplingPartition, isAllPartition, partitionCount,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisJob.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisJob.java
index f52764bd6c9..5fd5e43be53 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisJob.java
@@ -180,13 +180,13 @@ public class AnalysisJob {
public void deregisterJob() {
analysisManager.removeJob(jobInfo.jobId);
for (BaseAnalysisTask task : queryingTask) {
- task.info.colToPartitions.clear();
+ task.info.jobColumns.clear();
if (task.info.partitionNames != null) {
task.info.partitionNames.clear();
}
}
for (BaseAnalysisTask task : queryFinished) {
- task.info.colToPartitions.clear();
+ task.info.jobColumns.clear();
if (task.info.partitionNames != null) {
task.info.partitionNames.clear();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
index 258c33305af..66d6d38f381 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
@@ -38,6 +38,7 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.FeMetaVersion;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.ThreadPoolManager;
import org.apache.doris.common.ThreadPoolManager.BlockedPolicy;
import org.apache.doris.common.io.Text;
@@ -82,7 +83,6 @@ import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
@@ -219,7 +219,7 @@ public class AnalysisManager implements Writable {
@VisibleForTesting
protected AnalysisInfo buildAndAssignJob(AnalyzeTblStmt stmt) throws DdlException {
AnalysisInfo jobInfo = buildAnalysisJobInfo(stmt);
- if (jobInfo.colToPartitions.isEmpty()) {
+ if (jobInfo.jobColumns.isEmpty()) {
// No statistics need to be collected or updated
return null;
}
@@ -292,51 +292,6 @@ public class AnalysisManager implements Writable {
}
}
- /**
- * Gets the partitions for which statistics are to be collected. First verify that
- * there are partitions that have been deleted but have historical statistics(invalid statistics),
- * if there are these partitions, we need to delete them to avoid errors in summary table level statistics.
- * Then get the partitions for which statistics need to be collected based on collection mode (incremental/full).
- * <p>
- * note:
- * If there is no invalid statistics, it does not need to collect/update
- * statistics if the following conditions are met:
- * - in full collection mode, the partitioned table does not have partitions
- * - in incremental collection mode, partition statistics already exist
- * <p>
- * TODO Supports incremental collection of statistics from materialized views
- */
- private Map<String, Set<String>> validateAndGetPartitions(TableIf table, Set<String> columnNames,
- Set<String> partitionNames, AnalysisType analysisType) throws DdlException {
-
- Map<String, Set<String>> columnToPartitions = columnNames.stream()
- .collect(Collectors.toMap(
- columnName -> columnName,
- columnName -> new HashSet<>(partitionNames == null ? Collections.emptySet() : partitionNames)
- ));
-
- if (analysisType == AnalysisType.HISTOGRAM) {
- // Collecting histograms does not need to support incremental collection,
- // and will automatically cover historical statistics
- return columnToPartitions;
- }
-
- if (table instanceof HMSExternalTable) {
- // TODO Currently, we do not support INCREMENTAL collection for external table.
- // One reason is external table partition id couldn't convert to a Long value.
- // Will solve this problem later.
- return columnToPartitions;
- }
-
- if (analysisType == AnalysisType.FUNDAMENTALS) {
- Map<String, Set<String>> result = table.findReAnalyzeNeededPartitions();
- result.keySet().retainAll(columnNames);
- return result;
- }
-
- return columnToPartitions;
- }
-
// Make sure colName of job has all the column as this AnalyzeStmt specified, no matter whether it will be analyzed
// or not.
@VisibleForTesting
@@ -362,12 +317,6 @@ public class AnalysisManager implements Writable {
infoBuilder.setCatalogId(stmt.getCatalogId());
infoBuilder.setDBId(stmt.getDbId());
infoBuilder.setTblId(stmt.getTable().getId());
- // TODO: Refactor later, DON'T MODIFY IT RIGHT NOW
- StringJoiner stringJoiner = new StringJoiner(",", "[", "]");
- for (String colName : columnNames) {
- stringJoiner.add(colName);
- }
- infoBuilder.setColName(stringJoiner.toString());
infoBuilder.setPartitionNames(partitionNames);
infoBuilder.setPartitionOnly(partitionOnly);
infoBuilder.setSamplingPartition(isSamplingPartition);
@@ -391,20 +340,23 @@ public class AnalysisManager implements Writable {
if (analysisType == AnalysisType.HISTOGRAM) {
int numBuckets = stmt.getNumBuckets();
- int maxBucketNum = numBuckets > 0 ? numBuckets
- : StatisticConstants.HISTOGRAM_MAX_BUCKET_NUM;
+ int maxBucketNum = numBuckets > 0 ? numBuckets : StatisticConstants.HISTOGRAM_MAX_BUCKET_NUM;
infoBuilder.setMaxBucketNum(maxBucketNum);
}
long periodTimeInMs = stmt.getPeriodTimeInMs();
infoBuilder.setPeriodTimeInMs(periodTimeInMs);
-
- Map<String, Set<String>> colToPartitions = validateAndGetPartitions(table, columnNames,
- partitionNames, analysisType);
- infoBuilder.setColToPartitions(colToPartitions);
+ List<Pair<String, String>> jobColumns = table.getColumnIndexPairs(columnNames);
+ infoBuilder.setJobColumns(jobColumns);
+ StringJoiner stringJoiner = new StringJoiner(",", "[", "]");
+ for (Pair<String, String> pair : jobColumns) {
+ stringJoiner.add(pair.toString());
+ }
+ infoBuilder.setColName(stringJoiner.toString());
infoBuilder.setTaskIds(Lists.newArrayList());
infoBuilder.setTblUpdateTime(table.getUpdateTime());
- infoBuilder.setEmptyJob(table instanceof OlapTable && table.getRowCount() == 0);
+ infoBuilder.setEmptyJob(table instanceof OlapTable && table.getRowCount() == 0
+ && analysisMethod.equals(AnalysisMethod.SAMPLE));
return infoBuilder.build();
}
@@ -420,35 +372,28 @@ public class AnalysisManager implements Writable {
public void createTaskForEachColumns(AnalysisInfo jobInfo, Map<Long, BaseAnalysisTask> analysisTasks,
boolean isSync) throws DdlException {
- Map<String, Set<String>> columnToPartitions = jobInfo.colToPartitions;
+ List<Pair<String, String>> jobColumns = jobInfo.jobColumns;
TableIf table = jobInfo.getTable();
- for (Entry<String, Set<String>> entry : columnToPartitions.entrySet()) {
- String colName = entry.getKey();
- List<Long> indexIds = Lists.newArrayList();
- // Get index id this column belongs to for OlapTable. Set it to -1 for baseIndex id.
- if (table instanceof OlapTable) {
- indexIds = ((OlapTable) table).getMvColumnIndexIds(colName);
- } else {
- indexIds.add(-1L);
- }
+ for (Pair<String, String> pair : jobColumns) {
AnalysisInfoBuilder colTaskInfoBuilder = new AnalysisInfoBuilder(jobInfo);
- if (jobInfo.analysisType != AnalysisType.HISTOGRAM) {
- colTaskInfoBuilder.setAnalysisType(AnalysisType.FUNDAMENTALS);
- Map<String, Set<String>> colToParts = new HashMap<>();
- colToParts.put(colName, entry.getValue());
- colTaskInfoBuilder.setColToPartitions(colToParts);
- }
- for (long indexId : indexIds) {
- long taskId = Env.getCurrentEnv().getNextId();
- AnalysisInfo analysisInfo = colTaskInfoBuilder.setColName(colName).setIndexId(indexId)
- .setTaskId(taskId).setLastExecTimeInMs(System.currentTimeMillis()).build();
- analysisTasks.put(taskId, createTask(analysisInfo));
- jobInfo.addTaskId(taskId);
- if (isSync) {
- continue;
+ colTaskInfoBuilder.setAnalysisType(AnalysisType.FUNDAMENTALS);
+ long taskId = Env.getCurrentEnv().getNextId();
+ long indexId = -1;
+ if (table instanceof OlapTable) {
+ OlapTable olapTable = (OlapTable) table;
+ indexId = olapTable.getIndexIdByName(pair.first);
+ if (indexId == olapTable.getBaseIndexId()) {
+ indexId = -1;
}
- replayCreateAnalysisTask(analysisInfo);
}
+ AnalysisInfo analysisInfo = colTaskInfoBuilder.setColName(pair.second).setIndexId(indexId)
+ .setTaskId(taskId).setLastExecTimeInMs(System.currentTimeMillis()).build();
+ analysisTasks.put(taskId, createTask(analysisInfo));
+ jobInfo.addTaskId(taskId);
+ if (isSync) {
+ continue;
+ }
+ replayCreateAnalysisTask(analysisInfo);
}
}
@@ -565,7 +510,9 @@ public class AnalysisManager implements Writable {
tableStats.update(jobInfo, tbl);
logCreateTableStats(tableStats);
}
- jobInfo.colToPartitions.clear();
+ if (jobInfo.jobColumns != null) {
+ jobInfo.jobColumns.clear();
+ }
if (jobInfo.partitionNames != null) {
jobInfo.partitionNames.clear();
}
@@ -712,7 +659,16 @@ public class AnalysisManager implements Writable {
indexIds.add(-1L);
}
for (long indexId : indexIds) {
- tableStats.removeColumn(column);
+ String indexName = table.getName();
+ if (table instanceof OlapTable) {
+ OlapTable olapTable = (OlapTable) table;
+ if (indexId == -1) {
+ indexName = olapTable.getIndexNameById(olapTable.getBaseIndexId());
+ } else {
+ indexName = olapTable.getIndexNameById(indexId);
+ }
+ }
+ tableStats.removeColumn(indexName, column);
statisticsCache.invalidate(tableId, indexId, column);
}
}
@@ -1088,25 +1044,16 @@ public class AnalysisManager implements Writable {
analysisJobIdToTaskMap.put(jobInfo.jobId, taskInfos);
}
- // Remove col stats status from TableStats if failed load some col stats after analyze corresponding column so that
- // we could make sure it would be analyzed again soon if user or system submit job for that column again.
- public void removeColStatsStatus(long tblId, String colName) {
- TableStatsMeta tableStats = findTableStatsStatus(tblId);
- if (tableStats != null) {
- tableStats.removeColumn(colName);
- }
- }
-
public void removeTableStats(long tableId) {
idToTblStats.remove(tableId);
}
- public ColStatsMeta findColStatsMeta(long tblId, String colName) {
+ public ColStatsMeta findColStatsMeta(long tblId, String indexName, String colName) {
TableStatsMeta tableStats = findTableStatsStatus(tblId);
if (tableStats == null) {
return null;
}
- return tableStats.findColumnStatsMeta(colName);
+ return tableStats.findColumnStatsMeta(indexName, colName);
}
public AnalysisJob findJob(long id) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
index 68767843507..f871e8761a5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
@@ -82,6 +82,30 @@ public abstract class BaseAnalysisTask {
+ "NOW() "
+ "FROM `${catalogName}`.`${dbName}`.`${tblName}` ${index} ${sampleHints} ${limit}";
+ protected static final String DUJ1_ANALYZE_STRING_TEMPLATE = "SELECT "
+ + "CONCAT('${tblId}', '-', '${idxId}', '-', '${colId}') AS `id`, "
+ + "${catalogId} AS `catalog_id`, "
+ + "${dbId} AS `db_id`, "
+ + "${tblId} AS `tbl_id`, "
+ + "${idxId} AS `idx_id`, "
+ + "'${colId}' AS `col_id`, "
+ + "NULL AS `part_id`, "
+ + "${rowCount} AS `row_count`, "
+ + "${ndvFunction} as `ndv`, "
+ + "IFNULL(SUM(IF(`t1`.`column_key` IS NULL, `t1`.`count`, 0)), 0) * ${scaleFactor} as `null_count`, "
+ + "SUBSTRING(CAST(${min} AS STRING), 1, 1024) AS `min`, "
+ + "SUBSTRING(CAST(${max} AS STRING), 1, 1024) AS `max`, "
+ + "${dataSizeFunction} * ${scaleFactor} AS `data_size`, "
+ + "NOW() "
+ + "FROM ( "
+ + " SELECT t0.`colValue` as `column_key`, COUNT(1) as `count` "
+ + " FROM "
+ + " (SELECT SUBSTRING(CAST(`${colName}` AS STRING), 1, 1024) AS `colValue` "
+ + " FROM `${catalogName}`.`${dbName}`.`${tblName}` ${index} "
+ + " ${sampleHints} ${limit}) as `t0` "
+ + " GROUP BY `t0`.`colValue` "
+ + ") as `t1` ";
+
protected static final String DUJ1_ANALYZE_TEMPLATE = "SELECT "
+ "CONCAT('${tblId}', '-', '${idxId}', '-', '${colId}') AS `id`, "
+ "${catalogId} AS `catalog_id`, "
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/ExternalAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/ExternalAnalysisTask.java
index ef1b795bd13..287941be526 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/ExternalAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/ExternalAnalysisTask.java
@@ -129,7 +129,11 @@ public class ExternalAnalysisTask extends BaseAnalysisTask {
params.put("ndvFunction", "ROUND(NDV(`${colName}`) * ${scaleFactor})");
params.put("rowCount", "ROUND(count(1) * ${scaleFactor})");
} else {
- sb.append(DUJ1_ANALYZE_TEMPLATE);
+ if (col.getType().isStringType()) {
+ sb.append(DUJ1_ANALYZE_STRING_TEMPLATE);
+ } else {
+ sb.append(DUJ1_ANALYZE_TEMPLATE);
+ }
params.put("dataSizeFunction", getDataSizeFunction(col, true));
params.put("ndvFunction", getNdvFunction("ROUND(SUM(t1.count) * ${scaleFactor})"));
params.put("rowCount", "ROUND(SUM(t1.count) * ${scaleFactor})");
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
index d26de9d9de7..60bfcab6157 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
@@ -65,13 +65,8 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
}
public void doExecute() throws Exception {
- Set<String> partitionNames = info.colToPartitions.get(info.colName);
- if (StatisticsUtil.isEmptyTable(tbl, info.analysisMethod)
- || partitionNames == null || partitionNames.isEmpty()) {
- if (partitionNames == null) {
- LOG.warn("Table {}.{}.{}, partitionNames for column {} is null. ColToPartitions:[{}]",
- info.catalogId, info.dbId, info.tblId, info.colName, info.colToPartitions);
- }
+ List<Pair<String, String>> columnList = info.jobColumns;
+ if (StatisticsUtil.isEmptyTable(tbl, info.analysisMethod) || columnList == null || columnList.isEmpty()) {
StatsId statsId = new StatsId(concatColumnStatsId(), info.catalogId, info.dbId,
info.tblId, info.indexId, info.colName, null);
job.appendBuf(this, Arrays.asList(new ColStatsData(statsId)));
@@ -166,7 +161,11 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
sql = stringSubstitutor.replace(LINEAR_ANALYZE_TEMPLATE);
} else {
params.put("dataSizeFunction", getDataSizeFunction(col, true));
- sql = stringSubstitutor.replace(DUJ1_ANALYZE_TEMPLATE);
+ if (col.getType().isStringType()) {
+ sql = stringSubstitutor.replace(DUJ1_ANALYZE_STRING_TEMPLATE);
+ } else {
+ sql = stringSubstitutor.replace(DUJ1_ANALYZE_TEMPLATE);
+ }
}
LOG.info("Sample for column [{}]. Total rows [{}], rows to sample [{}], scale factor [{}], "
+ "limited [{}], distribute column [{}], partition column [{}], key column [{}], "
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
index dbb7046467a..9ca971845b7 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
@@ -25,6 +25,7 @@ import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.hive.HMSExternalTable;
@@ -39,10 +40,9 @@ import org.apache.logging.log4j.Logger;
import java.time.LocalTime;
import java.util.ArrayList;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
import java.util.Set;
+import java.util.StringJoiner;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
@@ -121,7 +121,7 @@ public class StatisticsAutoCollector extends StatisticsCollector {
analysisTaskExecutor.clear();
break;
}
- analysisInfo = getReAnalyzeRequiredPart(analysisInfo);
+ analysisInfo = getNeedAnalyzeColumns(analysisInfo);
if (analysisInfo == null) {
continue;
}
@@ -186,11 +186,7 @@ public class StatisticsAutoCollector extends StatisticsCollector {
.setCatalogId(db.getCatalog().getId())
.setDBId(db.getId())
.setTblId(table.getId())
- .setColName(
- table.getSchemaAllIndexes(false).stream()
- .filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
- .map(Column::getName).collect(Collectors.joining(","))
- )
+ .setColName(null)
.setAnalysisType(AnalysisInfo.AnalysisType.FUNDAMENTALS)
.setAnalysisMode(AnalysisInfo.AnalysisMode.INCREMENTAL)
.setAnalysisMethod(analysisMethod)
@@ -202,13 +198,14 @@ public class StatisticsAutoCollector extends StatisticsCollector {
.setLastExecTimeInMs(System.currentTimeMillis())
.setJobType(JobType.SYSTEM)
.setTblUpdateTime(table.getUpdateTime())
- .setEmptyJob(table instanceof OlapTable && table.getRowCount() == 0)
+ .setEmptyJob(table instanceof OlapTable && table.getRowCount() == 0
+ && analysisMethod.equals(AnalysisMethod.SAMPLE))
.build();
analysisInfos.add(jobInfo);
}
@VisibleForTesting
- protected AnalysisInfo getReAnalyzeRequiredPart(AnalysisInfo jobInfo) {
+ protected AnalysisInfo getNeedAnalyzeColumns(AnalysisInfo jobInfo) {
TableIf table = StatisticsUtil.findTable(jobInfo.catalogId, jobInfo.dbId, jobInfo.tblId);
// Skip tables that are too wide.
if (table.getBaseSchema().size() > StatisticsUtil.getAutoAnalyzeTableWidthThreshold()) {
@@ -218,26 +215,25 @@ public class StatisticsAutoCollector extends StatisticsCollector {
AnalysisManager analysisManager = Env.getServingEnv().getAnalysisManager();
TableStatsMeta tblStats = analysisManager.findTableStatsStatus(table.getId());
- Map<String, Set<String>> needRunPartitions = null;
- String colNames = jobInfo.colName;
+ List<Pair<String, String>> needRunColumns = null;
if (table.needReAnalyzeTable(tblStats)) {
- needRunPartitions = table.findReAnalyzeNeededPartitions();
+ needRunColumns = table.getColumnIndexPairs(table.getSchemaAllIndexes(false)
+ .stream().map(Column::getName).collect(Collectors.toSet()));
} else if (table instanceof OlapTable && tblStats.newPartitionLoaded.get()) {
OlapTable olapTable = (OlapTable) table;
- needRunPartitions = new HashMap<>();
- Set<String> partitionColumnNames = olapTable.getPartitionInfo().getPartitionColumns().stream()
- .map(Column::getName).collect(Collectors.toSet());
- colNames = partitionColumnNames.stream().collect(Collectors.joining(","));
Set<String> partitionNames = olapTable.getAllPartitions().stream()
.map(Partition::getName).collect(Collectors.toSet());
- for (String column : partitionColumnNames) {
- needRunPartitions.put(column, partitionNames);
- }
+ needRunColumns = olapTable.getColumnIndexPairs(partitionNames);
}
- if (needRunPartitions == null || needRunPartitions.isEmpty()) {
+ if (needRunColumns == null || needRunColumns.isEmpty()) {
return null;
}
- return new AnalysisInfoBuilder(jobInfo).setColName(colNames).setColToPartitions(needRunPartitions).build();
+ StringJoiner stringJoiner = new StringJoiner(",", "[", "]");
+ for (Pair<String, String> pair : needRunColumns) {
+ stringJoiner.add(pair.toString());
+ }
+ return new AnalysisInfoBuilder(jobInfo)
+ .setColName(stringJoiner.toString()).setJobColumns(needRunColumns).build();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
index 0985b9b2b95..ec187fe893a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
@@ -61,7 +61,7 @@ public abstract class StatisticsCollector extends MasterDaemon {
@VisibleForTesting
protected void createSystemAnalysisJob(AnalysisInfo jobInfo)
throws DdlException {
- if (jobInfo.colToPartitions.isEmpty()) {
+ if (jobInfo.jobColumns.isEmpty()) {
// No statistics need to be collected or updated
return;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsRepository.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsRepository.java
index 5ac9b7305c7..5caa5bd9751 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsRepository.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsRepository.java
@@ -29,6 +29,7 @@ import org.apache.doris.common.FeConstants;
import org.apache.doris.statistics.util.DBObjects;
import org.apache.doris.statistics.util.StatisticsUtil;
+import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import org.apache.commons.text.StringSubstitutor;
import org.apache.logging.log4j.LogManager;
@@ -320,7 +321,7 @@ public class StatisticsRepository {
AnalysisInfo mockedJobInfo = new AnalysisInfoBuilder()
.setTblUpdateTime(System.currentTimeMillis())
.setColName("")
- .setColToPartitions(Maps.newHashMap())
+ .setJobColumns(Lists.newArrayList())
.setUserInject(true)
.setJobType(AnalysisInfo.JobType.MANUAL)
.build();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java
index 9231c6a2bc7..3b9b1e2bead 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java
@@ -21,11 +21,11 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PartitionInfo;
import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.statistics.AnalysisInfo.JobType;
-import org.apache.doris.statistics.util.StatisticsUtil;
import com.google.common.annotations.VisibleForTesting;
import com.google.gson.annotations.SerializedName;
@@ -33,8 +33,6 @@ import com.google.gson.annotations.SerializedName;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
-import java.util.Arrays;
-import java.util.List;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
@@ -64,7 +62,11 @@ public class TableStatsMeta implements Writable {
public long updatedTime;
@SerializedName("colNameToColStatsMeta")
- private ConcurrentMap<String, ColStatsMeta> colNameToColStatsMeta = new ConcurrentHashMap<>();
+ private ConcurrentMap<String, ColStatsMeta> deprecatedColNameToColStatsMeta = new ConcurrentHashMap<>();
+
+ @SerializedName("colToColStatsMeta")
+ // <IndexName, ColumnName> -> ColStatsMeta
+ private ConcurrentMap<Pair<String, String>, ColStatsMeta> colToColStatsMeta = new ConcurrentHashMap<>();
@SerializedName("trigger")
public JobType jobType;
@@ -100,52 +102,34 @@ public class TableStatsMeta implements Writable {
String json = Text.readString(dataInput);
TableStatsMeta tableStats = GsonUtils.GSON.fromJson(json, TableStatsMeta.class);
// Might be null counterintuitively, for compatible
- if (tableStats.colNameToColStatsMeta == null) {
- tableStats.colNameToColStatsMeta = new ConcurrentHashMap<>();
+ if (tableStats.colToColStatsMeta == null) {
+ tableStats.colToColStatsMeta = new ConcurrentHashMap<>();
}
- return tableStats;
- }
-
- public long findColumnLastUpdateTime(String colName) {
- ColStatsMeta colStatsMeta = colNameToColStatsMeta.get(colName);
- if (colStatsMeta == null) {
- return 0;
+ if (tableStats.deprecatedColNameToColStatsMeta != null) {
+ tableStats.convertDeprecatedColStatsToNewVersion();
}
- return colStatsMeta.updatedTime;
- }
-
- public ColStatsMeta findColumnStatsMeta(String colName) {
- return colNameToColStatsMeta.get(colName);
+ return tableStats;
}
- public void removeColumn(String colName) {
- colNameToColStatsMeta.remove(colName);
+ public ColStatsMeta findColumnStatsMeta(String indexName, String colName) {
+ return colToColStatsMeta.get(Pair.of(indexName, colName));
}
- public Set<String> analyzeColumns() {
- return colNameToColStatsMeta.keySet();
+ public void removeColumn(String indexName, String colName) {
+ colToColStatsMeta.remove(Pair.of(indexName, colName));
}
- public void reset() {
- updatedTime = 0;
- colNameToColStatsMeta.values().forEach(ColStatsMeta::clear);
+ public Set<Pair<String, String>> analyzeColumns() {
+ return colToColStatsMeta.keySet();
}
public void update(AnalysisInfo analyzedJob, TableIf tableIf) {
updatedTime = analyzedJob.tblUpdateTime;
userInjected = analyzedJob.userInject;
- String colNameStr = analyzedJob.colName;
- // colName field AnalyzeJob's format likes: "[col1, col2]", we need to remove brackets here
- // TODO: Refactor this later
- if (analyzedJob.colName.startsWith("[") && analyzedJob.colName.endsWith("]")) {
- colNameStr = colNameStr.substring(1, colNameStr.length() - 1);
- }
- List<String> cols = Arrays.stream(colNameStr.split(","))
- .map(String::trim).filter(s -> !s.isEmpty()).collect(Collectors.toList());
- for (String col : cols) {
- ColStatsMeta colStatsMeta = colNameToColStatsMeta.get(col);
+ for (Pair<String, String> colPair : analyzedJob.jobColumns) {
+ ColStatsMeta colStatsMeta = colToColStatsMeta.get(colPair);
if (colStatsMeta == null) {
- colNameToColStatsMeta.put(col, new ColStatsMeta(updatedTime,
+ colToColStatsMeta.put(colPair, new ColStatsMeta(updatedTime,
analyzedJob.analysisMethod, analyzedJob.analysisType, analyzedJob.jobType, 0));
} else {
colStatsMeta.updatedTime = updatedTime;
@@ -159,21 +143,27 @@ public class TableStatsMeta implements Writable {
if (tableIf instanceof OlapTable) {
rowCount = analyzedJob.emptyJob ? 0 : tableIf.getRowCount();
}
- if (!analyzedJob.emptyJob && analyzedJob.colToPartitions.keySet()
- .containsAll(tableIf.getBaseSchema().stream()
- .filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
- .map(Column::getName).collect(Collectors.toSet()))) {
+ if (analyzedJob.emptyJob) {
+ return;
+ }
+ if (analyzedJob.jobColumns.containsAll(
+ tableIf.getColumnIndexPairs(
+ tableIf.getSchemaAllIndexes(false).stream().map(Column::getName).collect(Collectors.toSet())))) {
updatedRows.set(0);
newPartitionLoaded.set(false);
}
if (tableIf instanceof OlapTable) {
PartitionInfo partitionInfo = ((OlapTable) tableIf).getPartitionInfo();
- if (partitionInfo != null && analyzedJob.colToPartitions.keySet()
- .containsAll(partitionInfo.getPartitionColumns().stream()
- .map(Column::getName).collect(Collectors.toSet()))) {
+ if (partitionInfo != null && analyzedJob.jobColumns
+ .containsAll(tableIf.getColumnIndexPairs(partitionInfo.getPartitionColumns().stream()
+ .map(Column::getName).collect(Collectors.toSet())))) {
newPartitionLoaded.set(false);
}
}
}
}
+
+ public void convertDeprecatedColStatsToNewVersion() {
+ deprecatedColNameToColStatsMeta = null;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
index c84939ffbe6..ba727063712 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
@@ -19,6 +19,7 @@ package org.apache.doris.tablefunction;
import org.apache.doris.analysis.UserIdentity;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.MTMV;
import org.apache.doris.catalog.SchemaTable;
@@ -38,6 +39,9 @@ import org.apache.doris.job.extensions.mtmv.MTMVJob;
import org.apache.doris.job.task.AbstractTask;
import org.apache.doris.mtmv.MTMVPartitionUtil;
import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.plsql.metastore.PlsqlManager;
+import org.apache.doris.plsql.metastore.PlsqlProcedureKey;
+import org.apache.doris.plsql.metastore.PlsqlStoredProcedure;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.QeProcessorImpl;
import org.apache.doris.qe.QeProcessorImpl.QueryInfo;
@@ -92,6 +96,8 @@ public class MetadataGenerator {
private static final ImmutableMap<String, Integer> WORKLOAD_GROUPS_COLUMN_TO_INDEX;
+ private static final ImmutableMap<String, Integer> ROUTINE_INFO_COLUMN_TO_INDEX;
+
static {
ImmutableMap.Builder<String, Integer> activeQueriesbuilder = new ImmutableMap.Builder();
List<Column> activeQueriesColList = SchemaTable.TABLE_MAP.get("active_queries").getFullSchema();
@@ -105,6 +111,12 @@ public class MetadataGenerator {
workloadGroupBuilder.put(WorkloadGroupMgr.WORKLOAD_GROUP_PROC_NODE_TITLE_NAMES.get(i).toLowerCase(), i);
}
WORKLOAD_GROUPS_COLUMN_TO_INDEX = workloadGroupBuilder.build();
+
+ ImmutableMap.Builder<String, Integer> routineInfoBuilder = new ImmutableMap.Builder();
+ for (int i = 0; i < PlsqlManager.ROUTINE_INFO_TITLE_NAMES.size(); i++) {
+ routineInfoBuilder.put(PlsqlManager.ROUTINE_INFO_TITLE_NAMES.get(i).toLowerCase(), i);
+ }
+ ROUTINE_INFO_COLUMN_TO_INDEX = routineInfoBuilder.build();
}
public static TFetchSchemaTableDataResult getMetadataTable(TFetchSchemaTableDataRequest request) throws TException {
@@ -167,6 +179,10 @@ public class MetadataGenerator {
result = workloadGroupsMetadataResult(schemaTableParams);
columnIndex = WORKLOAD_GROUPS_COLUMN_TO_INDEX;
break;
+ case ROUTINES_INFO:
+ result = routineInfoMetadataResult(schemaTableParams);
+ columnIndex = ROUTINE_INFO_COLUMN_TO_INDEX;
+ break;
default:
return errorResult("invalid schema table name.");
}
@@ -189,7 +205,7 @@ public class MetadataGenerator {
return errorResult("Iceberg metadata params is not set.");
}
- TIcebergMetadataParams icebergMetadataParams = params.getIcebergMetadataParams();
+ TIcebergMetadataParams icebergMetadataParams = params.getIcebergMetadataParams();
TIcebergQueryType icebergQueryType = icebergMetadataParams.getIcebergQueryType();
IcebergMetadataCache icebergMetadataCache = Env.getCurrentEnv().getExtMetaCacheMgr().getIcebergMetadataCache();
List<TRow> dataBatch = Lists.newArrayList();
@@ -376,7 +392,7 @@ public class MetadataGenerator {
private static TFetchSchemaTableDataResult catalogsMetadataResult(TMetadataTableRequestParams params) {
TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult();
- List<CatalogIf> info = Env.getCurrentEnv().getCatalogMgr().listCatalogs();
+ List<CatalogIf> info = Env.getCurrentEnv().getCatalogMgr().listCatalogs();
List<TRow> dataBatch = Lists.newArrayList();
for (CatalogIf catalog : info) {
@@ -417,24 +433,25 @@ public class MetadataGenerator {
List<TRow> dataBatch = Lists.newArrayList();
for (List<String> rGroupsInfo : workloadGroupsInfo) {
TRow trow = new TRow();
- trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(0)))); // id
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(1))); // name
+ trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(0)))); // id
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(1))); // name
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(2)))); // cpu_share
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(3))); // mem_limit
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(4))); // mem overcommit
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(3))); // mem_limit
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(4))); // mem overcommit
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(5)))); // max concurrent
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(6)))); // max queue size
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(7)))); // queue timeout
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(8))); // cpu hard limit
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(8))); // cpu hard limit
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(9)))); // scan thread num
// max remote scan thread num
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(10))));
// min remote scan thread num
trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(rGroupsInfo.get(11))));
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(12)));
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(13)));
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(14)));
- trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(15)));
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(12))); // spill low watermark
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(13))); // spill high watermark
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(14))); // tag
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(15))); // running query num
+ trow.addToColumnValue(new TCell().setStringVal(rGroupsInfo.get(16))); // waiting query num
dataBatch.add(trow);
}
@@ -455,11 +472,11 @@ public class MetadataGenerator {
List<TRow> dataBatch = Lists.newArrayList();
for (List<String> policyRow : workloadPolicyList) {
TRow trow = new TRow();
- trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(policyRow.get(0)))); // id
- trow.addToColumnValue(new TCell().setStringVal(policyRow.get(1))); // name
- trow.addToColumnValue(new TCell().setStringVal(policyRow.get(2))); // condition
- trow.addToColumnValue(new TCell().setStringVal(policyRow.get(3))); // action
- trow.addToColumnValue(new TCell().setIntVal(Integer.valueOf(policyRow.get(4)))); // priority
+ trow.addToColumnValue(new TCell().setLongVal(Long.valueOf(policyRow.get(0)))); // id
+ trow.addToColumnValue(new TCell().setStringVal(policyRow.get(1))); // name
+ trow.addToColumnValue(new TCell().setStringVal(policyRow.get(2))); // condition
+ trow.addToColumnValue(new TCell().setStringVal(policyRow.get(3))); // action
+ trow.addToColumnValue(new TCell().setIntVal(Integer.valueOf(policyRow.get(4)))); // priority
trow.addToColumnValue(new TCell().setBoolVal(Boolean.valueOf(policyRow.get(5)))); // enabled
trow.addToColumnValue(new TCell().setIntVal(Integer.valueOf(policyRow.get(6)))); // version
dataBatch.add(trow);
@@ -552,7 +569,7 @@ public class MetadataGenerator {
List<TFetchSchemaTableDataResult> relayResults = forwardToOtherFrontends(replayFetchSchemaTableReq);
relayResults
.forEach(rs -> rs.getDataBatch()
- .forEach(row -> dataBatch.add(row)));
+ .forEach(row -> dataBatch.add(row)));
}
result.setDataBatch(dataBatch);
@@ -753,4 +770,58 @@ public class MetadataGenerator {
result.setStatus(new TStatus(TStatusCode.OK));
return result;
}
+
+ private static TFetchSchemaTableDataResult routineInfoMetadataResult(TSchemaTableRequestParams params) {
+ if (!params.isSetCurrentUserIdent()) {
+ return errorResult("current user ident is not set.");
+ }
+
+ PlsqlManager plSqlClient = Env.getCurrentEnv().getPlsqlManager();
+
+ TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult();
+ List<TRow> dataBatch = Lists.newArrayList();
+
+ Map<PlsqlProcedureKey, PlsqlStoredProcedure> allProc = plSqlClient.getAllPlsqlStoredProcedures();
+ for (Map.Entry<PlsqlProcedureKey, PlsqlStoredProcedure> entry : allProc.entrySet()) {
+ PlsqlStoredProcedure proc = entry.getValue();
+ TRow trow = new TRow();
+ trow.addToColumnValue(new TCell().setStringVal(proc.getName())); // SPECIFIC_NAME
+ trow.addToColumnValue(new TCell().setStringVal(Long.toString(proc.getCatalogId()))); // ROUTINE_CATALOG
+ CatalogIf catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(proc.getCatalogId());
+ if (catalog != null) {
+ DatabaseIf db = catalog.getDbNullable(proc.getDbId());
+ if (db != null) {
+ trow.addToColumnValue(new TCell().setStringVal(db.getFullName())); // ROUTINE_SCHEMA
+ } else {
+ trow.addToColumnValue(new TCell().setStringVal("")); // ROUTINE_SCHEMA
+ }
+ } else {
+ trow.addToColumnValue(new TCell().setStringVal("")); // ROUTINE_SCHEMA
+ }
+ trow.addToColumnValue(new TCell().setStringVal(proc.getName())); // ROUTINE_NAME
+ trow.addToColumnValue(new TCell().setStringVal("PROCEDURE")); // ROUTINE_TYPE
+ trow.addToColumnValue(new TCell().setStringVal("")); // DTD_IDENTIFIER
+ trow.addToColumnValue(new TCell().setStringVal(proc.getSource())); // ROUTINE_BODY
+ trow.addToColumnValue(new TCell().setStringVal("")); // ROUTINE_DEFINITION
+ trow.addToColumnValue(new TCell().setStringVal("NULL")); // EXTERNAL_NAME
+ trow.addToColumnValue(new TCell().setStringVal("")); // EXTERNAL_LANGUAGE
+ trow.addToColumnValue(new TCell().setStringVal("SQL")); // PARAMETER_STYLE
+ trow.addToColumnValue(new TCell().setStringVal("")); // IS_DETERMINISTIC
+ trow.addToColumnValue(new TCell().setStringVal("")); // SQL_DATA_ACCESS
+ trow.addToColumnValue(new TCell().setStringVal("NULL")); // SQL_PATH
+ trow.addToColumnValue(new TCell().setStringVal("DEFINER")); // SECURITY_TYPE
+ trow.addToColumnValue(new TCell().setStringVal(proc.getCreateTime())); // CREATED
+ trow.addToColumnValue(new TCell().setStringVal(proc.getModifyTime())); // LAST_ALTERED
+ trow.addToColumnValue(new TCell().setStringVal("")); // SQ_MODE
+ trow.addToColumnValue(new TCell().setStringVal("")); // ROUTINE_COMMENT
+ trow.addToColumnValue(new TCell().setStringVal(proc.getOwnerName())); // DEFINER
+ trow.addToColumnValue(new TCell().setStringVal("")); // CHARACTER_SET_CLIENT
+ trow.addToColumnValue(new TCell().setStringVal("")); // COLLATION_CONNECTION
+ trow.addToColumnValue(new TCell().setStringVal("")); // DATABASE_COLLATION
+ dataBatch.add(trow);
+ }
+ result.setDataBatch(dataBatch);
+ result.setStatus(new TStatus(TStatusCode.OK));
+ return result;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java
index f70f7059555..2ceebe2b53c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java
@@ -23,6 +23,7 @@ import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Partition;
import org.apache.doris.common.Config;
import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.MasterDaemon;
import org.apache.doris.metric.MetricRepo;
@@ -45,7 +46,6 @@ import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@@ -225,8 +225,8 @@ public class PublishVersionDaemon extends MasterDaemon {
.flatMap(Collection::stream)
.flatMap(tablet ->
tablet.getBackendIds()
- .stream().map(backendId -> Map.entry(backendId, tablet.getId())))
- .collect(Collectors.groupingBy(Entry::getKey,
- Collectors.mapping(Entry::getValue, Collectors.toSet())));
+ .stream().map(backendId -> Pair.of(backendId, tablet.getId())))
+ .collect(Collectors.groupingBy(p -> p.first,
+ Collectors.mapping(p -> p.second, Collectors.toSet())));
}
}
diff --git a/fe/fe-core/src/main/jflex/sql_scanner.flex b/fe/fe-core/src/main/jflex/sql_scanner.flex
index cd496a9fff3..b931bda9e92 100644
--- a/fe/fe-core/src/main/jflex/sql_scanner.flex
+++ b/fe/fe-core/src/main/jflex/sql_scanner.flex
@@ -505,6 +505,7 @@ import org.apache.doris.qe.SqlModeHelper;
keywordMap.put("verbose", new Integer(SqlParserSymbols.KW_VERBOSE));
keywordMap.put("version", new Integer(SqlParserSymbols.KW_VERSION));
keywordMap.put("view", new Integer(SqlParserSymbols.KW_VIEW));
+ keywordMap.put("views", new Integer(SqlParserSymbols.KW_VIEWS));
keywordMap.put("warnings", new Integer(SqlParserSymbols.KW_WARNINGS));
keywordMap.put("week", new Integer(SqlParserSymbols.KW_WEEK));
keywordMap.put("when", new Integer(SqlParserSymbols.KW_WHEN));
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/GrantStmtTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/GrantStmtTest.java
index 382e9348b79..05451e44b25 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/GrantStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/GrantStmtTest.java
@@ -108,8 +108,8 @@ public class GrantStmtTest {
stmt = new GrantStmt(new UserIdentity("testUser", "%"), null,
new ResourcePattern("*", ResourceTypeEnum.GENERAL), privileges, ResourceTypeEnum.GENERAL);
stmt.analyze(analyzer);
- Assert.assertEquals(Auth.PrivLevel.GLOBAL, stmt.getResourcePattern().getPrivLevel());
- Assert.assertEquals("GRANT Usage_priv ON RESOURCE '*' TO 'testUser'@'%'", stmt.toSql());
+ Assert.assertEquals(Auth.PrivLevel.RESOURCE, stmt.getResourcePattern().getPrivLevel());
+ Assert.assertEquals("GRANT Usage_priv ON RESOURCE '%' TO 'testUser'@'%'", stmt.toSql());
}
@Test(expected = AnalysisException.class)
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowReplicaTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowReplicaTest.java
index be913cbad26..21fe967a96d 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowReplicaTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowReplicaTest.java
@@ -64,7 +64,7 @@ public class ShowReplicaTest extends TestWithFeService {
executor = new ShowExecutor(connectContext, skewStmt);
resultSet = executor.execute();
Assert.assertEquals(10, resultSet.getResultRows().size());
- Assert.assertEquals(5, resultSet.getResultRows().get(0).size());
+ Assert.assertEquals(6, resultSet.getResultRows().get(0).size());
// update tablets' data size and row count
Database db = Env.getCurrentInternalCatalog().getDbOrAnalysisException("test");
@@ -87,8 +87,8 @@ public class ShowReplicaTest extends TestWithFeService {
executor = new ShowExecutor(connectContext, skewStmt);
resultSet = executor.execute();
Assert.assertEquals(10, resultSet.getResultRows().size());
- Assert.assertEquals("4", resultSet.getResultRows().get(4).get(0));
- Assert.assertEquals(5, resultSet.getResultRows().get(0).size());
+ Assert.assertEquals("4", resultSet.getResultRows().get(4).get(1));
+ Assert.assertEquals(6, resultSet.getResultRows().get(0).size());
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java
index cd72b41bafe..5c0015fd03a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java
@@ -17,6 +17,7 @@
package org.apache.doris.analysis;
+import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.mysql.privilege.AccessControllerManager;
import org.apache.doris.mysql.privilege.MockedAuth;
@@ -68,6 +69,28 @@ public class ShowTableStmtTest {
Assert.assertEquals("Table_type", stmt.getMetaData().getColumn(1).getName());
}
+ @Test
+ public void testShowViews() throws AnalysisException {
+ ShowTableStmt stmt = new ShowTableStmt("", null, false, TableType.VIEW,
+ null, null);
+ stmt.analyze(analyzer);
+ Assert.assertEquals("SHOW VIEWS FROM internal.testDb", stmt.toString());
+ Assert.assertEquals("testDb", stmt.getDb());
+ Assert.assertEquals(TableType.VIEW, stmt.getType());
+ Assert.assertFalse(stmt.isVerbose());
+ Assert.assertEquals(1, stmt.getMetaData().getColumnCount());
+ Assert.assertEquals("Tables_in_testDb", stmt.getMetaData().getColumn(0).getName());
+
+ stmt = new ShowTableStmt("abc", null, true, TableType.VIEW, "bcd", null);
+ stmt.analyze(analyzer);
+ Assert.assertEquals("bcd", stmt.getPattern());
+ Assert.assertEquals("SHOW FULL VIEWS FROM internal.abc LIKE 'bcd'", stmt.toString());
+ Assert.assertEquals(4, stmt.getMetaData().getColumnCount());
+ Assert.assertEquals("Tables_in_abc", stmt.getMetaData().getColumn(0).getName());
+ Assert.assertEquals("Table_type", stmt.getMetaData().getColumn(1).getName());
+ Assert.assertEquals(TableType.VIEW, stmt.getType());
+ }
+
@Test
public void testNoDb() {
ShowTableStmt stmt = new ShowTableStmt("", null, false, null);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java b/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java
index 5ed4c23d470..ffd5e21ca96 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java
@@ -18,6 +18,7 @@
package org.apache.doris.mysql.privilege;
import org.apache.doris.analysis.Analyzer;
+import org.apache.doris.analysis.CompoundPredicate.Operator;
import org.apache.doris.analysis.CreateRoleStmt;
import org.apache.doris.analysis.CreateUserStmt;
import org.apache.doris.analysis.DropRoleStmt;
@@ -1677,15 +1678,17 @@ public class AuthTest {
}
@Test
- public void testResource() {
+ public void testResource() throws UserException {
UserIdentity userIdentity = new UserIdentity("testUser", "%");
String role = "role0";
String resourceName = "spark0";
ResourcePattern resourcePattern = new ResourcePattern(resourceName, ResourceTypeEnum.GENERAL);
- String anyResource = "*";
+ String anyResource = "%";
ResourcePattern anyResourcePattern = new ResourcePattern(anyResource, ResourceTypeEnum.GENERAL);
List<AccessPrivilegeWithCols> usagePrivileges = Lists
.newArrayList(new AccessPrivilegeWithCols(AccessPrivilege.USAGE_PRIV));
+ List<AccessPrivilegeWithCols> grantPrivileges = Lists
+ .newArrayList(new AccessPrivilegeWithCols(AccessPrivilege.GRANT_PRIV));
UserDesc userDesc = new UserDesc(userIdentity, "12345", true);
// ------ grant|revoke resource to|from user ------
@@ -1833,8 +1836,9 @@ public class AuthTest {
Assert.fail();
}
Assert.assertTrue(accessManager.checkResourcePriv(userIdentity, resourceName, PrivPredicate.USAGE));
- Assert.assertTrue(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.USAGE));
- Assert.assertTrue(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.SHOW_RESOURCES));
+ // anyResource not belong to global auth
+ Assert.assertFalse(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.USAGE));
+ Assert.assertFalse(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.SHOW_RESOURCES));
Assert.assertFalse(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.SHOW));
// 3. revoke usage_priv on resource '*' from 'testUser'@'%'
@@ -1891,7 +1895,7 @@ public class AuthTest {
Assert.fail();
}
Assert.assertTrue(accessManager.checkResourcePriv(userIdentity, resourceName, PrivPredicate.USAGE));
- Assert.assertTrue(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.USAGE));
+ Assert.assertFalse(accessManager.checkGlobalPriv(userIdentity, PrivPredicate.USAGE));
// 3. revoke usage_priv on resource '*' from role 'role0'
revokeStmt = new RevokeStmt(null, role, anyResourcePattern, usagePrivileges, ResourceTypeEnum.GENERAL);
@@ -1960,10 +1964,26 @@ public class AuthTest {
GrantStmt grantStmt3 = new GrantStmt(userIdentity, "test_role", tablePattern, usagePrivileges);
ExceptionChecker.expectThrowsWithMsg(AnalysisException.class,
"Can not grant/revoke USAGE_PRIV to/from database or table", () -> grantStmt3.analyze(analyzer));
+
+ // 4.drop user
+ dropUser(userIdentity);
+
+ // -------- test anyPattern has usage_priv and g1 has grant_priv ----------
+ // 1. create user with no role
+ createUser(userIdentity);
+ // 2. grant usage_priv on workload group '%' to user
+ grant(new GrantStmt(userIdentity, null, anyResourcePattern, usagePrivileges, ResourceTypeEnum.GENERAL));
+ // 3.grant grant_priv on workload group g1
+ grant(new GrantStmt(userIdentity, null, resourcePattern, grantPrivileges, ResourceTypeEnum.GENERAL));
+ Assert.assertTrue(accessManager.checkResourcePriv(userIdentity, resourceName,
+ PrivPredicate.of(PrivBitSet.of(Privilege.USAGE_PRIV, Privilege.GRANT_PRIV),
+ Operator.AND)));
+ // 4. drop user
+ dropUser(userIdentity);
}
@Test
- public void testWorkloadGroupPriv() {
+ public void testWorkloadGroupPriv() throws UserException {
UserIdentity userIdentity = new UserIdentity("testUser", "%");
String role = "role0";
String workloadGroupName = "g1";
@@ -1972,6 +1992,8 @@ public class AuthTest {
WorkloadGroupPattern anyWorkloadGroupPattern = new WorkloadGroupPattern(anyWorkloadGroup);
List<AccessPrivilegeWithCols> usagePrivileges = Lists
.newArrayList(new AccessPrivilegeWithCols(AccessPrivilege.USAGE_PRIV));
+ List<AccessPrivilegeWithCols> grantPrivileges = Lists
+ .newArrayList(new AccessPrivilegeWithCols(AccessPrivilege.GRANT_PRIV));
UserDesc userDesc = new UserDesc(userIdentity, "12345", true);
// ------ grant|revoke workload group to|from user ------
@@ -2244,6 +2266,27 @@ public class AuthTest {
GrantStmt grantStmt3 = new GrantStmt(userIdentity, "test_role", tablePattern, usagePrivileges);
ExceptionChecker.expectThrowsWithMsg(AnalysisException.class,
"Can not grant/revoke USAGE_PRIV to/from database or table", () -> grantStmt3.analyze(analyzer));
+ // 4.drop user
+ dropUser(userIdentity);
+
+ // -------- test anyPattern has usage_priv and g1 has grant_priv ----------
+ // 1. create user with no role
+ createUser(userIdentity);
+ // 2. grant usage_priv on workload group '%' to user
+ grant(new GrantStmt(userIdentity, null, anyWorkloadGroupPattern, usagePrivileges));
+ // 3.grant grant_priv on workload group g1
+ grant(new GrantStmt(userIdentity, null, workloadGroupPattern, grantPrivileges));
+ Assert.assertTrue(accessManager.checkWorkloadGroupPriv(userIdentity, workloadGroupName,
+ PrivPredicate.of(PrivBitSet.of(Privilege.USAGE_PRIV, Privilege.GRANT_PRIV),
+ Operator.AND)));
+ // 4. drop user
+ dropUser(userIdentity);
+ }
+
+ private void dropUser(UserIdentity userIdentity) throws UserException {
+ DropUserStmt dropUserStmt = new DropUserStmt(userIdentity);
+ dropUserStmt.analyze(analyzer);
+ auth.dropUser(dropUserStmt);
}
private void createUser(UserIdentity userIdentity) throws UserException {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/ColumnPruningPostProcessorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/ColumnPruningPostProcessorTest.java
new file mode 100644
index 00000000000..0e65aa6d582
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/ColumnPruningPostProcessorTest.java
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.postprocess;
+
+import org.apache.doris.nereids.processor.post.ColumnPruningPostProcessor;
+import org.apache.doris.nereids.rules.rewrite.InferFilterNotNull;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalNestedLoopJoin;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalProject;
+import org.apache.doris.nereids.util.LogicalPlanBuilder;
+import org.apache.doris.nereids.util.MemoPatternMatchSupported;
+import org.apache.doris.nereids.util.MemoTestUtils;
+import org.apache.doris.nereids.util.PlanChecker;
+import org.apache.doris.nereids.util.PlanConstructor;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class ColumnPruningPostProcessorTest implements MemoPatternMatchSupported {
+ private final LogicalOlapScan scan1 = PlanConstructor.newLogicalOlapScan(0, "t1", 0);
+ private final LogicalOlapScan scan2 = PlanConstructor.newLogicalOlapScan(1, "t2", 0);
+
+ @Test
+ void test() {
+ LogicalPlan plan = new LogicalPlanBuilder(scan1)
+ .join(scan2, JoinType.INNER_JOIN, ImmutableList.of())
+ .project(ImmutableList.of(0, 2))
+ .build();
+
+ PhysicalPlan physicalPlan = PlanChecker.from(MemoTestUtils.createConnectContext(), plan)
+ .applyTopDown(new InferFilterNotNull())
+ .implement()
+ .getPhysicalPlan();
+
+ ColumnPruningPostProcessor processor = new ColumnPruningPostProcessor();
+ PhysicalPlan newPlan = (PhysicalPlan) physicalPlan.accept(processor, null);
+
+ Assertions.assertTrue(newPlan instanceof PhysicalProject);
+ Assertions.assertTrue(newPlan.child(0) instanceof PhysicalNestedLoopJoin);
+ Assertions.assertTrue(newPlan.child(0).child(0) instanceof PhysicalProject);
+ Assertions.assertTrue(newPlan.child(0).child(1) instanceof PhysicalProject);
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/MaterializedViewUtilsTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/MaterializedViewUtilsTest.java
index 02fb18edbf7..8416bd396b0 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/MaterializedViewUtilsTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/MaterializedViewUtilsTest.java
@@ -269,7 +269,7 @@ public class MaterializedViewUtilsTest extends TestWithFeService {
Plan rewrittenPlan = nereidsPlanner.getRewrittenPlan();
Optional<RelatedTableInfo> relatedTableInfo =
MaterializedViewUtils.getRelatedTableInfo("l_shipdate", rewrittenPlan);
- Assertions.assertFalse(relatedTableInfo.isPresent());
+ Assertions.assertTrue(relatedTableInfo.isPresent());
});
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/PlanVisitorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/PlanVisitorTest.java
index 775bc05e163..23cfc1cc452 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/PlanVisitorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/PlanVisitorTest.java
@@ -37,7 +37,9 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.List;
+import java.util.Set;
import java.util.stream.Collectors;
/**
@@ -78,9 +80,31 @@ public class PlanVisitorTest extends TestWithFeService {
+ "\"replication_num\" = \"1\"\n"
+ ");");
+ createTable("CREATE TABLE `table3` (\n"
+ + " `c1` bigint(20) NULL,\n"
+ + " `c2` bigint(20) NULL,\n"
+ + " `c3` bigint(20) not NULL,\n"
+ + " `k4` bitmap BITMAP_UNION NULL,\n"
+ + " `k5` bitmap BITMAP_UNION NULL\n"
+ + ") ENGINE=OLAP\n"
+ + "AGGREGATE KEY(`c1`, `c2`, `c3`)\n"
+ + "COMMENT 'OLAP'\n"
+ + "DISTRIBUTED BY HASH(`c2`) BUCKETS 1\n"
+ + "PROPERTIES (\n"
+ + "\"replication_num\" = \"1\"\n"
+ + ");");
+
createView("CREATE VIEW `view1` AS SELECT t1.*, random() FROM\n"
+ "`table1` t1 LEFT JOIN\n"
+ "`table2` t2 ON t1.c1 = t2.c1;");
+
+ createMvByNereids("create materialized view mv1 BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL\n"
+ + "DISTRIBUTED BY RANDOM BUCKETS 1\n"
+ + "PROPERTIES ('replication_num' = '1') \n"
+ + "as "
+ + "select t1.c1, t3.c2 "
+ + "from table1 t1 "
+ + "inner join table3 t3 on t1.c1= t3.c2;");
}
@Test
@@ -97,18 +121,16 @@ public class PlanVisitorTest extends TestWithFeService {
Assertions.assertEquals(1, collectResult.size());
Assertions.assertTrue(collectResult.get(0) instanceof Random);
// Check get tables
- TableCollectorContext collectorContext =
- new TableCollector.TableCollectorContext(Sets.newHashSet(TableType.OLAP));
+ TableCollectorContext collectorContext = new TableCollector.TableCollectorContext(
+ Sets.newHashSet(TableType.OLAP), true);
physicalPlan.accept(TableCollector.INSTANCE, collectorContext);
- Assertions.assertEquals(3, collectorContext.getCollectedTables().size());
- List<String> expectedTables = new ArrayList<>();
+ Set<String> expectedTables = new HashSet<>();
expectedTables.add("table1");
expectedTables.add("table2");
- expectedTables.add("table2");
Assertions.assertEquals(
collectorContext.getCollectedTables().stream()
.map(TableIf::getName)
- .collect(Collectors.toList()),
+ .collect(Collectors.toSet()),
expectedTables);
});
}
@@ -128,21 +150,83 @@ public class PlanVisitorTest extends TestWithFeService {
Assertions.assertTrue(collectResult.get(0) instanceof Uuid);
Assertions.assertTrue(collectResult.get(1) instanceof Random);
// Check get tables
- TableCollectorContext collectorContext =
- new TableCollector.TableCollectorContext(Sets.newHashSet(TableType.OLAP));
+ TableCollectorContext collectorContext = new TableCollector.TableCollectorContext(
+ Sets.newHashSet(TableType.OLAP), true);
physicalPlan.accept(TableCollector.INSTANCE, collectorContext);
- Assertions.assertEquals(4, collectorContext.getCollectedTables().size());
- List<String> expectedTables = new ArrayList<>();
+ Set<String> expectedTables = new HashSet<>();
expectedTables.add("table1");
expectedTables.add("table2");
+ Assertions.assertEquals(
+ collectorContext.getCollectedTables().stream()
+ .map(TableIf::getName)
+ .collect(Collectors.toSet()),
+ expectedTables);
+ });
+ }
+
+ @Test
+ public void test3() throws Exception {
+ PlanChecker.from(connectContext)
+ .checkPlannerResult("SELECT mv1.*, uuid() FROM mv1 "
+ + "INNER JOIN view1 on mv1.c1 = view1.c2 "
+ + "LEFT SEMI JOIN table2 ON mv1.c1 = table2.c1 "
+ + "WHERE mv1.c1 IN (SELECT c1 FROM table2) OR mv1.c1 < 10",
+ nereidsPlanner -> {
+ PhysicalPlan physicalPlan = nereidsPlanner.getPhysicalPlan();
+ List<TreeNode<Expression>> collectResult = new ArrayList<>();
+ // Check nondeterministic collect
+ physicalPlan.accept(NondeterministicFunctionCollector.INSTANCE, collectResult);
+ // Check get tables
+ TableCollectorContext collectorContext = new TableCollector.TableCollectorContext(
+ Sets.newHashSet(TableType.OLAP), true);
+ physicalPlan.accept(TableCollector.INSTANCE, collectorContext);
+ Set<String> expectedTables = new HashSet<>();
+ expectedTables.add("table1");
expectedTables.add("table2");
- expectedTables.add("table2");
+ expectedTables.add("table3");
Assertions.assertEquals(
collectorContext.getCollectedTables().stream()
.map(TableIf::getName)
- .collect(Collectors.toList()),
+ .collect(Collectors.toSet()),
expectedTables);
+
+ TableCollectorContext collectorContextWithNoExpand =
+ new TableCollector.TableCollectorContext(Sets.newHashSet(TableType.OLAP),
+ false);
+ physicalPlan.accept(TableCollector.INSTANCE, collectorContextWithNoExpand);
+ Set<String> expectedTablesWithNoExpand = new HashSet<>();
+ expectedTablesWithNoExpand.add("table1");
+ expectedTablesWithNoExpand.add("table2");
+ Assertions.assertEquals(
+ collectorContextWithNoExpand.getCollectedTables().stream()
+ .map(TableIf::getName)
+ .collect(Collectors.toSet()),
+ expectedTablesWithNoExpand);
+
+ TableCollectorContext mvCollectorContext = new TableCollector.TableCollectorContext(
+ Sets.newHashSet(TableType.MATERIALIZED_VIEW), true);
+ physicalPlan.accept(TableCollector.INSTANCE, mvCollectorContext);
+ Set<String> expectedMvs = new HashSet<>();
+ expectedMvs.add("mv1");
+ Assertions.assertEquals(
+ mvCollectorContext.getCollectedTables().stream()
+ .map(TableIf::getName)
+ .collect(Collectors.toSet()),
+ expectedMvs);
+
+ TableCollectorContext mvCollectorContextWithNoExpand =
+ new TableCollector.TableCollectorContext(
+ Sets.newHashSet(TableType.MATERIALIZED_VIEW), false);
+ physicalPlan.accept(TableCollector.INSTANCE, mvCollectorContextWithNoExpand);
+ Set<String> expectedMvsWithNoExpand = new HashSet<>();
+ expectedMvsWithNoExpand.add("mv1");
+ Assertions.assertEquals(
+ mvCollectorContextWithNoExpand.getCollectedTables().stream()
+ .map(TableIf::getName)
+ .collect(Collectors.toSet()),
+ expectedMvsWithNoExpand);
});
+ dropMvByNereids("drop materialized view mv1");
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/ShowExecutorTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/ShowExecutorTest.java
index 114159142dc..5b8b374ad5a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/ShowExecutorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/ShowExecutorTest.java
@@ -324,6 +324,16 @@ public class ShowExecutorTest {
Assert.assertFalse(resultSet.next());
}
+ @Test
+ public void testShowViews() throws AnalysisException {
+ ShowTableStmt stmt = new ShowTableStmt("testDb", null, false, TableType.VIEW,
+ null, null);
+ ShowExecutor executor = new ShowExecutor(ctx, stmt);
+ ShowResultSet resultSet = executor.execute();
+
+ Assert.assertFalse(resultSet.next());
+ }
+
@Test
public void testShowTableFromCatalog() throws AnalysisException {
ShowTableStmt stmt = new ShowTableStmt("testDb", "internal", false, null);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java
index f8a77fe06db..674456b0b46 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java
@@ -27,6 +27,7 @@ import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.JobType;
import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
@@ -45,9 +46,9 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
// CHECKSTYLE OFF
public class AnalysisManagerTest {
@@ -109,7 +110,7 @@ public class AnalysisManagerTest {
// test build sync job
@Test
public void testBuildAndAssignJob1() throws Exception {
- AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setColToPartitions(new HashMap<>()).build();
+ AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setJobColumns(new ArrayList<>()).build();
new MockUp<StatisticsUtil>() {
@Mock
@@ -167,12 +168,7 @@ public class AnalysisManagerTest {
AnalysisManager analysisManager = new AnalysisManager();
Assertions.assertNull(analysisManager.buildAndAssignJob(analyzeTblStmt));
- analysisInfo.colToPartitions.put("c1", new HashSet<String>() {
- {
- add("p1");
- add("p2");
- }
- });
+ analysisInfo.jobColumns.add(Pair.of("index1", "c1"));
analysisManager.buildAndAssignJob(analyzeTblStmt);
new Expectations() {
{
@@ -191,7 +187,7 @@ public class AnalysisManagerTest {
// test build async job
@Test
public void testBuildAndAssignJob2(@Injectable OlapAnalysisTask analysisTask) throws Exception {
- AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setColToPartitions(new HashMap<>())
+ AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setJobColumns(new ArrayList<>())
.setScheduleType(ScheduleType.PERIOD)
.build();
new MockUp<StatisticsUtil>() {
@@ -255,12 +251,7 @@ public class AnalysisManagerTest {
}
}));
AnalysisManager analysisManager = new AnalysisManager();
- analysisInfo.colToPartitions.put("c1", new HashSet<String>() {
- {
- add("p1");
- add("p2");
- }
- });
+ analysisInfo.jobColumns.add(Pair.of("index1", "c1"));
analysisManager.buildAndAssignJob(analyzeTblStmt);
new Expectations() {
{
@@ -274,15 +265,7 @@ public class AnalysisManagerTest {
public void testReAnalyze() {
new MockUp<OlapTable>() {
- int count = 0;
- int[] rowCount = new int[]{100, 100, 200, 200, 1, 1};
-
final Column c = new Column("col1", PrimitiveType.INT);
- @Mock
- public long getRowCount() {
- return rowCount[count++];
- }
-
@Mock
public List<Column> getBaseSchema() {
return Lists.newArrayList(c);
@@ -291,22 +274,52 @@ public class AnalysisManagerTest {
@Mock
public List<Column> getColumns() { return Lists.newArrayList(c); }
+ @Mock
+ public List<Pair<String, String>> getColumnIndexPairs(Set<String> columns) {
+ List<Pair<String, String>> jobList = Lists.newArrayList();
+ jobList.add(Pair.of("1", "1"));
+ jobList.add(Pair.of("2", "2"));
+ jobList.add(Pair.of("3", "3"));
+ return jobList;
+ }
};
OlapTable olapTable = new OlapTable();
+ List<Pair<String, String>> jobList = Lists.newArrayList();
+ jobList.add(Pair.of("1", "1"));
+ jobList.add(Pair.of("2", "2"));
+ TableStatsMeta stats0 = new TableStatsMeta(
+ 0, new AnalysisInfoBuilder().setJobColumns(jobList)
+ .setColName("col1").build(), olapTable);
+ Assertions.assertTrue(olapTable.needReAnalyzeTable(stats0));
+
+ new MockUp<OlapTable>() {
+ int count = 0;
+ int[] rowCount = new int[]{100, 100, 200, 200, 1, 1};
+
+ @Mock
+ public long getRowCount() {
+ return rowCount[count++];
+ }
+ @Mock
+ public List<Pair<String, String>> getColumnIndexPairs(Set<String> columns) {
+ List<Pair<String, String>> jobList = Lists.newArrayList();
+ return jobList;
+ }
+ };
TableStatsMeta stats1 = new TableStatsMeta(
- 50, new AnalysisInfoBuilder().setColToPartitions(new HashMap<>())
+ 50, new AnalysisInfoBuilder().setJobColumns(new ArrayList<>())
.setColName("col1").build(), olapTable);
stats1.updatedRows.addAndGet(50);
Assertions.assertTrue(olapTable.needReAnalyzeTable(stats1));
TableStatsMeta stats2 = new TableStatsMeta(
190, new AnalysisInfoBuilder()
- .setColToPartitions(new HashMap<>()).setColName("col1").build(), olapTable);
+ .setJobColumns(new ArrayList<>()).setColName("col1").build(), olapTable);
stats2.updatedRows.addAndGet(20);
Assertions.assertFalse(olapTable.needReAnalyzeTable(stats2));
TableStatsMeta stats3 = new TableStatsMeta(0, new AnalysisInfoBuilder()
- .setColToPartitions(new HashMap<>()).setEmptyJob(true).setColName("col1").build(), olapTable);
+ .setJobColumns(new ArrayList<>()).setEmptyJob(true).setColName("col1").build(), olapTable);
Assertions.assertTrue(olapTable.needReAnalyzeTable(stats3));
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
index b17ba3e68db..5698f0e9b20 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
@@ -22,6 +22,7 @@ import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.InternalSchemaInitializer;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.Pair;
import org.apache.doris.common.jmockit.Deencapsulation;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.qe.StmtExecutor;
@@ -33,7 +34,7 @@ import org.apache.doris.statistics.util.DBObjects;
import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.utframe.TestWithFeService;
-import com.google.common.collect.Maps;
+import com.google.common.collect.Lists;
import mockit.Mock;
import mockit.MockUp;
import mockit.Mocked;
@@ -41,10 +42,8 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.atomic.AtomicBoolean;
@@ -158,8 +157,8 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
};
AnalysisTaskExecutor analysisTaskExecutor = new AnalysisTaskExecutor(1);
- HashMap<String, Set<String>> colToPartitions = Maps.newHashMap();
- colToPartitions.put("col1", Collections.singleton("t1"));
+ List<Pair<String, String>> columns = Lists.newArrayList();
+ columns.add(Pair.of("col1", "t1"));
AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setJobId(0).setTaskId(0)
.setCatalogId(0).setDBId(0).setTblId(0)
.setColName("col1").setJobType(JobType.MANUAL)
@@ -167,7 +166,7 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
.setAnalysisMethod(AnalysisMethod.FULL)
.setAnalysisType(AnalysisType.FUNDAMENTALS)
.setState(AnalysisState.RUNNING)
- .setColToPartitions(colToPartitions)
+ .setJobColumns(columns)
.build();
OlapAnalysisTask task = new OlapAnalysisTask(analysisInfo);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalyzeTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalyzeTest.java
index 483cd3c0326..bf6ce32e155 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalyzeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalyzeTest.java
@@ -23,6 +23,7 @@ import org.apache.doris.catalog.InternalSchemaInitializer;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.Pair;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.qe.AutoCloseConnectContext;
import org.apache.doris.qe.ConnectContext;
@@ -35,7 +36,7 @@ import org.apache.doris.statistics.util.DBObjects;
import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.utframe.TestWithFeService;
-import com.google.common.collect.Maps;
+import com.google.common.collect.Lists;
import mockit.Expectations;
import mockit.Mock;
import mockit.MockUp;
@@ -45,10 +46,8 @@ import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import java.util.Set;
public class AnalyzeTest extends TestWithFeService {
@@ -160,8 +159,8 @@ public class AnalyzeTest extends TestWithFeService {
@Mock
protected void runQuery(String sql) {}
};
- HashMap<String, Set<String>> colToPartitions = Maps.newHashMap();
- colToPartitions.put("col1", Collections.singleton("t1"));
+ List<Pair<String, String>> colList = Lists.newArrayList();
+ colList.add(Pair.of("col1", "index1"));
AnalysisInfo analysisJobInfo = new AnalysisInfoBuilder().setJobId(0).setTaskId(0)
.setCatalogId(0)
.setDBId(0)
@@ -170,7 +169,7 @@ public class AnalyzeTest extends TestWithFeService {
.setAnalysisMode(AnalysisMode.FULL)
.setAnalysisMethod(AnalysisMethod.FULL)
.setAnalysisType(AnalysisType.FUNDAMENTALS)
- .setColToPartitions(colToPartitions)
+ .setJobColumns(colList)
.setState(AnalysisState.RUNNING)
.build();
new OlapAnalysisTask(analysisJobInfo).doExecute();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java
index 4217fb5a0db..09bf4dd94c6 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java
@@ -90,7 +90,6 @@ public class HistogramTaskTest extends TestWithFeService {
for (Entry<Long, BaseAnalysisTask> infoEntry : taskInfo.entrySet()) {
BaseAnalysisTask task = infoEntry.getValue();
- Assertions.assertEquals(AnalysisType.HISTOGRAM, task.info.analysisType);
Assertions.assertEquals("col1", task.info.colName);
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java
index 5d9d57406a3..75506b1c85a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/OlapAnalysisTaskTest.java
@@ -159,11 +159,10 @@ public class OlapAnalysisTaskTest {
+ " IS NULL, `t1`.`count`, 0)), 0) * 5.0 as `null_count`, "
+ "SUBSTRING(CAST('1' AS STRING), 1, 1024) AS `min`,"
+ " SUBSTRING(CAST('2' AS STRING), 1, 1024) AS `max`, "
- + "SUM(LENGTH(`column_key`) * count) * 5.0 AS `data_size`, NOW() "
+ + "SUM(t1.count) * 4 * 5.0 AS `data_size`, NOW() "
+ "FROM ( SELECT t0.`${colName}` as `column_key`, COUNT(1) "
- + "as `count` FROM (SELECT `${colName}` FROM "
- + "`catalogName`.`${dbName}`.`${tblName}` "
- + " limit 100) as `t0` GROUP BY `t0`.`${colName}` ) as `t1` ", sql);
+ + "as `count` FROM (SELECT `${colName}` FROM `catalogName`.`${dbName}`.`${tblName}`"
+ + " limit 100) as `t0` GROUP BY `t0`.`${colName}` ) as `t1` ", sql);
return;
}
};
@@ -183,7 +182,7 @@ public class OlapAnalysisTaskTest {
};
OlapAnalysisTask olapAnalysisTask = new OlapAnalysisTask();
- olapAnalysisTask.col = new Column("test", PrimitiveType.STRING);
+ olapAnalysisTask.col = new Column("test", PrimitiveType.INT);
olapAnalysisTask.tbl = tableIf;
AnalysisInfoBuilder analysisInfoBuilder = new AnalysisInfoBuilder();
analysisInfoBuilder.setJobType(AnalysisInfo.JobType.MANUAL);
@@ -322,7 +321,8 @@ public class OlapAnalysisTaskTest {
+ "SUBSTRING(CAST('1' AS STRING), 1, 1024) AS `min`, "
+ "SUBSTRING(CAST('2' AS STRING), 1, 1024) AS `max`, "
+ "SUM(LENGTH(`column_key`) * count) * 5.0 AS `data_size`, NOW() "
- + "FROM ( SELECT t0.`${colName}` as `column_key`, COUNT(1) as `count` FROM (SELECT `${colName}` FROM `catalogName`.`${dbName}`.`${tblName}` limit 100) as `t0` GROUP BY `t0`.`${colName}` ) as `t1` ", sql);
+ + "FROM ( SELECT t0.`colValue` as `column_key`, COUNT(1) as `count` FROM "
+ + "(SELECT SUBSTRING(CAST(`${colName}` AS STRING), 1, 1024) AS `colValue` FROM `catalogName`.`${dbName}`.`${tblName}` limit 100) as `t0` GROUP BY `t0`.`colValue` ) as `t1` ", sql);
return;
}
};
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
index 678e7580f8e..f7b75261cc5 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
@@ -31,16 +31,12 @@ import org.apache.doris.catalog.View;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.Pair;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.InternalCatalog;
-import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
-import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
-import org.apache.doris.statistics.AnalysisInfo.JobType;
import org.apache.doris.statistics.util.StatisticsUtil;
import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
import mockit.Expectations;
import mockit.Injectable;
import mockit.Mock;
@@ -54,7 +50,6 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -144,97 +139,32 @@ public class StatisticsAutoCollectorTest {
StatisticsAutoCollector saa = new StatisticsAutoCollector();
List<AnalysisInfo> analysisInfoList = saa.constructAnalysisInfo(new Database(1, "anydb"));
Assertions.assertEquals(1, analysisInfoList.size());
- Assertions.assertEquals("c1", analysisInfoList.get(0).colName.split(",")[0]);
+ Assertions.assertNull(analysisInfoList.get(0).colName);
}
@Test
- public void testGetReAnalyzeRequiredPart0() {
+ public void testSkipWideTable() {
TableIf tableIf = new OlapTable();
new MockUp<OlapTable>() {
- @Mock
- protected Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- Set<String> partitionNames = new HashSet<>();
- partitionNames.add("p1");
- partitionNames.add("p2");
- Map<String, Set<String>> map = new HashMap<>();
- map.put("col1", partitionNames);
- return map;
- }
-
- @Mock
- public long getRowCount() {
- return 100;
- }
-
@Mock
public List<Column> getBaseSchema() {
return Lists.newArrayList(new Column("col1", Type.INT), new Column("col2", Type.INT));
}
- };
-
- new MockUp<StatisticsUtil>() {
- @Mock
- public TableIf findTable(long catalogName, long dbName, long tblName) {
- return tableIf;
- }
- };
- AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setAnalysisMethod(AnalysisMethod.FULL)
- .setColToPartitions(new HashMap<>()).setAnalysisType(
- AnalysisType.FUNDAMENTALS).setColName("col1").setJobType(JobType.SYSTEM).build();
- new MockUp<AnalysisManager>() {
-
- int count = 0;
-
- TableStatsMeta[] tableStatsArr =
- new TableStatsMeta[] {new TableStatsMeta(0, analysisInfo, tableIf),
- new TableStatsMeta(0, analysisInfo, tableIf), null};
-
- {
- tableStatsArr[0].updatedRows.addAndGet(100);
- tableStatsArr[1].updatedRows.addAndGet(0);
- }
-
- @Mock
- public TableStatsMeta findTableStatsStatus(long tblId) {
- return tableStatsArr[count++];
- }
- };
-
- new MockUp<StatisticsAutoCollector>() {
- @Mock
- public AnalysisInfo getAnalysisJobInfo(AnalysisInfo jobInfo, TableIf table,
- Set<String> needRunPartitions) {
- return new AnalysisInfoBuilder().build();
- }
- };
- StatisticsAutoCollector statisticsAutoCollector = new StatisticsAutoCollector();
- AnalysisInfo analysisInfo2 = new AnalysisInfoBuilder()
- .setCatalogId(0)
- .setDBId(0)
- .setTblId(0).build();
- Assertions.assertNotNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo2));
- // uncomment it when updatedRows gets ready
- // Assertions.assertNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo2));
- Assertions.assertNotNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo2));
- }
-
- @Test
- public void testSkipWideTable() {
-
- TableIf tableIf = new OlapTable();
- new MockUp<OlapTable>() {
@Mock
- public List<Column> getBaseSchema() {
- return Lists.newArrayList(new Column("col1", Type.INT), new Column("col2", Type.INT));
+ public List<Pair<String, String>> getColumnIndexPairs(Set<String> columns) {
+ ArrayList<Pair<String, String>> list = Lists.newArrayList();
+ list.add(Pair.of("1", "1"));
+ return list;
}
};
new MockUp<StatisticsUtil>() {
int count = 0;
- int [] thresholds = {1, 10};
+ int[] thresholds = {1, 10};
+
@Mock
public TableIf findTable(long catalogName, long dbName, long tblName) {
return tableIf;
@@ -246,19 +176,10 @@ public class StatisticsAutoCollectorTest {
}
};
- new MockUp<OlapTable>() {
- @Mock
- public Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- HashMap<String, Set<String>> ret = Maps.newHashMap();
- ret.put("key1", Sets.newHashSet());
- return ret;
- }
- };
-
AnalysisInfo analysisInfo = new AnalysisInfoBuilder().build();
StatisticsAutoCollector statisticsAutoCollector = new StatisticsAutoCollector();
- Assertions.assertNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo));
- Assertions.assertNotNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo));
+ Assertions.assertNull(statisticsAutoCollector.getNeedAnalyzeColumns(analysisInfo));
+ Assertions.assertNotNull(statisticsAutoCollector.getNeedAnalyzeColumns(analysisInfo));
}
@Test
@@ -400,13 +321,9 @@ public class StatisticsAutoCollectorTest {
List<AnalysisInfo> jobInfos = new ArrayList<>();
sac.createAnalyzeJobForTbl(db, jobInfos, t1);
AnalysisInfo jobInfo = jobInfos.get(0);
- Map<String, Set<String>> colToPartitions = new HashMap<>();
- colToPartitions.put("test", new HashSet<String>() {
- {
- add("p1");
- }
- });
- jobInfo = new AnalysisInfoBuilder(jobInfo).setColToPartitions(colToPartitions).build();
+ List<Pair<String, String>> columnNames = Lists.newArrayList();
+ columnNames.add(Pair.of("test", "t1"));
+ jobInfo = new AnalysisInfoBuilder(jobInfo).setJobColumns(columnNames).build();
Map<Long, BaseAnalysisTask> analysisTasks = new HashMap<>();
AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
analysisManager.createTaskForEachColumns(jobInfo, analysisTasks, false);
@@ -472,13 +389,9 @@ public class StatisticsAutoCollectorTest {
List<AnalysisInfo> jobInfos = new ArrayList<>();
sac.createAnalyzeJobForTbl(db, jobInfos, t1);
AnalysisInfo jobInfo = jobInfos.get(0);
- Map<String, Set<String>> colToPartitions = new HashMap<>();
- colToPartitions.put("test", new HashSet<String>() {
- {
- add("p1");
- }
- });
- jobInfo = new AnalysisInfoBuilder(jobInfo).setColToPartitions(colToPartitions).build();
+ List<Pair<String, String>> colNames = Lists.newArrayList();
+ colNames.add(Pair.of("test", "1"));
+ jobInfo = new AnalysisInfoBuilder(jobInfo).setJobColumns(colNames).build();
Map<Long, BaseAnalysisTask> analysisTasks = new HashMap<>();
AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
analysisManager.createTaskForEachColumns(jobInfo, analysisTasks, false);
... 12262 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org