You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by li...@apache.org on 2024/02/29 02:27:37 UTC
(doris) branch high-priority-column updated (88ed1980d86 -> 3ac73f32582)
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a change to branch high-priority-column
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 88ed1980d86 High priority queue and map. (#31509)
omit ab5b7c19c58 Collect high priority columns. (#31235)
add c8f6a04c7ba [fix](case) fix testcase test_decimal256_index #31486
add 9216262c2dd Reset report_workload_runtime_status to optional (#31479)
add 7c0c06eb4ec [Chore](function) remove unused check on count function (#31400)
add a2420dc1c96 [fix] Fix the incorrect class name in the getLogger method call of MysqlTable. (#31465)
add 5a15f0f22e9 [chore](log) reduce a lot inject debug point log #31474
add 00fbc72f036 [fix](docs)Correcting spelling errors,add isolated file to siderbars. (#31409)
add b5b9d54ed68 [Feat](nereids) support multi-leading (#30379)
add a7b7ff5ebc7 Fix replay binlog gc when not found db binlog (#31463)
add 0bf47824d67 [fix](HDFS) Create hdfs fs with it's resource id to prevent data loss for cold heat separation #31493
add 4915e775447 [improvement](iceberg/paimon)add show table stats (#31473)
add a4dad47d3c9 [docs](docs) Update 2.0.5 Release Note of EN & CN Version (#31506)
add 54e6b837975 [enhance](S3) Init default retry strategy for aws s3 sdk (#31329)
add 20fca781c65 [fix](cloud) Fix missing rowset kv introduced by CloudTabletsChannel (#31481)
add 0e7b16394fd [fix](paimon) fix hadoop.username does not take effect in paimon catalog (#31478)
add 237adf883fa [cases](regression) Add data reliability check meta cases (#31432)
add 94ea17aa87d [fix](fe ut) fix TabletRepairAndBalanceTest (#31397)
add 90d9122b5c8 [fix](Nereids) push down topn distinct through join by mistake (#31396)
add 611c1f9f903 [Improvement](regression-test) generate user token by configured user and password rather than hard cord (#31345)
add 4f5d405fc37 [fix](storage medium) Fix show partition storage medium not right whe… (#30160)
add a48ce1cb1f3 [Improve](Variant) support prune segment for quering variant (#31310)
add 5132b71e586 [Chore](agg-state) adjust AggStateType constructor check input (#31401)
add 685a8e3c40d [Fix](fe-core) Fix The EliminateSortUnderSubquery will not affect the EliminateOrderByConstant rule (#31402) (#31403)
add 420a91d0e1c [fix](merge-on-write) incorrect result caused by key range filter with pk (#31456)
add 233b2ba2aae Return UNKNOWN column stats if ndv is 0. (#31439)
add fa7b476b6ba [Opt](exec) enable top opt in string type (#31489)
add e9237ca1444 [Fix](regression) add disable_auto_compaction for show data (#31519)
add dae058516c1 [enhance](git)Ignore `fe/gen` (#31512)
add 90ab97b4817 [fix](test) fix error messages in fault injection tests (#31515)
add 8dddd92146a [fix](Nereids): other cond should be kept for each anti join when expanding anti join such as (#31521)
add 45ea6c581da Fix analyze mv from follower case bug. (#31523)
add a459e6f35a9 [fix](Nereids) Make the case sensitivity of the result labels compatible with MySQL (#31510)
add ae83a838d06 [nereids] config global partition topn (#31476)
add 76a0edb2bff [cloud](Variant) support variant in cloud mode (#31483)
add 5129557cd57 [chore](ci) fix case and script about cloud_p0 (#31527)
add 26a32099519 [Fix](cloud) Fix auto partition cannot load data(#31491)
add 7accce0c7ac [fix](regression) fix ssb shape case rf type (#31529)
add 7df63e9bf44 Fix hive p2 cases. (#31541)
add c68451a6db4 [fix](multi-catalog) add config to disable external DDL (#31453)
add aa366f6b195 [fix](load) Fix using uint32 for tablet id overflow (#31532)
add 8dfb249ad7e [fix](meta-tool) Fix compile meta tool (#31457)
add 9e6f5bf6a1e [fix](group_commit) GroupCommitBlockSink shoud not use load_block_queue when creating load task fail (#31416)
add ccaac10de8d [Opt](Exec) Support runtime update topn filter (#31250)
add 5620c678fd9 [enhancement](docs) Clarify JAR file locations for JDBC catalog (#31542)
add 8b8e8e88de7 [refactor](jdbc catalog) split jdbc executor for different data sources (step-1) (#31406)
add 35e1d36dc05 [type](docs)add example for sql_dialect #31536
add f71f0a127b9 [fix](nereids)need add mark join slot to upper project in PullUpProjectUnderApply rule (#31408)
add bb0cedeee8b [pipelineX](refactor) Rebuild relationship between dep and operator (#31487)
new ae0db0a5140 Collect high priority columns. (#31235)
new 3ac73f32582 High priority queue and map. (#31509)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (88ed1980d86)
\
N -- N -- N refs/heads/high-priority-column (3ac73f32582)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitignore | 1 +
README.md | 2 +-
be/src/agent/task_worker_pool.cpp | 3 +-
be/src/cloud/cloud_delta_writer.cpp | 13 +-
be/src/cloud/cloud_delta_writer.h | 3 +
be/src/cloud/cloud_schema_change_job.cpp | 3 +
be/src/cloud/cloud_tablets_channel.cpp | 59 +-
be/src/cloud/pb_convert.cpp | 2 +
be/src/common/config.cpp | 2 +
be/src/common/config.h | 3 +
be/src/io/file_factory.cpp | 5 +-
be/src/io/fs/benchmark/hdfs_benchmark.hpp | 9 +-
be/src/io/fs/hdfs_file_system.cpp | 13 +-
be/src/io/fs/hdfs_file_system.h | 4 +-
be/src/olap/accept_null_predicate.h | 17 +-
be/src/olap/column_predicate.h | 5 -
be/src/olap/comparison_predicate.h | 31 +-
be/src/olap/rowset/rowset_meta.cpp | 5 +
be/src/olap/rowset/rowset_meta.h | 2 +
be/src/olap/rowset/segment_v2/segment.cpp | 53 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 43 +-
be/src/olap/rowset/segment_v2/segment_iterator.h | 2 -
be/src/olap/shared_predicate.h | 179 +
be/src/olap/tablet.cpp | 6 -
be/src/olap/tablet_schema.cpp | 4 +-
be/src/olap/wal/wal_writer.cpp | 9 +
be/src/pipeline/exec/analytic_source_operator.h | 1 -
be/src/pipeline/exec/es_scan_operator.cpp | 2 +-
be/src/pipeline/exec/exchange_sink_operator.cpp | 7 -
be/src/pipeline/exec/exchange_sink_operator.h | 18 +-
be/src/pipeline/exec/exchange_source_operator.cpp | 5 +-
be/src/pipeline/exec/exchange_source_operator.h | 11 +-
be/src/pipeline/exec/file_scan_operator.cpp | 2 +-
be/src/pipeline/exec/multi_cast_data_stream_sink.h | 11 +-
be/src/pipeline/exec/olap_scan_operator.cpp | 4 +-
be/src/pipeline/exec/scan_operator.cpp | 41 +-
be/src/pipeline/exec/scan_operator.h | 9 +-
be/src/pipeline/exec/set_probe_sink_operator.h | 2 +
be/src/pipeline/exec/set_sink_operator.cpp | 2 -
be/src/pipeline/exec/set_sink_operator.h | 5 +-
be/src/pipeline/exec/set_source_operator.cpp | 7 +-
be/src/pipeline/exec/set_source_operator.h | 6 +-
be/src/pipeline/exec/sort_sink_operator.cpp | 18 +-
be/src/pipeline/exec/union_sink_operator.h | 13 +
be/src/pipeline/exec/union_source_operator.cpp | 32 +-
be/src/pipeline/exec/union_source_operator.h | 4 +
be/src/pipeline/pipeline_x/dependency.cpp | 29 +-
be/src/pipeline/pipeline_x/dependency.h | 87 +-
.../local_exchange_source_operator.cpp | 1 -
be/src/pipeline/pipeline_x/operator.cpp | 131 +-
be/src/pipeline/pipeline_x/operator.h | 35 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 15 +-
.../pipeline_x/pipeline_x_fragment_context.h | 2 -
be/src/pipeline/pipeline_x/pipeline_x_task.cpp | 44 +-
be/src/pipeline/pipeline_x/pipeline_x_task.h | 70 +-
be/src/runtime/group_commit_mgr.cpp | 11 +-
be/src/runtime/runtime_predicate.cpp | 106 +-
be/src/runtime/runtime_predicate.h | 120 +-
be/src/runtime/runtime_query_statistics_mgr.cpp | 2 +-
be/src/runtime/snapshot_loader.cpp | 3 +-
be/src/runtime/tablets_channel.cpp | 2 +-
be/src/service/backend_options.cpp | 8 +-
be/src/service/backend_options.h | 1 +
be/src/tools/meta_tool.cpp | 13 +-
be/src/util/key_util.h | 17 +-
be/src/util/s3_util.cpp | 4 +
be/src/vec/exec/vjdbc_connector.cpp | 35 +-
be/src/vec/exec/vjdbc_connector.h | 3 +
be/src/vec/exec/vsort_node.cpp | 23 +-
be/src/vec/olap/vcollect_iterator.cpp | 9 +-
be/src/vec/sink/vrow_distribution.cpp | 8 +-
be/src/vec/sink/writer/vfile_result_writer.cpp | 4 +-
be/src/vec/sink/writer/vtablet_writer.cpp | 1 +
be/test/olap/test_data/rowset_meta.json | 3 +-
cloud/src/meta-service/keys.cpp | 11 +-
cloud/src/meta-service/keys.h | 5 +
cloud/src/meta-service/meta_service.cpp | 60 +-
cloud/src/recycler/recycler.cpp | 32 +-
.../docker-compose/mysql/init/04-insert.sql | 4 +-
docs/en/community/developer-guide/minidump.md | 177 -
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 220 +-
docs/en/docs/lakehouse/sql-dialect.md | 62 +-
docs/en/docs/releasenotes/release-2.0.5.md | 73 +
.../{bitshfitleft.md => bitshiftleft.md} | 0
.../{bitshfitright.md => bitshiftright.md} | 0
...=> SHOW-CONVERT-LIGHT-SCHEMA-CHANGE-PROCESS.md} | 0
docs/sidebars.json | 36 +-
docs/zh-CN/community/developer-guide/minidump.md | 185 -
docs/zh-CN/docs/lakehouse/sql-dialect.md | 62 +-
docs/zh-CN/docs/releasenotes/release-2.0.5.md | 68 +
.../java/org/apache/doris/hudi/HudiJniScanner.java | 15 +-
.../src/main/java/org/apache/doris/hudi/Utils.java | 22 +-
.../org/apache/doris/jdbc/BaseJdbcExecutor.java | 520 +
...{JdbcExecutor.java => DefaultJdbcExecutor.java} | 6 +-
.../java/org/apache/doris/jdbc/JdbcExecutor.java | 1085 +-
.../apache/doris/jdbc/JdbcExecutorFactory.java} | 24 +-
.../org/apache/doris/jdbc/MySQLJdbcExecutor.java | 291 +
.../org/apache/doris/jdbc/OracleJdbcExecutor.java | 131 +
.../org/apache/doris/catalog/AggStateType.java | 33 +-
.../java/org/apache/doris/catalog/ScalarType.java | 4 -
.../main/java/org/apache/doris/catalog/Type.java | 49 +-
.../main/java/org/apache/doris/common/Config.java | 6 +
.../authentication/AuthenticationConfig.java | 8 +-
.../common/security/authentication/HadoopUGI.java | 20 +-
fe/fe-core/src/main/cup/sql_parser.cup | 25 +-
.../java/org/apache/doris/analysis/ColumnDef.java | 41 -
.../org/apache/doris/analysis/CreateTableStmt.java | 4 +
.../org/apache/doris/analysis/DropTableStmt.java | 4 +
.../main/java/org/apache/doris/analysis/Expr.java | 14 +-
.../apache/doris/analysis/FunctionCallExpr.java | 6 -
.../java/org/apache/doris/analysis/QueryStmt.java | 15 +-
.../java/org/apache/doris/backup/RestoreJob.java | 3 +-
.../org/apache/doris/binlog/BinlogManager.java | 4 +
.../org/apache/doris/catalog/DataProperty.java | 4 +
.../main/java/org/apache/doris/catalog/Env.java | 8 +-
.../java/org/apache/doris/catalog/MysqlTable.java | 2 +-
.../java/org/apache/doris/catalog/OlapTable.java | 3 +-
.../main/java/org/apache/doris/catalog/Tablet.java | 25 +-
.../apache/doris/cloud/catalog/CloudReplica.java | 25 +-
.../apache/doris/cloud/catalog/CloudTablet.java | 16 +-
.../cloud/datasource/CloudInternalCatalog.java | 7 +-
.../doris/cloud/system/CloudSystemInfoService.java | 13 +-
.../apache/doris/datasource/ExternalCatalog.java | 17 +-
.../apache/doris/datasource/InternalCatalog.java | 38 +-
.../datasource/hive/HiveMetaStoreClientHelper.java | 33 +-
.../datasource/iceberg/IcebergExternalTable.java | 12 -
.../datasource/paimon/PaimonExternalCatalog.java | 63 +-
.../paimon/PaimonFileExternalCatalog.java | 1 +
.../apache/doris/fs/remote/dfs/DFSFileSystem.java | 21 +-
.../org/apache/doris/nereids/CascadesContext.java | 10 +
.../org/apache/doris/nereids/hint/LeadingHint.java | 3 +-
.../jobs/cascades/OptimizeGroupExpressionJob.java | 1 +
.../doris/nereids/jobs/executor/Optimizer.java | 1 +
.../jobs/joinorder/hypergraph/GraphSimplifier.java | 2 +-
.../jobs/joinorder/hypergraph/HyperGraph.java | 2 +-
.../jobs/joinorder/hypergraph/edge/JoinEdge.java | 2 +-
.../hypergraph/receiver/PlanReceiver.java | 2 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 4 +-
.../doris/nereids/processor/post/TopNScanOpt.java | 3 +-
.../nereids/processor/pre/PlanPreprocessors.java | 7 +-
.../processor/pre/PullUpSubqueryAliasToCTE.java | 76 +
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../nereids/rules/analysis/BindExpression.java | 6 +-
.../nereids/rules/analysis/CheckAfterRewrite.java | 4 +-
.../rules/analysis/EliminateLogicalSelectHint.java | 3 +-
.../doris/nereids/rules/analysis/LeadingJoin.java | 14 +-
.../doris/nereids/rules/analysis/SlotBinder.java | 4 +-
.../nereids/rules/analysis/SubqueryToApply.java | 2 +-
.../rules/exploration/join/InnerJoinLAsscom.java | 8 +-
.../exploration/join/InnerJoinLAsscomProject.java | 4 +-
.../exploration/join/InnerJoinLeftAssociate.java | 8 +-
.../join/InnerJoinLeftAssociateProject.java | 4 +-
.../exploration/join/InnerJoinRightAssociate.java | 8 +-
.../join/InnerJoinRightAssociateProject.java | 8 +-
.../rules/exploration/join/JoinCommute.java | 5 +-
.../rules/exploration/join/JoinExchange.java | 29 +-
.../exploration/join/JoinExchangeBothProject.java | 11 +-
.../exploration/join/JoinExchangeLeftProject.java | 20 +-
.../exploration/join/JoinExchangeRightProject.java | 10 +-
.../rules/exploration/join/JoinReorderContext.java | 12 +
.../join/LogicalJoinSemiJoinTranspose.java | 8 +-
.../join/LogicalJoinSemiJoinTransposeProject.java | 8 +-
.../rules/exploration/join/OuterJoinAssoc.java | 4 +-
.../exploration/join/OuterJoinAssocProject.java | 4 +-
.../rules/exploration/join/OuterJoinLAsscom.java | 4 +-
.../exploration/join/OuterJoinLAsscomProject.java | 4 +-
.../join/SemiJoinSemiJoinTranspose.java | 4 +-
.../join/SemiJoinSemiJoinTransposeProject.java | 4 +-
.../rules/expression/ExpressionRewrite.java | 3 +-
...ogicalPartitionTopNToPhysicalPartitionTopN.java | 70 +-
.../rules/rewrite/AdjustConjunctsReturnType.java | 2 +-
.../nereids/rules/rewrite/AdjustNullable.java | 3 +-
.../rules/rewrite/ConvertInnerOrCrossJoin.java | 4 +-
.../rules/rewrite/ConvertOuterJoinToAntiJoin.java | 4 +-
.../rules/rewrite/EliminateDedupJoinCondition.java | 3 +-
.../rules/rewrite/EliminateJoinCondition.java | 3 +-
.../nereids/rules/rewrite/EliminateMarkJoin.java | 2 +-
.../nereids/rules/rewrite/EliminateNotNull.java | 3 +-
.../rewrite/EliminateNullAwareLeftAntiJoin.java | 2 +-
.../nereids/rules/rewrite/EliminateOuterJoin.java | 4 +-
.../rules/rewrite/EliminateSortUnderSubquery.java | 2 +-
.../nereids/rules/rewrite/ExistsApplyToJoin.java | 8 +-
.../rules/rewrite/ExtractFilterFromCrossJoin.java | 2 +-
.../rules/rewrite/FindHashConditionForJoin.java | 2 +-
.../doris/nereids/rules/rewrite/InApplyToJoin.java | 8 +-
.../doris/nereids/rules/rewrite/OrExpansion.java | 26 +-
.../rules/rewrite/PullUpJoinFromUnionAll.java | 4 +-
.../rules/rewrite/PullUpProjectUnderApply.java | 3 +
.../rules/rewrite/PushDownAliasThroughJoin.java | 3 +-
.../PushDownExpressionsInHashCondition.java | 2 +-
.../rules/rewrite/PushDownFilterThroughJoin.java | 3 +-
.../rules/rewrite/PushDownJoinOtherCondition.java | 2 +-
.../rewrite/PushDownTopNDistinctThroughJoin.java | 94 +-
.../rules/rewrite/PushFilterInsideJoin.java | 2 +-
.../doris/nereids/rules/rewrite/ReorderJoin.java | 10 +-
.../nereids/rules/rewrite/ScalarApplyToJoin.java | 4 +-
.../nereids/rules/rewrite/SemiJoinCommute.java | 3 +-
.../rewrite/TransposeSemiJoinLogicalJoin.java | 1 +
.../TransposeSemiJoinLogicalJoinProject.java | 1 +
.../trees/copier/LogicalPlanDeepCopier.java | 2 +-
.../trees/expressions/ArrayItemReference.java | 10 +
.../trees/expressions/MarkJoinSlotReference.java | 7 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 87 +-
.../trees/plans/logical/LogicalSelectHint.java | 4 +
.../nereids/trees/plans/logical/LogicalTopN.java | 7 +
.../org/apache/doris/planner/OriginalPlanner.java | 3 +-
.../java/org/apache/doris/qe/SessionVariable.java | 13 +
.../java/org/apache/doris/qe/ShowExecutor.java | 4 +-
.../apache/doris/service/FrontendServiceImpl.java | 8 +-
.../org/apache/doris/statistics/ColStatsData.java | 5 +
.../apache/doris/statistics/ColumnStatistic.java | 2 +-
.../doris/statistics/ColumnStatisticBuilder.java | 4 +-
.../statistics/ColumnStatisticsCacheLoader.java | 31 +-
.../apache/doris/statistics/StatisticsCache.java | 5 +-
.../org/apache/doris/system/SystemInfoService.java | 9 +-
.../ExternalFileTableValuedFunction.java | 4 +-
.../analysis/CreateMaterializedViewStmtTest.java | 16 -
.../apache/doris/catalog/ModifyBackendTest.java | 2 +-
.../doris/catalog/ReplicaAllocationTest.java | 5 +-
.../doris/clone/TabletRepairAndBalanceTest.java | 15 +-
.../org/apache/doris/nereids/memo/MemoTest.java | 21 +-
.../pattern/GroupExpressionMatchingTest.java | 16 +-
.../nereids/postprocess/TopNRuntimeFilterTest.java | 15 -
.../rules/analysis/BindSlotReferenceTest.java | 6 +-
...ctSingleTableExpressionFromDisjunctionTest.java | 6 +-
.../rewrite/FindHashConditionForJoinTest.java | 4 +-
.../rewrite/PushDownAliasThroughJoinTest.java | 18 +-
.../nereids/rules/rewrite/PushDownLimitTest.java | 4 +-
.../doris/nereids/stats/FilterEstimationTest.java | 2 +
.../doris/nereids/stats/JoinEstimateTest.java | 4 +-
.../doris/nereids/trees/plans/PlanEqualsTest.java | 6 +-
.../nereids/trees/plans/PlanToStringTest.java | 2 +-
.../doris/nereids/util/HyperGraphBuilder.java | 8 +-
.../doris/nereids/util/LogicalPlanBuilder.java | 10 +-
.../org/apache/doris/planner/QueryPlanTest.java | 6 +-
.../apache/doris/system/SystemInfoServiceTest.java | 4 +-
.../apache/doris/utframe/MockedBackendFactory.java | 7 +-
gensrc/proto/olap_file.proto | 2 +
gensrc/thrift/FrontendService.thrift | 4 +-
.../jdbc/test_mysql_jdbc_catalog.out | 2 +
regression-test/data/nereids_p0/cte/cte.out | 243 +
.../data/nereids_p0/hint/multi_leading.out | 649 ++
.../data/nereids_p0/hint/test_distribute.out | 10482 +++++++++++++++++++
.../data/nereids_p0/hint/test_leading.out | 24 +-
.../data/nereids_p0/union/or_expansion.out | 301 +
regression-test/data/nereids_rules_p0/cte/cte.out | 347 +
.../cte/test_cte_filter_pushdown.out | 0
.../cte/test_cte_name_reuse.out | 0
.../limit_push_down/limit_push_down.out | 2 +-
.../limit_push_down/order_push_down.out | 10 +-
.../push_filter_through_ptopn.out | 2 +-
.../noStatsRfPrune/query67.out | 2 +-
.../no_stats_shape/query67.out | 2 +-
.../test_primary_key_simple_case.out | 22 +
.../org/apache/doris/regression/suite/Suite.groovy | 4 +-
.../cloud_p0/conf/regression-conf-custom.groovy | 2 +-
regression-test/pipeline/cloud_p0/deploy.sh | 4 +-
regression-test/pipeline/common/doris-utils.sh | 5 +-
.../pipeline/p1/conf/regression-conf.groovy | 2 +
.../bloom_filter_p0/test_bloom_filter_hit.groovy | 5 +-
.../suites/data_reliability/check_meta.groovy | 33 +-
.../decimalv3/test_decimal256_index.groovy | 1 +
.../iceberg/test_iceberg_table_stats.groovy | 65 +
.../paimon/test_paimon_table_stats.groovy | 59 +
.../hive/test_hive_analyze_db.groovy | 225 +-
.../hive/test_hive_statistic_cache.groovy | 46 +-
.../hive/test_hive_statistics_from_hms.groovy | 18 +-
.../test_load_stream_stub_failure_injection.groovy | 2 +-
.../test_multi_replica_fault_injection.groovy | 4 +-
.../suites/inverted_index_p0/test_show_data.groovy | 12 +-
.../explain/test_global_partition_topn_plan.groovy | 76 +
.../suites/nereids_p0/hint/fix_leading.groovy | 1 +
.../suites/nereids_p0/hint/multi_leading.groovy | 132 +
.../suites/nereids_p0/hint/test_distribute.groovy | 927 ++
.../suites/nereids_p0/hint/test_leading.groovy | 1 +
.../suites/nereids_p0/union/or_expansion.groovy | 152 +-
.../cte/test_cte_filter_pushdown.groovy} | 107 +-
.../cte/test_cte_name_reuse.groovy | 0
.../cte/test_cte_reuse_with_window.groovy | 0
.../transposeJoin/transposeSemiJoinAgg.groovy | 50 +-
.../nereids_ssb_shape_sf100_p0/shape/flat.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q1.1.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q1.2.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q1.3.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q2.1.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q2.2.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q2.3.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q3.1.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q3.2.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q3.3.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q3.4.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q4.1.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q4.2.groovy | 1 +
.../nereids_ssb_shape_sf100_p0/shape/q4.3.groovy | 1 +
.../suites/statistics/analyze_stats.groovy | 17 +
.../suites/statistics/test_analyze_mtmv.groovy | 37 +-
.../suites/statistics/test_analyze_mv.groovy | 37 +-
.../test_partition_default_medium.groovy | 109 +
.../test_primary_key_simple_case.groovy | 42 +
.../suites/variant_github_events_p0/load.groovy | 1 -
.../suites/variant_p0/complexjson.groovy | 1 -
regression-test/suites/variant_p0/desc.groovy | 3 +
regression-test/suites/variant_p0/sql/gh_data.sql | 2 +-
.../suites/variant_p0/sql/implicit_cast.sql | 2 +-
304 files changed, 17024 insertions(+), 3325 deletions(-)
create mode 100644 be/src/olap/shared_predicate.h
delete mode 100644 docs/en/community/developer-guide/minidump.md
create mode 100644 docs/en/docs/releasenotes/release-2.0.5.md
rename docs/en/docs/sql-manual/sql-functions/bitwise-functions/{bitshfitleft.md => bitshiftleft.md} (100%)
rename docs/en/docs/sql-manual/sql-functions/bitwise-functions/{bitshfitright.md => bitshiftright.md} (100%)
rename docs/en/docs/sql-manual/sql-reference/Show-Statements/{SHOW-CONVERT-LIGHR-SCHEMA-CHANGE-PROCESS.md => SHOW-CONVERT-LIGHT-SCHEMA-CHANGE-PROCESS.md} (100%)
delete mode 100644 docs/zh-CN/community/developer-guide/minidump.md
create mode 100644 docs/zh-CN/docs/releasenotes/release-2.0.5.md
create mode 100644 fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/BaseJdbcExecutor.java
copy fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/{JdbcExecutor.java => DefaultJdbcExecutor.java} (99%)
copy fe/{fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSubquery.java => be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcExecutorFactory.java} (61%)
create mode 100644 fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/MySQLJdbcExecutor.java
create mode 100644 fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleJdbcExecutor.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/processor/pre/PullUpSubqueryAliasToCTE.java
create mode 100644 regression-test/data/nereids_p0/cte/cte.out
create mode 100644 regression-test/data/nereids_p0/hint/multi_leading.out
create mode 100644 regression-test/data/nereids_p0/hint/test_distribute.out
create mode 100644 regression-test/data/nereids_rules_p0/cte/cte.out
copy regression-test/data/{nereids_p0 => nereids_rules_p0}/cte/test_cte_filter_pushdown.out (100%)
copy regression-test/data/{nereids_p0 => nereids_rules_p0}/cte/test_cte_name_reuse.out (100%)
create mode 100644 regression-test/data/unique_with_mow_p0/test_primary_key_simple_case.out
copy fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSubquery.java => regression-test/suites/data_reliability/check_meta.groovy (52%)
create mode 100644 regression-test/suites/external_table_p0/iceberg/test_iceberg_table_stats.groovy
create mode 100644 regression-test/suites/external_table_p0/paimon/test_paimon_table_stats.groovy
create mode 100644 regression-test/suites/nereids_p0/explain/test_global_partition_topn_plan.groovy
create mode 100644 regression-test/suites/nereids_p0/hint/multi_leading.groovy
create mode 100644 regression-test/suites/nereids_p0/hint/test_distribute.groovy
copy regression-test/suites/{fault_injection_p0/test_load_stream_stub_failure_injection.groovy => nereids_rules_p0/cte/test_cte_filter_pushdown.groovy} (55%)
rename regression-test/suites/{nereids_p0 => nereids_rules_p0}/cte/test_cte_name_reuse.groovy (100%)
rename regression-test/suites/{nereids_p0 => nereids_rules_p0}/cte/test_cte_reuse_with_window.groovy (100%)
create mode 100644 regression-test/suites/storage_medium_p0/test_partition_default_medium.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/02: Collect high priority columns. (#31235)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch high-priority-column
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ae0db0a5140115c930445427bd8e6471cbf19a29
Author: Jibing-Li <64...@users.noreply.github.com>
AuthorDate: Wed Feb 28 11:54:40 2024 +0800
Collect high priority columns. (#31235)
---
.../doris/nereids/jobs/executor/Rewriter.java | 4 +-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
.../expression/HighPriorityColumnCollector.java | 202 +++++++++++++++++++++
.../apache/doris/statistics/AnalysisManager.java | 44 +++++
.../doris/statistics/HighPriorityColumn.java | 55 ++++++
5 files changed, 305 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
index e2d386dc910..fdba72fbf6e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
@@ -32,6 +32,7 @@ import org.apache.doris.nereids.rules.expression.CheckLegalityAfterRewrite;
import org.apache.doris.nereids.rules.expression.ExpressionNormalization;
import org.apache.doris.nereids.rules.expression.ExpressionOptimization;
import org.apache.doris.nereids.rules.expression.ExpressionRewrite;
+import org.apache.doris.nereids.rules.expression.HighPriorityColumnCollector;
import org.apache.doris.nereids.rules.rewrite.AddDefaultLimit;
import org.apache.doris.nereids.rules.rewrite.AdjustConjunctsReturnType;
import org.apache.doris.nereids.rules.rewrite.AdjustNullable;
@@ -440,7 +441,8 @@ public class Rewriter extends AbstractBatchJobExecutor {
new CollectFilterAboveConsumer(),
new CollectProjectAboveConsumer()
)
- )
+ ),
+ topic("Collect used column", custom(RuleType.COLLECT_COLUMNS, HighPriorityColumnCollector::new))
);
private static final List<RewriteJob> WHOLE_TREE_REWRITE_JOBS
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index ea76eddd1d7..363a97db0a5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -296,6 +296,7 @@ public enum RuleType {
LEADING_JOIN(RuleTypeClass.REWRITE),
REWRITE_SENTINEL(RuleTypeClass.REWRITE),
+ COLLECT_COLUMNS(RuleTypeClass.REWRITE),
// topn opts
DEFER_MATERIALIZE_TOP_N_RESULT(RuleTypeClass.REWRITE),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/HighPriorityColumnCollector.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/HighPriorityColumnCollector.java
new file mode 100644
index 00000000000..ed67ad97005
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/HighPriorityColumnCollector.java
@@ -0,0 +1,202 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.rules.expression.HighPriorityColumnCollector.CollectorContext;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCatalogRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalOlapScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.statistics.AnalysisManager;
+import org.apache.doris.statistics.util.StatisticsUtil;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Used to collect High priority column.
+ */
+public class HighPriorityColumnCollector extends DefaultPlanRewriter<CollectorContext> implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext connectContext = ConnectContext.get();
+ if (connectContext != null && connectContext.getSessionVariable().internalSession) {
+ return plan;
+ }
+ CollectorContext context = new CollectorContext();
+ plan.accept(this, context);
+ if (StatisticsUtil.enableAutoAnalyze()) {
+ context.queried.removeAll(context.usedInPredicate);
+ AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
+ analysisManager.updateColumnUsedInPredicate(context.usedInPredicate);
+ analysisManager.updateQueriedColumn(context.queried);
+ }
+ return plan;
+ }
+
+ /**
+ * Context.
+ */
+ public static class CollectorContext {
+ public Map<Slot/*project output column*/, NamedExpression/*Actual project expr*/> projects = new HashMap<>();
+
+ public Set<Slot> usedInPredicate = new HashSet<>();
+
+ public Set<Slot> queried = new HashSet<>();
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, CollectorContext context) {
+ project.child().accept(this, context);
+ List<NamedExpression> projects = project.getOutputs();
+ List<Slot> slots = project.computeOutput();
+ for (int i = 0; i < slots.size(); i++) {
+ context.projects.put(slots.get(i), projects.get(i));
+ }
+ if (project.child() instanceof LogicalCatalogRelation
+ || project.child() instanceof LogicalFilter
+ && ((LogicalFilter) project.child()).child() instanceof LogicalCatalogRelation) {
+ Set<Slot> allUsed = project.getExpressions()
+ .stream().flatMap(e -> e.<Set<SlotReference>>collect(n -> n instanceof SlotReference).stream())
+ .collect(Collectors.toSet());
+ LogicalCatalogRelation scan = project.child() instanceof LogicalCatalogRelation
+ ? (LogicalCatalogRelation) project.child()
+ : (LogicalCatalogRelation) project.child().child(0);
+ List<Slot> outputOfScan = scan.getOutput();
+ for (Slot slot : outputOfScan) {
+ if (!allUsed.contains(slot)) {
+ context.queried.remove(slot);
+ }
+ }
+ }
+ return project;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, CollectorContext context) {
+ join.child(0).accept(this, context);
+ join.child(1).accept(this, context);
+ context.usedInPredicate.addAll(
+ (join.isMarkJoin() ? join.getLeftConditionSlot() : join.getConditionSlot())
+ .stream().flatMap(s -> backtrace(s, context).stream())
+ .collect(Collectors.toSet())
+ );
+ return join;
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, CollectorContext context) {
+ aggregate.child(0).accept(this, context);
+ context.usedInPredicate.addAll(aggregate.getGroupByExpressions()
+ .stream()
+ .flatMap(e -> e.<Set<SlotReference>>collect(n -> n instanceof SlotReference).stream())
+ .flatMap(s -> backtrace(s, context).stream())
+ .collect(Collectors.toSet()));
+ return aggregate;
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, CollectorContext context) {
+ having.child(0).accept(this, context);
+ context.usedInPredicate.addAll(
+ having.getExpressions().stream()
+ .flatMap(e -> e.<Set<SlotReference>>collect(n -> n instanceof SlotReference).stream())
+ .flatMap(s -> backtrace(s, context).stream())
+ .collect(Collectors.toSet()));
+ return having;
+ }
+
+ @Override
+ public Plan visitLogicalOlapScan(LogicalOlapScan olapScan, CollectorContext context) {
+ List<Slot> slots = olapScan.getOutput();
+ context.queried.addAll(slots);
+ return olapScan;
+ }
+
+ @Override
+ public Plan visitLogicalFileScan(LogicalFileScan fileScan, CollectorContext context) {
+ List<Slot> slots = fileScan.getOutput();
+ context.queried.addAll(slots);
+ return fileScan;
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, CollectorContext context) {
+ filter.child(0).accept(this, context);
+ context.usedInPredicate.addAll(filter
+ .getExpressions()
+ .stream()
+ .flatMap(e -> e.<Set<SlotReference>>collect(n -> n instanceof SlotReference).stream())
+ .flatMap(s -> backtrace(s, context).stream())
+ .collect(Collectors.toSet()));
+ return filter;
+ }
+
+ private Set<Slot> backtrace(Slot slot, CollectorContext context) {
+ return backtrace(slot, new HashSet<>(), context);
+ }
+
+ private Set<Slot> backtrace(Slot slot, Set<Slot> path, CollectorContext context) {
+ if (path.contains(slot)) {
+ return Collections.emptySet();
+ }
+ path.add(slot);
+ if (slot instanceof SlotReference) {
+ SlotReference slotReference = (SlotReference) slot;
+ Optional<Column> col = slotReference.getColumn();
+ Optional<TableIf> table = slotReference.getTable();
+ if (col.isPresent() && table.isPresent()) {
+ return Collections.singleton(slot);
+ }
+ }
+ NamedExpression namedExpression = context.projects.get(slot);
+ if (namedExpression == null) {
+ return Collections.emptySet();
+ }
+ Set<SlotReference> slotReferences
+ = namedExpression.<Set<SlotReference>>collect(n -> n instanceof SlotReference);
+ Set<Slot> refCol = new HashSet<>();
+ for (SlotReference slotReference : slotReferences) {
+ refCol.addAll(backtrace(slotReference, path, context));
+ }
+ return refCol;
+ }
+
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
index eac50b40757..da80a48081b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
@@ -47,6 +47,8 @@ import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.persist.AnalyzeDeletionLog;
import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.qe.ConnectContext;
@@ -89,9 +91,11 @@ import java.util.Map;
import java.util.Map.Entry;
import java.util.NavigableMap;
import java.util.Optional;
+import java.util.Queue;
import java.util.Set;
import java.util.StringJoiner;
import java.util.TreeMap;
+import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.CountDownLatch;
@@ -104,6 +108,16 @@ public class AnalysisManager implements Writable {
private static final Logger LOG = LogManager.getLogger(AnalysisManager.class);
+ /**
+ * Mem only.
+ */
+ public final Queue<HighPriorityColumn> predicateColumns = new ArrayBlockingQueue<>(100);
+
+ /**
+ * Mem only.
+ */
+ public final Queue<HighPriorityColumn> queryColumns = new ArrayBlockingQueue<>(100);
+
// Tracking running manually submitted async tasks, keep in mem only
protected final ConcurrentMap<Long, Map<Long, BaseAnalysisTask>> analysisJobIdToTaskMap = new ConcurrentHashMap<>();
@@ -1086,4 +1100,34 @@ public class AnalysisManager implements Writable {
}
return false;
}
+
+
+ public void updateColumnUsedInPredicate(Set<Slot> slotReferences) {
+ updateColumn(slotReferences, predicateColumns);
+ }
+
+ public void updateQueriedColumn(Collection<Slot> slotReferences) {
+ updateColumn(slotReferences, queryColumns);
+ }
+
+ protected void updateColumn(Collection<Slot> slotReferences, Queue<HighPriorityColumn> queue) {
+ for (Slot s : slotReferences) {
+ if (!(s instanceof SlotReference)) {
+ return;
+ }
+ Optional<Column> optionalColumn = ((SlotReference) s).getColumn();
+ Optional<TableIf> optionalTable = ((SlotReference) s).getTable();
+ if (optionalColumn.isPresent() && optionalTable.isPresent()) {
+ TableIf table = optionalTable.get();
+ DatabaseIf database = table.getDatabase();
+ if (database != null) {
+ CatalogIf catalog = database.getCatalog();
+ if (catalog != null) {
+ queue.offer(new HighPriorityColumn(catalog.getId(), database.getId(),
+ table.getId(), optionalColumn.get().getName()));
+ }
+ }
+ }
+ }
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/HighPriorityColumn.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/HighPriorityColumn.java
new file mode 100644
index 00000000000..c4bc20c399a
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/HighPriorityColumn.java
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics;
+
+import java.util.Objects;
+
+public class HighPriorityColumn {
+
+ public final long catalogId;
+ public final long dbId;
+ public final long tblId;
+ public final String colName;
+
+ public HighPriorityColumn(long catalogId, long dbId, long tblId, String colName) {
+ this.catalogId = catalogId;
+ this.dbId = dbId;
+ this.tblId = tblId;
+ this.colName = colName;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(catalogId, dbId, tblId, colName);
+ }
+
+ @Override
+ public boolean equals(Object other) {
+ if (this == other) {
+ return true;
+ }
+ if (!(other instanceof HighPriorityColumn)) {
+ return false;
+ }
+ HighPriorityColumn otherCriticalColumn = (HighPriorityColumn) other;
+ return this.catalogId == otherCriticalColumn.catalogId
+ && this.dbId == otherCriticalColumn.dbId
+ && this.tblId == otherCriticalColumn.tblId
+ && this.colName.equals(otherCriticalColumn.colName);
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/02: High priority queue and map. (#31509)
Posted by li...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch high-priority-column
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3ac73f3258292fdda3fc13bd396e15c8a8ab33ef
Author: Jibing-Li <64...@users.noreply.github.com>
AuthorDate: Wed Feb 28 12:01:55 2024 +0800
High priority queue and map. (#31509)
---
.../main/java/org/apache/doris/catalog/Env.java | 20 +-
.../apache/doris/statistics/AnalysisManager.java | 33 +-
.../doris/statistics/StatisticsAutoCollector.java | 202 ++------
.../doris/statistics/StatisticsCollector.java | 4 -
.../doris/statistics/StatisticsJobAppender.java | 135 +++++
.../statistics/StatisticsAutoCollectorTest.java | 546 ---------------------
6 files changed, 213 insertions(+), 727 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 10417dca367..823b7f18fda 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -243,6 +243,7 @@ import org.apache.doris.statistics.AnalysisManager;
import org.apache.doris.statistics.StatisticsAutoCollector;
import org.apache.doris.statistics.StatisticsCache;
import org.apache.doris.statistics.StatisticsCleaner;
+import org.apache.doris.statistics.StatisticsJobAppender;
import org.apache.doris.statistics.query.QueryStats;
import org.apache.doris.system.Backend;
import org.apache.doris.system.Frontend;
@@ -516,6 +517,8 @@ public class Env {
private StatisticsAutoCollector statisticsAutoCollector;
+ private StatisticsJobAppender statisticsJobAppender;
+
private HiveTransactionMgr hiveTransactionMgr;
private TopicPublisherThread topicPublisherThread;
@@ -743,6 +746,7 @@ public class Env {
this.analysisManager = new AnalysisManager();
this.statisticsCleaner = new StatisticsCleaner();
this.statisticsAutoCollector = new StatisticsAutoCollector();
+ this.statisticsJobAppender = new StatisticsJobAppender();
this.globalFunctionMgr = new GlobalFunctionMgr();
this.workloadGroupMgr = new WorkloadGroupMgr();
this.workloadSchedPolicyMgr = new WorkloadSchedPolicyMgr();
@@ -1019,13 +1023,6 @@ public class Env {
// If not using bdb, we need to notify the FE type transfer manually.
notifyNewFETypeTransfer(FrontendNodeType.MASTER);
}
- if (statisticsCleaner != null) {
- statisticsCleaner.start();
- }
- if (statisticsAutoCollector != null) {
- statisticsAutoCollector.start();
- }
-
queryCancelWorker.start();
TopicPublisher wgPublisher = new WorkloadGroupPublisher(this);
@@ -1663,6 +1660,11 @@ public class Env {
binlogGcer.start();
columnIdFlusher.start();
insertOverwriteManager.start();
+
+ // auto analyze related threads.
+ statisticsCleaner.start();
+ statisticsAutoCollector.start();
+ statisticsJobAppender.start();
}
// start threads that should running on all FE
@@ -5998,6 +6000,10 @@ public class Env {
return statisticsAutoCollector;
}
+ public StatisticsJobAppender getStatisticsJobAppender() {
+ return statisticsJobAppender;
+ }
+
public void alterMTMVRefreshInfo(AlterMTMVRefreshInfo info) {
AlterMTMV alter = new AlterMTMV(info.getMvName(), info.getRefreshInfo(), MTMVAlterOpType.ALTER_REFRESH_INFO);
this.alter.processAlterMTMV(alter, false);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
index da80a48081b..31928491e3c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
@@ -85,6 +85,7 @@ import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
@@ -108,15 +109,12 @@ public class AnalysisManager implements Writable {
private static final Logger LOG = LogManager.getLogger(AnalysisManager.class);
- /**
- * Mem only.
- */
- public final Queue<HighPriorityColumn> predicateColumns = new ArrayBlockingQueue<>(100);
-
- /**
- * Mem only.
- */
- public final Queue<HighPriorityColumn> queryColumns = new ArrayBlockingQueue<>(100);
+ private static final int COLUMN_QUEUE_SIZE = 1000;
+ public final Queue<HighPriorityColumn> highPriorityColumns = new ArrayBlockingQueue<>(COLUMN_QUEUE_SIZE);
+ public final Queue<HighPriorityColumn> midPriorityColumns = new ArrayBlockingQueue<>(COLUMN_QUEUE_SIZE);
+ public final Map<TableIf, Set<String>> highPriorityJobs = new LinkedHashMap<>();
+ public final Map<TableIf, Set<String>> midPriorityJobs = new LinkedHashMap<>();
+ public final Map<TableIf, Set<String>> lowPriorityJobs = new LinkedHashMap<>();
// Tracking running manually submitted async tasks, keep in mem only
protected final ConcurrentMap<Long, Map<Long, BaseAnalysisTask>> analysisJobIdToTaskMap = new ConcurrentHashMap<>();
@@ -167,11 +165,6 @@ public class AnalysisManager implements Writable {
public void createAnalysisJobs(AnalyzeDBStmt analyzeDBStmt, boolean proxy) throws DdlException, AnalysisException {
DatabaseIf<TableIf> db = analyzeDBStmt.getDb();
- // Using auto analyzer if user specifies.
- if (analyzeDBStmt.getAnalyzeProperties().getProperties().containsKey("use.auto.analyzer")) {
- Env.getCurrentEnv().getStatisticsAutoCollector().analyzeDb(db);
- return;
- }
List<AnalysisInfo> analysisInfos = buildAnalysisInfosForDB(db, analyzeDBStmt.getAnalyzeProperties());
if (!analyzeDBStmt.isSync()) {
sendJobId(analysisInfos, proxy);
@@ -219,6 +212,11 @@ public class AnalysisManager implements Writable {
// Each analyze stmt corresponding to an analysis job.
public void createAnalysisJob(AnalyzeTblStmt stmt, boolean proxy) throws DdlException {
+ // Using auto analyzer if user specifies.
+ if (stmt.getAnalyzeProperties().getProperties().containsKey("use.auto.analyzer")) {
+ Env.getCurrentEnv().getStatisticsAutoCollector().processOneJob(stmt.getTable(), stmt.getColumnNames());
+ return;
+ }
AnalysisInfo jobInfo = buildAndAssignJob(stmt);
if (jobInfo == null) {
return;
@@ -1103,11 +1101,11 @@ public class AnalysisManager implements Writable {
public void updateColumnUsedInPredicate(Set<Slot> slotReferences) {
- updateColumn(slotReferences, predicateColumns);
+ updateColumn(slotReferences, highPriorityColumns);
}
public void updateQueriedColumn(Collection<Slot> slotReferences) {
- updateColumn(slotReferences, queryColumns);
+ updateColumn(slotReferences, midPriorityColumns);
}
protected void updateColumn(Collection<Slot> slotReferences, Queue<HighPriorityColumn> queue) {
@@ -1117,7 +1115,8 @@ public class AnalysisManager implements Writable {
}
Optional<Column> optionalColumn = ((SlotReference) s).getColumn();
Optional<TableIf> optionalTable = ((SlotReference) s).getTable();
- if (optionalColumn.isPresent() && optionalTable.isPresent()) {
+ if (optionalColumn.isPresent() && optionalTable.isPresent()
+ && !StatisticsUtil.isUnsupportedType(optionalColumn.get().getType())) {
TableIf table = optionalTable.get();
DatabaseIf database = table.getDatabase();
if (database != null) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
index dbb7046467a..19b9c69db08 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
@@ -17,31 +17,24 @@
package org.apache.doris.statistics;
-import org.apache.doris.catalog.Column;
-import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.OlapTable;
-import org.apache.doris.catalog.Partition;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.util.TimeUtils;
-import org.apache.doris.datasource.CatalogIf;
-import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.JobType;
import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
import org.apache.doris.statistics.util.StatisticsUtil;
-import org.apache.hudi.common.util.VisibleForTesting;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import java.time.LocalTime;
import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
import java.util.Map;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
@@ -59,8 +52,23 @@ public class StatisticsAutoCollector extends StatisticsCollector {
@Override
protected void collect() {
- if (canCollect()) {
- analyzeAll();
+ while (canCollect()) {
+ Map.Entry<TableIf, Set<String>> job = getJob();
+ if (job == null) {
+ // No more job to process, break and sleep.
+ break;
+ }
+ try {
+ TableIf table = job.getKey();
+ Set<String> columns = job.getValue()
+ .stream()
+ .filter(c -> needAnalyzeColumn(table, c))
+ .collect(Collectors.toSet());
+ processOneJob(table, columns);
+ } catch (Exception e) {
+ LOG.warn("Failed to analyze table {} with columns [{}]",
+ job.getKey().getName(), job.getValue().stream().collect(Collectors.joining(",")), e);
+ }
}
}
@@ -69,133 +77,56 @@ public class StatisticsAutoCollector extends StatisticsCollector {
&& StatisticsUtil.inAnalyzeTime(LocalTime.now(TimeUtils.getTimeZone().toZoneId()));
}
- protected void analyzeAll() {
- List<CatalogIf> catalogs = getCatalogsInOrder();
- for (CatalogIf ctl : catalogs) {
- if (!canCollect()) {
- analysisTaskExecutor.clear();
- break;
- }
- if (!ctl.enableAutoAnalyze()) {
- continue;
- }
- List<DatabaseIf> dbs = getDatabasesInOrder(ctl);
- for (DatabaseIf<TableIf> databaseIf : dbs) {
- if (!canCollect()) {
- analysisTaskExecutor.clear();
- break;
- }
- if (StatisticConstants.SYSTEM_DBS.contains(databaseIf.getFullName())) {
- continue;
- }
- try {
- analyzeDb(databaseIf);
- } catch (Throwable t) {
- LOG.warn("Failed to analyze database {}.{}", ctl.getName(), databaseIf.getFullName(), t);
- continue;
- }
- }
+ protected Map.Entry<TableIf, Set<String>> getJob() {
+ AnalysisManager manager = Env.getServingEnv().getAnalysisManager();
+ Optional<Map.Entry<TableIf, Set<String>>> job = fetchJobFromMap(manager.highPriorityJobs);
+ if (job.isPresent()) {
+ return job.get();
}
+ job = fetchJobFromMap(manager.midPriorityJobs);
+ if (job.isPresent()) {
+ return job.get();
+ }
+ job = fetchJobFromMap(manager.lowPriorityJobs);
+ return job.isPresent() ? job.get() : null;
}
- public List<CatalogIf> getCatalogsInOrder() {
- return Env.getCurrentEnv().getCatalogMgr().getCopyOfCatalog().stream()
- .sorted((c1, c2) -> (int) (c1.getId() - c2.getId())).collect(Collectors.toList());
- }
-
- public List<DatabaseIf<? extends TableIf>> getDatabasesInOrder(CatalogIf<DatabaseIf> catalog) {
- return catalog.getAllDbs().stream()
- .sorted((d1, d2) -> (int) (d1.getId() - d2.getId())).collect(Collectors.toList());
- }
-
- public List<TableIf> getTablesInOrder(DatabaseIf<? extends TableIf> db) {
- return db.getTables().stream()
- .sorted((t1, t2) -> (int) (t1.getId() - t2.getId())).collect(Collectors.toList());
- }
-
- public void analyzeDb(DatabaseIf<TableIf> databaseIf) throws DdlException {
- List<AnalysisInfo> analysisInfos = constructAnalysisInfo(databaseIf);
- for (AnalysisInfo analysisInfo : analysisInfos) {
- try {
- if (!canCollect()) {
- analysisTaskExecutor.clear();
- break;
- }
- analysisInfo = getReAnalyzeRequiredPart(analysisInfo);
- if (analysisInfo == null) {
- continue;
- }
- createSystemAnalysisJob(analysisInfo);
- } catch (Throwable t) {
- analysisInfo.message = t.getMessage();
- LOG.warn("Failed to auto analyze table {}.{}, reason {}",
- databaseIf.getFullName(), analysisInfo.tblId, analysisInfo.message, t);
- continue;
- }
+ protected Optional<Map.Entry<TableIf, Set<String>>> fetchJobFromMap(Map<TableIf, Set<String>> jobMap) {
+ synchronized (jobMap) {
+ Optional<Map.Entry<TableIf, Set<String>>> first = jobMap.entrySet().stream().findFirst();
+ first.ifPresent(entry -> jobMap.remove(entry.getKey()));
+ return first;
}
}
- protected List<AnalysisInfo> constructAnalysisInfo(DatabaseIf<? extends TableIf> db) {
- List<AnalysisInfo> analysisInfos = new ArrayList<>();
- for (TableIf table : getTablesInOrder(db)) {
- try {
- if (skip(table)) {
- continue;
- }
- createAnalyzeJobForTbl(db, analysisInfos, table);
- } catch (Throwable t) {
- LOG.warn("Failed to analyze table {}.{}.{}",
- db.getCatalog().getName(), db.getFullName(), table.getName(), t);
- continue;
- }
+ protected void processOneJob(TableIf table, Set<String> columns) throws DdlException {
+ Set<String> collect = columns.stream().filter(c -> needAnalyzeColumn(table, c)).collect(Collectors.toSet());
+ if (collect.isEmpty()) {
+ return;
}
- return analysisInfos;
+ AnalysisInfo analyzeJob = createAnalyzeJobForTbl(table, columns);
+ createSystemAnalysisJob(analyzeJob);
}
- // return true if skip auto analyze this time.
- protected boolean skip(TableIf table) {
- if (!(table instanceof OlapTable || table instanceof HMSExternalTable)) {
- return true;
- }
- // For now, only support Hive HMS table auto collection.
- if (table instanceof HMSExternalTable
- && !((HMSExternalTable) table).getDlaType().equals(HMSExternalTable.DLAType.HIVE)) {
- return true;
- }
- if (table.getDataSize(true) < StatisticsUtil.getHugeTableLowerBoundSizeInBytes() * 5) {
- return false;
- }
- TableStatsMeta tableStats = Env.getCurrentEnv().getAnalysisManager().findTableStatsStatus(table.getId());
- // means it's never got analyzed or new partition loaded data.
- if (tableStats == null || tableStats.newPartitionLoaded.get()) {
- return false;
- }
- if (tableStats.userInjected) {
- return true;
- }
- return System.currentTimeMillis()
- - tableStats.updatedTime < StatisticsUtil.getHugeTableAutoAnalyzeIntervalInMillis();
+ protected boolean needAnalyzeColumn(TableIf table, String column) {
+ //TODO: Calculate column health value.
+ return true;
}
- protected void createAnalyzeJobForTbl(DatabaseIf<? extends TableIf> db,
- List<AnalysisInfo> analysisInfos, TableIf table) {
+ protected AnalysisInfo createAnalyzeJobForTbl(TableIf table, Set<String> columns) {
AnalysisMethod analysisMethod = table.getDataSize(true) >= StatisticsUtil.getHugeTableLowerBoundSizeInBytes()
? AnalysisMethod.SAMPLE : AnalysisMethod.FULL;
- AnalysisInfo jobInfo = new AnalysisInfoBuilder()
+ return new AnalysisInfoBuilder()
.setJobId(Env.getCurrentEnv().getNextId())
- .setCatalogId(db.getCatalog().getId())
- .setDBId(db.getId())
+ .setCatalogId(table.getDatabase().getCatalog().getId())
+ .setDBId(table.getDatabase().getId())
.setTblId(table.getId())
- .setColName(
- table.getSchemaAllIndexes(false).stream()
- .filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
- .map(Column::getName).collect(Collectors.joining(","))
- )
+ .setColName(columns.stream().collect(Collectors.joining(",")))
.setAnalysisType(AnalysisInfo.AnalysisType.FUNDAMENTALS)
.setAnalysisMode(AnalysisInfo.AnalysisMode.INCREMENTAL)
.setAnalysisMethod(analysisMethod)
.setSampleRows(analysisMethod.equals(AnalysisMethod.SAMPLE)
- ? StatisticsUtil.getHugeTableSampleRows() : -1)
+ ? StatisticsUtil.getHugeTableSampleRows() : -1)
.setScheduleType(ScheduleType.AUTOMATIC)
.setState(AnalysisState.PENDING)
.setTaskIds(new ArrayList<>())
@@ -204,40 +135,5 @@ public class StatisticsAutoCollector extends StatisticsCollector {
.setTblUpdateTime(table.getUpdateTime())
.setEmptyJob(table instanceof OlapTable && table.getRowCount() == 0)
.build();
- analysisInfos.add(jobInfo);
- }
-
- @VisibleForTesting
- protected AnalysisInfo getReAnalyzeRequiredPart(AnalysisInfo jobInfo) {
- TableIf table = StatisticsUtil.findTable(jobInfo.catalogId, jobInfo.dbId, jobInfo.tblId);
- // Skip tables that are too wide.
- if (table.getBaseSchema().size() > StatisticsUtil.getAutoAnalyzeTableWidthThreshold()) {
- return null;
- }
-
- AnalysisManager analysisManager = Env.getServingEnv().getAnalysisManager();
- TableStatsMeta tblStats = analysisManager.findTableStatsStatus(table.getId());
-
- Map<String, Set<String>> needRunPartitions = null;
- String colNames = jobInfo.colName;
- if (table.needReAnalyzeTable(tblStats)) {
- needRunPartitions = table.findReAnalyzeNeededPartitions();
- } else if (table instanceof OlapTable && tblStats.newPartitionLoaded.get()) {
- OlapTable olapTable = (OlapTable) table;
- needRunPartitions = new HashMap<>();
- Set<String> partitionColumnNames = olapTable.getPartitionInfo().getPartitionColumns().stream()
- .map(Column::getName).collect(Collectors.toSet());
- colNames = partitionColumnNames.stream().collect(Collectors.joining(","));
- Set<String> partitionNames = olapTable.getAllPartitions().stream()
- .map(Partition::getName).collect(Collectors.toSet());
- for (String column : partitionColumnNames) {
- needRunPartitions.put(column, partitionNames);
- }
- }
-
- if (needRunPartitions == null || needRunPartitions.isEmpty()) {
- return null;
- }
- return new AnalysisInfoBuilder(jobInfo).setColName(colNames).setColToPartitions(needRunPartitions).build();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
index 0985b9b2b95..e26d3170178 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
@@ -61,10 +61,6 @@ public abstract class StatisticsCollector extends MasterDaemon {
@VisibleForTesting
protected void createSystemAnalysisJob(AnalysisInfo jobInfo)
throws DdlException {
- if (jobInfo.colToPartitions.isEmpty()) {
- // No statistics need to be collected or updated
- return;
- }
Map<Long, BaseAnalysisTask> analysisTasks = new HashMap<>();
AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
analysisManager.createTaskForEachColumns(jobInfo, analysisTasks, false);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobAppender.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobAppender.java
new file mode 100644
index 00000000000..73d0d1340ad
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsJobAppender.java
@@ -0,0 +1,135 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.statistics.util.StatisticsUtil;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class StatisticsJobAppender extends MasterDaemon {
+
+ private static final Logger LOG = LogManager.getLogger(StatisticsJobAppender.class);
+
+ public static final long INTERVAL = 1000;
+ public static final int JOB_MAP_SIZE = 1000;
+
+ private long currentDbId;
+ private long currentTableId;
+
+ public StatisticsJobAppender() {
+ super("Statistics Job Appender", INTERVAL);
+ }
+
+ @Override
+ protected void runAfterCatalogReady() {
+ if (!StatisticsUtil.enableAutoAnalyze()) {
+ return;
+ }
+ if (!Env.getCurrentEnv().isMaster()) {
+ return;
+ }
+ if (!StatisticsUtil.statsTblAvailable()) {
+ LOG.info("Stats table not available, skip");
+ return;
+ }
+ if (Env.isCheckpointThread()) {
+ return;
+ }
+ appendJobs();
+ }
+
+ protected void appendJobs() {
+ AnalysisManager manager = Env.getCurrentEnv().getAnalysisManager();
+ appendColumnsToJobs(manager.highPriorityColumns, manager.highPriorityJobs);
+ appendColumnsToJobs(manager.midPriorityColumns, manager.midPriorityJobs);
+ appendToLowQueue(manager.lowPriorityJobs);
+ }
+
+ protected void appendColumnsToJobs(Queue<HighPriorityColumn> columnQueue, Map<TableIf, Set<String>> jobsMap) {
+ int size = columnQueue.size();
+ for (int i = 0; i < size; i++) {
+ HighPriorityColumn column = columnQueue.poll();
+ TableIf table = StatisticsUtil.findTable(column.catalogId, column.dbId, column.tblId);
+ synchronized (jobsMap) {
+ // If job map reach the upper limit, stop putting new jobs.
+ if (!jobsMap.containsKey(table) && jobsMap.size() >= JOB_MAP_SIZE) {
+ break;
+ }
+ if (jobsMap.containsKey(table)) {
+ jobsMap.get(table).add(column.colName);
+ } else {
+ jobsMap.put(table, Collections.singleton(column.colName));
+ }
+ }
+ }
+ }
+
+ protected void appendToLowQueue(Map<TableIf, Set<String>> jobsMap) {
+
+ InternalCatalog catalog = Env.getCurrentInternalCatalog();
+ List<Long> sortedDbs = catalog.getDbIds().stream().sorted().collect(Collectors.toList());
+ for (long dbId : sortedDbs) {
+ if (dbId < currentDbId
+ || StatisticConstants.SYSTEM_DBS.contains(catalog.getDbNullable(dbId).getFullName())) {
+ continue;
+ }
+ currentDbId = dbId;
+ Optional<Database> db = catalog.getDb(dbId);
+ List<Table> tables = db.get().getTables().stream()
+ .sorted((t1, t2) -> (int) (t1.getId() - t2.getId())).collect(Collectors.toList());
+ for (Table t : tables) {
+ if (t.getId() <= currentTableId) {
+ continue;
+ }
+ synchronized (jobsMap) {
+ // If job map reach the upper limit, stop putting new jobs.
+ if (!jobsMap.containsKey(t) && jobsMap.size() >= JOB_MAP_SIZE) {
+ return;
+ }
+ Set<String> columns
+ = t.getColumns().stream().filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
+ .map(c -> c.getName()).collect(Collectors.toSet());
+ if (jobsMap.containsKey(t)) {
+ jobsMap.get(t).addAll(columns);
+ } else {
+ jobsMap.put(t, columns);
+ }
+ }
+ currentTableId = t.getId();
+ }
+ }
+ // All tables have been processed once, reset for the next loop.
+ currentDbId = 0;
+ currentTableId = 0;
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
deleted file mode 100644
index ddc19959a2d..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
+++ /dev/null
@@ -1,546 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.statistics;
-
-import org.apache.doris.catalog.Column;
-import org.apache.doris.catalog.Database;
-import org.apache.doris.catalog.DatabaseIf;
-import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.EnvFactory;
-import org.apache.doris.catalog.OlapTable;
-import org.apache.doris.catalog.PrimitiveType;
-import org.apache.doris.catalog.Table;
-import org.apache.doris.catalog.TableIf;
-import org.apache.doris.catalog.Type;
-import org.apache.doris.catalog.View;
-import org.apache.doris.common.Config;
-import org.apache.doris.common.DdlException;
-import org.apache.doris.common.FeConstants;
-import org.apache.doris.datasource.CatalogIf;
-import org.apache.doris.datasource.InternalCatalog;
-import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
-import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
-import org.apache.doris.statistics.AnalysisInfo.JobType;
-import org.apache.doris.statistics.util.StatisticsUtil;
-
-import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
-import com.google.common.collect.Sets;
-import mockit.Expectations;
-import mockit.Injectable;
-import mockit.Mock;
-import mockit.MockUp;
-import mockit.Mocked;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.time.LocalTime;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.atomic.AtomicBoolean;
-
-public class StatisticsAutoCollectorTest {
-
- @Test
- public void testAnalyzeAll(@Injectable AnalysisInfo analysisInfo) {
- new MockUp<CatalogIf>() {
- @Mock
- public Collection<DatabaseIf> getAllDbs() {
- Database db1 = new Database(1, FeConstants.INTERNAL_DB_NAME);
- Database db2 = new Database(2, "anyDB");
- List<DatabaseIf> databaseIfs = new ArrayList<>();
- databaseIfs.add(db1);
- databaseIfs.add(db2);
- return databaseIfs;
- }
- };
- new MockUp<StatisticsAutoCollector>() {
- @Mock
- public List<AnalysisInfo> constructAnalysisInfo(DatabaseIf<TableIf> db) {
- return Arrays.asList(analysisInfo, analysisInfo);
- }
-
- int count = 0;
-
- @Mock
- public AnalysisInfo getReAnalyzeRequiredPart(AnalysisInfo jobInfo) {
- return count++ == 0 ? null : jobInfo;
- }
-
- @Mock
- public void createSystemAnalysisJob(AnalysisInfo jobInfo)
- throws DdlException {
-
- }
- };
-
- StatisticsAutoCollector saa = new StatisticsAutoCollector();
- saa.runAfterCatalogReady();
- new Expectations() {
- {
- try {
- saa.createSystemAnalysisJob((AnalysisInfo) any);
- times = 1;
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
- };
- }
-
- @Test
- public void testConstructAnalysisInfo(
- @Injectable OlapTable o2, @Injectable View v) {
- new MockUp<Database>() {
- @Mock
- public List<Table> getTables() {
- List<Table> tableIfs = new ArrayList<>();
- tableIfs.add(o2);
- tableIfs.add(v);
- return tableIfs;
- }
-
- @Mock
- public String getFullName() {
- return "anyDb";
- }
- };
-
- new MockUp<OlapTable>() {
- @Mock
- public String getName() {
- return "anytable";
- }
-
- @Mock
- public List<Column> getSchemaAllIndexes(boolean full) {
- List<Column> columns = new ArrayList<>();
- columns.add(new Column("c1", PrimitiveType.INT));
- columns.add(new Column("c2", PrimitiveType.HLL));
- return columns;
- }
- };
- StatisticsAutoCollector saa = new StatisticsAutoCollector();
- List<AnalysisInfo> analysisInfoList = saa.constructAnalysisInfo(new Database(1, "anydb"));
- Assertions.assertEquals(1, analysisInfoList.size());
- Assertions.assertEquals("c1", analysisInfoList.get(0).colName.split(",")[0]);
- }
-
- @Test
- public void testGetReAnalyzeRequiredPart0() {
-
- TableIf tableIf = new OlapTable();
-
- new MockUp<OlapTable>() {
- @Mock
- protected Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- Set<String> partitionNames = new HashSet<>();
- partitionNames.add("p1");
- partitionNames.add("p2");
- Map<String, Set<String>> map = new HashMap<>();
- map.put("col1", partitionNames);
- return map;
- }
-
- @Mock
- public long getRowCount() {
- return 100;
- }
-
- @Mock
- public List<Column> getBaseSchema() {
- return Lists.newArrayList(new Column("col1", Type.INT), new Column("col2", Type.INT));
- }
- };
-
- new MockUp<StatisticsUtil>() {
- @Mock
- public TableIf findTable(long catalogName, long dbName, long tblName) {
- return tableIf;
- }
- };
- AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setAnalysisMethod(AnalysisMethod.FULL)
- .setColToPartitions(new HashMap<>()).setAnalysisType(
- AnalysisType.FUNDAMENTALS).setColName("col1").setJobType(JobType.SYSTEM).build();
- new MockUp<AnalysisManager>() {
-
- int count = 0;
-
- TableStatsMeta[] tableStatsArr =
- new TableStatsMeta[] {new TableStatsMeta(0, analysisInfo, tableIf),
- new TableStatsMeta(0, analysisInfo, tableIf), null};
-
- {
- tableStatsArr[0].updatedRows.addAndGet(100);
- tableStatsArr[1].updatedRows.addAndGet(0);
- }
-
- @Mock
- public TableStatsMeta findTableStatsStatus(long tblId) {
- return tableStatsArr[count++];
- }
- };
-
- new MockUp<StatisticsAutoCollector>() {
- @Mock
- public AnalysisInfo getAnalysisJobInfo(AnalysisInfo jobInfo, TableIf table,
- Set<String> needRunPartitions) {
- return new AnalysisInfoBuilder().build();
- }
- };
- StatisticsAutoCollector statisticsAutoCollector = new StatisticsAutoCollector();
- AnalysisInfo analysisInfo2 = new AnalysisInfoBuilder()
- .setCatalogId(0)
- .setDBId(0)
- .setTblId(0).build();
- Assertions.assertNotNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo2));
- // uncomment it when updatedRows gets ready
- // Assertions.assertNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo2));
- Assertions.assertNotNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo2));
- }
-
- @Test
- public void testSkipWideTable() {
-
- TableIf tableIf = new OlapTable();
-
- new MockUp<OlapTable>() {
- @Mock
- public List<Column> getBaseSchema() {
- return Lists.newArrayList(new Column("col1", Type.INT), new Column("col2", Type.INT));
- }
- };
-
- new MockUp<StatisticsUtil>() {
- int count = 0;
- int [] thresholds = {1, 10};
- @Mock
- public TableIf findTable(long catalogName, long dbName, long tblName) {
- return tableIf;
- }
-
- @Mock
- public int getAutoAnalyzeTableWidthThreshold() {
- return thresholds[count++];
- }
- };
-
- new MockUp<OlapTable>() {
- @Mock
- public Map<String, Set<String>> findReAnalyzeNeededPartitions() {
- HashMap<String, Set<String>> ret = Maps.newHashMap();
- ret.put("key1", Sets.newHashSet());
- return ret;
- }
- };
-
- AnalysisInfo analysisInfo = new AnalysisInfoBuilder().build();
- StatisticsAutoCollector statisticsAutoCollector = new StatisticsAutoCollector();
- Assertions.assertNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo));
- Assertions.assertNotNull(statisticsAutoCollector.getReAnalyzeRequiredPart(analysisInfo));
- }
-
- @Test
- public void testLoop() {
- AtomicBoolean timeChecked = new AtomicBoolean();
- AtomicBoolean switchChecked = new AtomicBoolean();
- new MockUp<StatisticsUtil>() {
-
- @Mock
- public boolean inAnalyzeTime(LocalTime now) {
- timeChecked.set(true);
- return true;
- }
-
- @Mock
- public boolean enableAutoAnalyze() {
- switchChecked.set(true);
- return true;
- }
- };
- StatisticsAutoCollector autoCollector = new StatisticsAutoCollector();
- autoCollector.collect();
- Assertions.assertTrue(timeChecked.get() && switchChecked.get());
-
- }
-
- @Test
- public void checkAvailableThread() {
- StatisticsAutoCollector autoCollector = new StatisticsAutoCollector();
- Assertions.assertEquals(Config.auto_analyze_simultaneously_running_task_num,
- autoCollector.analysisTaskExecutor.executors.getMaximumPoolSize());
- }
-
- @Test
- public void testSkip(@Mocked OlapTable olapTable, @Mocked TableStatsMeta stats, @Mocked TableIf anyOtherTable) {
- new MockUp<OlapTable>() {
-
- @Mock
- public long getDataSize(boolean singleReplica) {
- return StatisticsUtil.getHugeTableLowerBoundSizeInBytes() * 5 + 1000000000;
- }
- };
-
- new MockUp<AnalysisManager>() {
-
- @Mock
- public TableStatsMeta findTableStatsStatus(long tblId) {
- return stats;
- }
- };
- // A very huge table has been updated recently, so we should skip it this time
- stats.updatedTime = System.currentTimeMillis() - 1000;
- stats.newPartitionLoaded = new AtomicBoolean();
- stats.newPartitionLoaded.set(true);
- StatisticsAutoCollector autoCollector = new StatisticsAutoCollector();
- // Test new partition loaded data for the first time. Not skip.
- Assertions.assertFalse(autoCollector.skip(olapTable));
- stats.newPartitionLoaded.set(false);
- // Assertions.assertTrue(autoCollector.skip(olapTable));
- // The update of this huge table is long time ago, so we shouldn't skip it this time
- stats.updatedTime = System.currentTimeMillis()
- - StatisticsUtil.getHugeTableAutoAnalyzeIntervalInMillis() - 10000;
- Assertions.assertFalse(autoCollector.skip(olapTable));
- new MockUp<AnalysisManager>() {
-
- @Mock
- public TableStatsMeta findTableStatsStatus(long tblId) {
- return null;
- }
- };
- // can't find table stats meta, which means this table never get analyzed, so we shouldn't skip it this time
- Assertions.assertFalse(autoCollector.skip(olapTable));
- new MockUp<AnalysisManager>() {
-
- @Mock
- public TableStatsMeta findTableStatsStatus(long tblId) {
- return stats;
- }
- };
- stats.userInjected = true;
- Assertions.assertTrue(autoCollector.skip(olapTable));
- // this is not olap table nor external table, so we should skip it this time
- Assertions.assertTrue(autoCollector.skip(anyOtherTable));
- }
-
- // For small table, use full
- @Test
- public void testCreateAnalyzeJobForTbl1(
- @Injectable OlapTable t1,
- @Injectable Database db
- ) throws Exception {
- new MockUp<Database>() {
-
- @Mock
- public CatalogIf getCatalog() {
- return Env.getCurrentInternalCatalog();
- }
-
- @Mock
- public long getId() {
- return 0;
- }
- };
- new MockUp<OlapTable>() {
-
- int count = 0;
-
- @Mock
- public List<Column> getBaseSchema() {
- return Lists.newArrayList(new Column("test", PrimitiveType.INT));
- }
-
- @Mock
- public long getDataSize(boolean singleReplica) {
- return StatisticsUtil.getHugeTableLowerBoundSizeInBytes() - 1;
- }
-
- @Mock
- public BaseAnalysisTask createAnalysisTask(AnalysisInfo info) {
- return new OlapAnalysisTask(info);
- }
- };
-
- new MockUp<StatisticsUtil>() {
- @Mock
- public TableIf findTable(long catalogId, long dbId, long tblId) {
- return t1;
- }
- };
-
- StatisticsAutoCollector sac = new StatisticsAutoCollector();
- List<AnalysisInfo> jobInfos = new ArrayList<>();
- sac.createAnalyzeJobForTbl(db, jobInfos, t1);
- AnalysisInfo jobInfo = jobInfos.get(0);
- Map<String, Set<String>> colToPartitions = new HashMap<>();
- colToPartitions.put("test", new HashSet<String>() {
- {
- add("p1");
- }
- });
- jobInfo = new AnalysisInfoBuilder(jobInfo).setColToPartitions(colToPartitions).build();
- Map<Long, BaseAnalysisTask> analysisTasks = new HashMap<>();
- AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
- analysisManager.createTaskForEachColumns(jobInfo, analysisTasks, false);
- Assertions.assertEquals(1, analysisTasks.size());
- for (BaseAnalysisTask task : analysisTasks.values()) {
- Assertions.assertNull(task.getTableSample());
- }
- }
-
- // for big table, use sample
- @Test
- public void testCreateAnalyzeJobForTbl2(
- @Injectable OlapTable t1,
- @Injectable Database db
- ) throws Exception {
- new MockUp<Database>() {
-
- @Mock
- public CatalogIf getCatalog() {
- return Env.getCurrentInternalCatalog();
- }
-
- @Mock
- public long getId() {
- return 0;
- }
- };
- new MockUp<OlapTable>() {
-
- int count = 0;
-
- @Mock
- public List<Column> getBaseSchema() {
- return Lists.newArrayList(new Column("test", PrimitiveType.INT));
- }
-
- @Mock
- public long getDataSize(boolean singleReplica) {
- return StatisticsUtil.getHugeTableLowerBoundSizeInBytes() * 2;
- }
-
- @Mock
- public BaseAnalysisTask createAnalysisTask(AnalysisInfo info) {
- return new OlapAnalysisTask(info);
- }
- };
-
- new MockUp<StatisticsUtil>() {
- @Mock
- public TableIf findTable(long catalogId, long dbId, long tblId) {
- return t1;
- }
- };
-
- StatisticsAutoCollector sac = new StatisticsAutoCollector();
- List<AnalysisInfo> jobInfos = new ArrayList<>();
- sac.createAnalyzeJobForTbl(db, jobInfos, t1);
- AnalysisInfo jobInfo = jobInfos.get(0);
- Map<String, Set<String>> colToPartitions = new HashMap<>();
- colToPartitions.put("test", new HashSet<String>() {
- {
- add("p1");
- }
- });
- jobInfo = new AnalysisInfoBuilder(jobInfo).setColToPartitions(colToPartitions).build();
- Map<Long, BaseAnalysisTask> analysisTasks = new HashMap<>();
- AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
- analysisManager.createTaskForEachColumns(jobInfo, analysisTasks, false);
- Assertions.assertEquals(1, analysisTasks.size());
- for (BaseAnalysisTask task : analysisTasks.values()) {
- Assertions.assertNotNull(task.getTableSample());
- }
- }
-
- @Test
- public void testDisableAuto1() throws Exception {
- InternalCatalog catalog1 = EnvFactory.getInstance().createInternalCatalog();
- List<CatalogIf> catalogs = Lists.newArrayList();
- catalogs.add(catalog1);
-
- new MockUp<StatisticsAutoCollector>() {
- @Mock
- public List<CatalogIf> getCatalogsInOrder() {
- return catalogs;
- }
-
- @Mock
- protected boolean canCollect() {
- return false;
- }
-
- };
-
- StatisticsAutoCollector sac = new StatisticsAutoCollector();
- new Expectations(catalog1) {{
- catalog1.enableAutoAnalyze();
- times = 0;
- }};
-
- sac.analyzeAll();
- }
-
- @Test
- public void testDisableAuto2() throws Exception {
- InternalCatalog catalog1 = EnvFactory.getInstance().createInternalCatalog();
- List<CatalogIf> catalogs = Lists.newArrayList();
- catalogs.add(catalog1);
-
- Database db1 = new Database();
- List<DatabaseIf<? extends TableIf>> dbs = Lists.newArrayList();
- dbs.add(db1);
-
- new MockUp<StatisticsAutoCollector>() {
- int count = 0;
- boolean[] canCollectReturn = {true, false};
- @Mock
- public List<CatalogIf> getCatalogsInOrder() {
- return catalogs;
- }
-
- @Mock
- public List<DatabaseIf<? extends TableIf>> getDatabasesInOrder(CatalogIf<DatabaseIf> catalog) {
- return dbs;
- }
-
- @Mock
- protected boolean canCollect() {
- return canCollectReturn[count++];
- }
-
- };
-
- StatisticsAutoCollector sac = new StatisticsAutoCollector();
- new Expectations(catalog1, db1) {{
- catalog1.enableAutoAnalyze();
- result = true;
- times = 1;
- db1.getFullName();
- times = 0;
- }};
-
- sac.analyzeAll();
- }
-}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org