You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by zy...@apache.org on 2024/01/26 05:08:56 UTC
(doris) branch doris-jdk17 updated (d12cd6a7e23 -> 2afa2cdec35)
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a change to branch doris-jdk17
in repository https://gitbox.apache.org/repos/asf/doris.git
discard d12cd6a7e23 [Fix](JDK17) Fix start_be.sh (#30300)
omit 9f3788edbdf [Fix](pre-load)Fix pom (#30287)
omit 6eceef6dc89 [fix](JDK17) The objects stored in `PriorityQueue` must implement the `Comparable` interface (#30050)
omit 24fe2dac90f [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30003)
omit 835bbfc246a [Fix](JDK) Start Fe with JDK17 (#29658)
add 389dde16d38 [fix](Nereids) update assignment column name should case insensitive (#30071)
add d138b523f8d [improvement](create tablet) backend create tablet round robin among … (#29818)
add 0c4e83f5fd3 [fix](Nereids): eliminate redundant join condition after inferring condition (#30093)
add 110b55a2091 [Improvement](aggregate) optimize for small string aggregate (#29919)
add 9c7d4b518e2 [fix](Nereids): fix AssertNumRows StatsCalculator (#30053)
add e49027a5e66 [feature](cloud) Add CloudTabletMgr (#30089)
add c2b7de8f77d [Fix](Show-Delete)Missing Delete job information causes query exception (#30092)
add 53a3d787f34 [opt](docker)optimize ES docker compose (#30068)
add a38890023b5 [fix](injection) don't disturb CLOSE_LOAD message in LoadStream (#30097)
add b2a9c5b7d43 [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0 (#30096)
add 52ae3a92e8c [fix](Nereids): fix eliminate join test for pk-fk constraint (#30094)
add 6b9d7c00940 [improvement](ip function) refactor some ip functions and remove dirty codes (#30080)
add bcce5a932c6 [pipelineX](DOC) Add pipelineX doc (#30065)
add a949e1a761e [test](regression-test) fix case bugs (#30115)
add 6360ccf23d1 [improve](profile) add init probe side timer in join node (#30079)
add ae0220217ba [fix](move-memtable) only do close wait on the last sink (#30098)
add acbd78c56b6 [opt](Nereids)make orToIn rule appliable to in-pred (#29990)
add 4c2fb951f6b [refactor](close) refactor ispendingfinish logic and close logic to do close more quickly (#30021)
add a261c2788e0 [enhance-wip](multi-catalog) Speed up consume rate of hms events. (#27666)
add 6d2c7c08def [fix](tablet meta) stale rowset is not loaded while BE start (#30077)
add 1321c9224b4 [fix](cloud) Fix cloud start.sh (#30135)
add ff9c1dfd15b [pipelineX](filescan) Support parallel executing for external table scanning (#30121)
add 0fd46fb7651 [feature](merge-cloud) rewrite cloud table property (#30069)
add f9c274c338a [ci](perf) add required (#30155)
add 0af3b1ea80b [doc](ranger) update ranger doc #30101
add decc6e2585f [fix](ci) remove required of performance (#30185)
add 0f3f3b75b6f Revert "[ci](perf) add required (#30155)" (#30165)
add d6ed4149f5b [feature](function) support ip function is_ipv4_compat, is_ipv4_mapped (#29954)
add 78236868677 [Bug](fix) try to fix the coredump of streambyte decode of sse (#30190)
add 4a8669020af [fix](group commit) Fix some group commit case (#30132)
add 45ca7d587ef [opt](nereids) do not change RuntimeFilter Type from IN-OR_BLOOM to BLOOM on broadcast join (#30148)
add 7ecf4e7bc7d [config](load) set default memtable_flush_running_count_limit to 2 (#30146)
add e02f2a03071 [feature](merge-cloud) Add retrying service config (#30137)
add 37e903d5b1e [fix](phrase_prefix) fix match_phrase_prefix query incorrect result (#29946)
add 10032044cc8 [Improve](Variant) support implicit cast to numeric and string type (#30029)
add a9dd715991a [improvement](function) switch inet(6)_aton alias origin function (#30196)
add c753b220362 [regression test](schema change) add boolean type check for agg (#30186)
add ff5216a27a3 [refactor](Nereids): Eager Aggregation unify pushdown agg function (#30142)
add 75c0d6c3985 [Improvement](nereids) Make sure to catch and record exception for every materialization context (#29953)
add 27f64c549e3 [fix](routine-load) optimize error msg when meet out of range (#30118)
add f97dbd159d8 [fix](routineload) check offset when schedule tasks (#30136)
add 25da0567ae5 [improvement](statistics) Optimize drop stats operation (#30144)
add f137d4ea4c2 [temporary](log) Add temporary log for debug insert into lost result (#30140)
add f124a78fe7f [docs](ip function) add ipv4/v6 datatype and some ip functions' docs (#30194)
add bb2c90a82b7 [feature](bitwise function) bit_count/bit_shift_left/bit_shift_right implementation (#30046)
add 36591258499 [fix](planner) should return outputTupleDesc's id instead of tupleIds if outputTupleDesc is set in Plan Node (#30150)
add 84ea93652d9 [Fix](Job)Incorrect task query result of insert type (#30024)
add 9b9e077b1d0 [fix](Nereids) nullable not adjust in output exprs in result sink node (#30206)
add 1eabcc3d8b0 [fix](exec_node) crashing caused by cancelled query in ExecNode (#30192)
add 6bf835f75a6 [docker](container)create data path after rm residual data avoiding docker start fail (#30141)
add c2d731d506e [fix](Nereids) Fixed a bug where the execution plan was incorrect after ddl (#30107)
add 01c227e7ce9 [Improvement](doc) add table-buckets doc for flink connector (#29672)
add acc064565e7 [feature](function) support ip functions named to_ipv4[or_default, or_null](string) and to_ipv6[or_default, or_null](string) (#29838)
add b93b7d2b9b2 [fix](auto-partition) Fix a concurrent bug (#30086)
add 78f8df336df [fix](Nereids) result nullable of sum distinct in scalar agg is wrong (#30221)
add c929d26996e [Fix](Expr&code-style) check prepare&open before every VExpr execute (#26673)
add b61a057bb06 [fix](Nereids) user identify syntax is wrong (#30226)
add 3922c9ac018 [feature](merge-cloud) Add CloudGlobalTransactionMgr (#30117)
add d924a978bea [improvement](docs) delete unsupported data type for jdbc catalog MSSQL (#30217)
add 2b1ec70afd0 [Feature](executor)Workload Group support Non-Pipeline Execution (#30164)
add 7606538484a [fix](group commit) abort txn should use label if replay wal failed (#30219)
add 6a4c0d8706b [fix](routine-load) pause job when json path is invalid #30197
add c4291bcbacf [fix](move-memtable) make sure index descriptor is set when creating delta writer (#30157)
add c21ddbeeff6 [fix](mtmv)return MTMV with at least one available partition #30156
add 7db104e88f0 [fix](fe) Upgrade mariadb client version from 3.0.4 to 3.0.9 (#30242)
add b2e84d5158d [improvement](docs) add faq for flink-connector-doris (#30043)
add e2b5c2a3d69 [pipelineX](fix) Fix use-after-free MultiCastSourceDependency (#30199)
add 23dfb28beff [Feature](materialized-view) support predicate apprear both on key and value mv column (#30215)
add ae5b38f95e2 [fix](paimon)set timestamp's scale for parquet which has no logical type (#30119)
add 5fe0e6c215f [chore](docs) fix some docs wrong & add important comment & fe start config for old machine (#29742)
add 56fed88dfe8 [Enhancement](plan) Optimize preagg for aggregate function (#28886)
add a92172c54a4 [doc](Variant) fix doc rendering and add version bar (#30249)
add 341f14269a3 [improvement](jdbc catalog) Optimize the Close logic of JDBC client (#30236)
add 7037abc6231 [refactor](close)Full refactor async writer (#30082)
add 2560816ab27 Add 2.1 version workload group document (#30212)
add e62960d9cbc [feature](merge-cloud) Add warn log filenum quota flags for glog 0.6.0 (#30213)
add a373a07e7a0 [fix](planner) row policy rewriter generate wrong plan on join table ref (#30233)
add 079ac86d1a6 [Imporve](RF) Support remote broadcast join build bf exactly (#30247)
add bbc99119e0d [pipelineX](fix) Fix coredump if prepare failed (#30250)
add ca5bb62817f [chore](ci) remove some unused code (#30253)
add 7d840cfa51e [enhancement](Nereids): add builder for hyper graph (#30061)
add 5a68fc7ca97 [ci](cloud ut) trigger new teamcity pipeline cloud ut (#30154)
add 62b5210b59e [feature](Planner): Push down TopNDistinct through Join (#30216)
add 84ee3182ae9 [enhancement](recover) Support skipping bad tablet in select by session variable (#30241)
add e8e33ddacc2 [fix](mtmv)mtmv default open enableNereidsDML #30235
add fe602977069 [security] Don't print token (#30227)
add b203497bcd4 remove some unused member function of IFunctionBase (#30260)
add 52e20cff66d [improve](column) support append_data_by_selector function in const column (#29996)
add 06cbb96e79d [feature](merge-cloud) cloud meta mgr impl prepare/commit/sync rowset (#30128)
add 0f8e30be469 [fix](move-memtable) fix schema use-after-free in delta writer v2 (#30254)
add 140b349c936 [fix](Nereids) fix bug in case-when/if stats estimation (#30265)
add 1108cc5fb2a [feature](merge-cloud) Register runtime adaptor factory for CloudReplica/CloudPartition (#30251)
add 2e6367ffaad [feature](merge-cloud) Support cloud snapshot version (#30232)
add 5836294c45d [opt](Nereids) make runtime filter target support expression (#30131)
add dd2e0ab8543 [fix](nereids)group by expr may be lost in EliminateGroupByConstant rule (#30274)
add 8167a756f2e [fix](Nereids) group by key should be required (#30273)
add 7feb5ea6ad7 [fix](chore) path gc should consider tablet migration (#30095)
add 838aa20f1e7 [improvement](cooldown) print the cooldown version when follow cooldown version (#30239)
add 00283a5f421 [fix](fe-memory) Fix fe schema change high memory usage (#30231)
add 1358e259c1e [feature](function) support ip function named ipv4_cidr_to_range(addr, cidr) (#29819)
add 064717f2803 [feature](merge-cloud) Add `CloudClusterChecker` to sync cluster info… (#30243)
add 85c19a7ce8d [test](ut) added UT cases for show create materialized view (#29563)
add 54aad27a4ee [fix](datatype) fixed decimal type implicit cast handling in BinaryPredicate (#30181)
add 22fc5ef3076 [feature](cloud) Abstract StorageEngine (#30245)
add 4e48e19ca74 [Improvement](Nereids) Support to query rewrite by materialized view when join input has aggregate (#30230)
add 9dbf059dbf4 [fix](catalog) fix Uninitialized connection pool parameters in hmsjdbcclient (#30262)
add 48793d79097 [fix](function) make STRLEFT and STRRIGHT and SUBSTR function DEPEND_ON_ARGUMENT (#28352)
add b57369103c6 [fix](Cooldown) Enhance calculate logic of _has_data_to_cooldown (#30244)
add 6d32a856897 [test](ut) added UT cases for show build index (#29561)
add 7d9300947b3 [feat](Nereids): eliminate left outer join by unique when comparing mv (#30228)
add 171f48d6d88 [fix](Nereids) relation in constraints should be multi parts (#30293)
add 276625d17d4 [pipelineX](refactor) refine relationship between shared state and dependency (#30294)
add fd74762b26f [regression test](schema change) add case for partition (#30195)
add fad3ecc3774 [regression test](schema change) add case for tinyint/smallint/int/bigint/float/double type in agg (#30193)
add 7928d2785db [Feature](materialized-view) support create mv with count(*) (#30313)
add 5810ce751ac [optimize](inverted index) Use inverted index filter null field (#30237)
add 49c35900c74 [fix](ci) add single quote to the value of the session variables when setting it (#30295)
add aaa854f8ba8 [fix](Nereids): Except just can merge with left deep shape (#30270)
add cf7c6cb95c6 [improvement](statistics)Catch load column stats exception, avoid print too much stack info to fe.out #30315
add b304d265c50 [docs](fix)fix max_filter_ratio default value #30318
add 6bfb85b665e [improvement](statistics nereids)Nereids support select mv. (#30267)
add 7f843294359 [fix](char) fix char column type not check lenght (#30266)
add eca32d35370 [Fix](executor)Fix return stack to fe #30316
add 23694053e01 [Docs](arm) update arm compilation and auto partition (#30311)
add 71de5d835f1 [Fix](smooth-upgrade) fix unnecessary high version of smooth upgrade (#30283)
add 26d6855d5b2 [fix](merge-cloud) Fix ali open sdk gcc compile issue (#30308)
add b24fe86485e [improve](streamload) do not print stack trace when get data from pipe meet cancel (#30297)
add 05331556f31 [minor](Nereids): enable PushDownTopNDistinctThroughJoin (#30275)
add 7bc5a48ec90 [regression test](framework) add waitFor action (#30289)
add 4c9c8b851bb [feature](replica) Add drop replica safely on backend (#30303)
add 4c570c34b7f [Fix](executor)Release resource correctly when drop workload group (#30279)
add 9315f53ef8b [Feature](materialized-view) support rewrite case when to if on legacy planner to make mv work (#30320)
add b3fb0e10b52 [Chore](join) split out join hash map from hash map (#30280)
add 691cb0e4436 [Fix](inverted index) fix data size when drop inverted index (#30327)
add e8797ed59a9 [doc](fix) K8s doc modify and add pvc network docs (#30307)
add aa3ff1ebf72 [fix](fe) Fix `FrontendHbResponse` serialization compatibility problem (#30337)
add 0800fdc626f [fix](inverted index)support merge null_bitmap during index compaction (#30326)
add 70ce5f3e389 [Fix](doc) Modify some default values in the documentation (#30304)
add d2fcf008605 [bugfix](core) fix core due to send rpc and request is deconstructed (#30344)
add 7de6744f1e1 [Bug](pipeline) Fix pipeline load lose data use wrong var check order (#30341)
add 8f49333c4a2 [fix](Nereids) fix create array function type coercion (#30329)
add c9969587457 [feat](log) Add more log about cost time for `dropTable` (#30284)
add cb9cfcf9678 [refactor](cmake) Refactor be CMakeLists ENABLE_CLANG_COVERAGE CMAKE_CXX_FLAGS byadd_compile_options (#30333)
add 35c2a529fac [fix](Nereids): when predicate contains right output, don't convert outer to anti join (#30276)
add 8db0cf35f2d [feat](Nereids): eliminate inner join by pk fk when comparing mv (#30258)
add 168abaeb279 [improve](routine-load) optimize error msg when failed to fetch Kafka info #30298
add 2cd442509a1 Fix statistics p0. (#30351)
add b92b3d1e58c [fix](meta) remove default_cluster prefix in recover db log (#30350)
add 7e9f410b624 [improvement](replica) set replica drop write editlog (#30345)
add cd6a6c0de2e [fix](Nereids) some special expression should not be constant (#30305)
add d5df146ff1a [doc](release) fix connector release doc (#30362)
add ed7325c5588 [fix](ci) trigger cloudut (#30370)
add 42fbed2c0cf [opt](inverted index)Add RAM directory null cases (#30353)
add 81ec54c365f [feat](log) Update RestApiExceptionHandler log level from debug to warn (#30306)
add d732824addb [memoryleak](execontext) dctor should be virtual to avoid memory leak (#30359)
add 520293aeba2 [fix](move-memtable) all sinks wait stream close for load timeout (#30356)
add 9cde5f70106 [feature](load) enable memtable on sink node by default (#30372)
add 27e816c0987 [fix](group_commit) Fix commit txn problem when replaying wal (#30361)
add dfed34eca67 Revert "[temporary](log) Add temporary log for debug insert into lost result (#30140)" (#30382)
add d24254e146a [feature](journal) Add a method to write a set of journals in batch (#30380)
add 74e0f441bb4 [BugFix](MutilCatalog) fix local file system unavailable in iceberg hadoop catalog (#26769)
add d33fce203b3 [fix](fe)add from_unixtime to null_result_with_one_null_param_functions set (#30388)
add 2f2f9543c9e [minor](Cooldown) Log the old rowset id when uploading rowset to S3 #30339
add 53760a54dd6 [improvement](create tablet) be choose disk tolerate with little skew (#30354)
add d52b1a0f622 [chore] Remove unused test_show_create_catalog.out (#30290)
add 76a352fa9e6 [opt](nereids)set flag to indicate if bloom filter size is calculated by ndv (#30278)
add c2dd41ca41c [pipelineX](localexchange) Adjust local exchange plan rule (#30393)
add 1e3a167d649 [Refactor](Rf) refactor the code of runtime filter (#30268)
add a13a80035d3 [fix](function) fix error result in time_to_sec and timediff (#30248)
add 192dc9cc2c1 [chore](Nereids) enable nereids if update from doris 1.x (#30369)
new 2592742fce2 [Fix](JDK) Start Fe with JDK17 (#29658)
new 1c2dd126deb [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30003)
new f7946fb7a33 [fix](JDK17) The objects stored in `PriorityQueue` must implement the `Comparable` interface (#30050)
new 1320b3aabfb [Fix](pre-load)Fix pom (#30287)
new 2afa2cdec35 [Fix](JDK17) Fix start_be.sh (#30300)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (d12cd6a7e23)
\
N -- N -- N refs/heads/doris-jdk17 (2afa2cdec35)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/comment-to-trigger-teamcity.yml | 36 +-
be/CMakeLists.txt | 3 +-
be/src/agent/agent_server.cpp | 17 +-
be/src/agent/agent_server.h | 9 +-
be/src/agent/be_exec_version_manager.h | 11 +-
be/src/agent/heartbeat_server.cpp | 14 +-
be/src/agent/heartbeat_server.h | 4 +-
be/src/agent/task_worker_pool.cpp | 39 +-
be/src/agent/task_worker_pool.h | 2 +-
be/src/agent/workload_group_listener.cpp | 6 +-
be/src/cloud/cloud_meta_mgr.cpp | 345 ++++-
be/src/cloud/cloud_meta_mgr.h | 2 +-
be/src/cloud/cloud_storage_engine.cpp | 62 +
be/src/cloud/cloud_storage_engine.h | 28 +-
be/src/cloud/cloud_tablet.cpp | 67 +-
be/src/cloud/cloud_tablet.h | 35 +-
be/src/cloud/cloud_tablet_mgr.cpp | 356 ++++++
be/src/cloud/cloud_tablet_mgr.h | 75 ++
be/src/cloud/config.cpp | 9 +
be/src/cloud/config.h | 12 +
be/src/clucene | 2 +-
be/src/common/config.cpp | 12 +-
be/src/common/config.h | 8 +-
be/src/common/daemon.cpp | 21 +-
be/src/common/exception.h | 2 +-
be/src/common/status.h | 14 +-
be/src/exec/data_sink.h | 6 +-
be/src/exec/exec_node.cpp | 72 +-
be/src/exec/exec_node.h | 23 +-
.../exec/schema_scanner/schema_rowsets_scanner.cpp | 6 +-
be/src/exprs/runtime_filter.cpp | 4 +-
be/src/http/action/check_tablet_segment_action.cpp | 14 +-
be/src/http/action/check_tablet_segment_action.h | 4 +-
be/src/http/action/checksum_action.cpp | 7 +-
be/src/http/action/checksum_action.h | 7 +-
be/src/http/action/compaction_action.cpp | 28 +-
be/src/http/action/compaction_action.h | 8 +-
be/src/http/action/download_binlog_action.cpp | 28 +-
be/src/http/action/download_binlog_action.h | 4 +-
be/src/http/action/http_stream.cpp | 6 +-
be/src/http/action/meta_action.cpp | 27 +-
be/src/http/action/pad_rowset_action.cpp | 24 +-
be/src/http/action/pad_rowset_action.h | 17 +-
be/src/http/action/reload_tablet_action.cpp | 8 +-
be/src/http/action/reload_tablet_action.h | 7 +-
be/src/http/action/report_action.cpp | 3 +-
be/src/http/action/restore_tablet_action.cpp | 14 +-
be/src/http/action/restore_tablet_action.h | 6 +-
be/src/http/action/snapshot_action.cpp | 8 +-
be/src/http/action/snapshot_action.h | 7 +-
be/src/http/action/stream_load.cpp | 6 +-
be/src/http/action/tablet_migration_action.cpp | 6 +-
be/src/http/action/tablet_migration_action.h | 15 +-
be/src/http/action/tablets_distribution_action.cpp | 38 +-
be/src/http/action/tablets_distribution_action.h | 4 +-
be/src/http/action/tablets_info_action.cpp | 19 +-
be/src/http/action/tablets_info_action.h | 3 +-
be/src/http/default_path_handlers.cpp | 13 +-
be/src/io/fs/stream_load_pipe.cpp | 6 +-
be/src/olap/base_tablet.cpp | 160 ++-
be/src/olap/base_tablet.h | 29 +-
be/src/olap/compaction.cpp | 42 +-
be/src/olap/data_dir.cpp | 15 +-
be/src/olap/delta_writer_context.h | 4 +-
be/src/olap/delta_writer_v2.cpp | 14 +-
be/src/olap/delta_writer_v2.h | 8 +-
be/src/olap/memtable_flush_executor.cpp | 8 +-
be/src/olap/memtable_flush_executor.h | 2 +-
be/src/olap/memtable_writer.cpp | 5 +-
be/src/olap/merger.cpp | 12 +-
be/src/olap/olap_server.cpp | 9 +-
be/src/olap/push_handler.cpp | 49 +-
be/src/olap/push_handler.h | 14 +-
be/src/olap/rowset/beta_rowset.cpp | 20 +
be/src/olap/rowset/beta_rowset.h | 1 +
be/src/olap/rowset/rowset_factory.cpp | 9 +-
be/src/olap/rowset/rowset_factory.h | 11 +-
be/src/olap/rowset/rowset_meta.cpp | 18 +-
be/src/olap/rowset/rowset_meta.h | 8 +-
be/src/olap/rowset/segcompaction.cpp | 7 +-
.../inverted_index/query/phrase_prefix_query.cpp | 5 +-
be/src/olap/rowset/segment_v2/segment.cpp | 15 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 +
be/src/olap/rowset/segment_v2/segment_writer.cpp | 5 +-
.../rowset/segment_v2/vertical_segment_writer.cpp | 5 +-
be/src/olap/rowset_builder.cpp | 7 +-
be/src/olap/schema_change.cpp | 25 +-
be/src/olap/single_replica_compaction.cpp | 20 +-
be/src/olap/snapshot_manager.cpp | 39 +-
be/src/olap/snapshot_manager.h | 29 +-
be/src/olap/storage_engine.cpp | 238 +++-
be/src/olap/storage_engine.h | 219 ++--
be/src/olap/tablet.cpp | 185 +--
be/src/olap/tablet.h | 27 +-
be/src/olap/tablet_manager.cpp | 4 +-
be/src/olap/tablet_meta.cpp | 11 +
be/src/olap/tablet_meta.h | 6 +
be/src/olap/tablet_schema.cpp | 12 +
be/src/olap/tablet_schema.h | 2 +
be/src/olap/task/engine_batch_load_task.cpp | 29 +-
be/src/olap/task/engine_batch_load_task.h | 17 +-
be/src/olap/task/engine_checksum_task.cpp | 23 +-
be/src/olap/task/engine_checksum_task.h | 15 +-
be/src/olap/task/engine_clone_task.cpp | 50 +-
be/src/olap/task/engine_clone_task.h | 31 +-
be/src/olap/task/engine_index_change_task.cpp | 22 +-
be/src/olap/task/engine_index_change_task.h | 12 +-
be/src/olap/task/engine_publish_version_task.cpp | 73 +-
be/src/olap/task/engine_publish_version_task.h | 33 +-
be/src/olap/task/engine_storage_migration_task.cpp | 26 +-
be/src/olap/task/engine_storage_migration_task.h | 22 +-
be/src/olap/task/engine_task.h | 11 +-
be/src/olap/task/index_builder.cpp | 73 +-
be/src/olap/task/index_builder.h | 17 +-
be/src/olap/wal/wal_manager.cpp | 10 +-
be/src/olap/wal/wal_reader.cpp | 16 +-
be/src/olap/wal/wal_reader.h | 2 -
be/src/olap/wal/wal_table.cpp | 30 +-
be/src/olap/wal/wal_table.h | 2 +-
be/src/olap/wal/wal_writer.cpp | 6 +-
be/src/pipeline/exec/aggregation_sink_operator.cpp | 4 +-
be/src/pipeline/exec/aggregation_sink_operator.h | 14 +-
.../pipeline/exec/aggregation_source_operator.cpp | 9 +-
be/src/pipeline/exec/aggregation_source_operator.h | 2 +-
be/src/pipeline/exec/analytic_sink_operator.h | 4 +-
be/src/pipeline/exec/analytic_source_operator.h | 3 +-
be/src/pipeline/exec/exchange_sink_buffer.cpp | 62 +-
be/src/pipeline/exec/exchange_sink_buffer.h | 5 +-
be/src/pipeline/exec/exchange_sink_operator.cpp | 34 +-
be/src/pipeline/exec/exchange_sink_operator.h | 2 -
be/src/pipeline/exec/hashjoin_build_sink.cpp | 2 +-
be/src/pipeline/exec/hashjoin_probe_operator.cpp | 1 +
be/src/pipeline/exec/hashjoin_probe_operator.h | 1 +
be/src/pipeline/exec/jdbc_table_sink_operator.h | 1 +
.../pipeline/exec/multi_cast_data_stream_sink.cpp | 11 -
be/src/pipeline/exec/multi_cast_data_stream_sink.h | 3 +-
.../exec/multi_cast_data_stream_source.cpp | 8 +-
.../pipeline/exec/multi_cast_data_stream_source.h | 1 -
be/src/pipeline/exec/olap_table_sink_operator.h | 1 +
be/src/pipeline/exec/olap_table_sink_v2_operator.h | 1 +
be/src/pipeline/exec/operator.h | 26 +-
.../pipeline/exec/partition_sort_sink_operator.cpp | 6 +-
be/src/pipeline/exec/result_file_sink_operator.cpp | 6 +-
be/src/pipeline/exec/result_file_sink_operator.h | 1 +
be/src/pipeline/exec/set_probe_sink_operator.cpp | 2 +-
be/src/pipeline/exec/set_probe_sink_operator.h | 2 +-
be/src/pipeline/exec/set_sink_operator.cpp | 18 +-
be/src/pipeline/exec/set_sink_operator.h | 2 +-
be/src/pipeline/exec/set_source_operator.cpp | 9 +-
be/src/pipeline/exec/sort_sink_operator.cpp | 1 -
be/src/pipeline/exec/table_function_operator.cpp | 1 -
be/src/pipeline/exec/union_source_operator.cpp | 30 +-
be/src/pipeline/exec/union_source_operator.h | 1 -
be/src/pipeline/pipeline_fragment_context.cpp | 41 +-
be/src/pipeline/pipeline_fragment_context.h | 5 +-
be/src/pipeline/pipeline_task.cpp | 13 +-
be/src/pipeline/pipeline_task.h | 5 -
be/src/pipeline/pipeline_x/dependency.cpp | 5 +
be/src/pipeline/pipeline_x/dependency.h | 68 +-
.../local_exchange_source_operator.cpp | 6 +-
be/src/pipeline/pipeline_x/operator.cpp | 102 +-
be/src/pipeline/pipeline_x/operator.h | 49 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 28 +-
be/src/pipeline/pipeline_x/pipeline_x_task.cpp | 49 +-
be/src/pipeline/pipeline_x/pipeline_x_task.h | 29 +-
be/src/pipeline/task_scheduler.cpp | 84 +-
be/src/pipeline/task_scheduler.h | 6 +-
be/src/runtime/exec_env.cpp | 16 +-
be/src/runtime/exec_env.h | 15 +-
be/src/runtime/exec_env_init.cpp | 10 +-
be/src/runtime/fragment_mgr.cpp | 71 +-
be/src/runtime/group_commit_mgr.cpp | 12 +-
be/src/runtime/load_channel.cpp | 12 +-
be/src/runtime/load_stream.cpp | 38 +-
be/src/runtime/load_stream.h | 3 +-
be/src/runtime/load_stream_writer.cpp | 5 +-
be/src/runtime/memory/cache_policy.h | 6 +
be/src/runtime/plan_fragment_executor.cpp | 1 -
be/src/runtime/query_context.cpp | 29 +-
be/src/runtime/query_context.h | 17 +-
be/src/runtime/result_writer.h | 2 +
be/src/runtime/runtime_filter_mgr.cpp | 182 +--
be/src/runtime/runtime_filter_mgr.h | 46 +-
be/src/runtime/runtime_state.cpp | 18 -
be/src/runtime/runtime_state.h | 5 -
be/src/runtime/snapshot_loader.cpp | 62 +-
be/src/runtime/snapshot_loader.h | 14 +-
be/src/runtime/tablets_channel.cpp | 16 +-
be/src/runtime/tablets_channel.h | 2 +-
be/src/runtime/task_execution_context.h | 3 +
be/src/runtime/task_group/task_group.h | 17 +
be/src/runtime/task_group/task_group_manager.cpp | 139 +-
be/src/runtime/task_group/task_group_manager.h | 11 +-
.../workload_sched_policy_mgr.cpp | 1 -
be/src/service/backend_service.cpp | 17 +-
be/src/service/backend_service.h | 14 +-
be/src/service/brpc_service.cpp | 9 +-
be/src/service/http_service.cpp | 213 ++--
be/src/service/http_service.h | 6 +
be/src/service/point_query_executor.cpp | 14 +-
be/src/vec/columns/column_const.h | 4 +-
be/src/vec/columns/column_nullable.cpp | 6 +-
be/src/vec/columns/column_nullable.h | 4 +-
be/src/vec/common/hash_table/hash_map.h | 382 +-----
be/src/vec/common/hash_table/hash_map_context.h | 76 +-
be/src/vec/common/hash_table/hash_table.h | 28 -
.../hash_table/{hash_map.h => join_hash_table.h} | 282 +---
.../vec/common/hash_table/partitioned_hash_map.h | 3 +-
.../vec/common/hash_table/partitioned_hash_table.h | 11 -
be/src/vec/common/hash_table/ph_hash_map.h | 8 -
be/src/vec/common/hash_table/string_hash_map.h | 68 +-
be/src/vec/common/hash_table/string_hash_table.h | 219 ++--
be/src/vec/common/memcpy_small.h | 32 +
be/src/vec/common/unaligned.h | 33 +-
be/src/vec/core/block.cpp | 3 +-
be/src/vec/data_types/data_type_decimal.cpp | 14 +-
be/src/vec/data_types/data_type_nullable.cpp | 33 +-
be/src/vec/data_types/data_type_number_base.cpp | 15 +-
be/src/vec/data_types/data_type_string.cpp | 14 +-
be/src/vec/exec/distinct_vaggregation_node.cpp | 4 +-
be/src/vec/exec/format/json/new_json_reader.cpp | 6 +-
.../exec/format/parquet/parquet_column_convert.h | 28 +
be/src/vec/exec/join/process_hash_table_probe.h | 1 +
.../vec/exec/join/process_hash_table_probe_impl.h | 18 +-
be/src/vec/exec/join/vhash_join_node.cpp | 3 +-
be/src/vec/exec/join/vhash_join_node.h | 1 +
be/src/vec/exec/join/vjoin_node_base.h | 19 +-
be/src/vec/exec/scan/new_olap_scanner.cpp | 23 +-
be/src/vec/exec/scan/scanner_context.h | 2 +-
be/src/vec/exec/vaggregation_node.cpp | 30 +-
be/src/vec/exec/vpartition_sort_node.cpp | 5 +-
be/src/vec/exec/vset_operation_node.cpp | 23 +-
be/src/vec/exec/vset_operation_node.h | 28 +-
be/src/vec/exec/vsort_node.cpp | 2 -
be/src/vec/exec/vtable_function_node.cpp | 1 -
be/src/vec/exec/vunion_node.cpp | 21 +-
be/src/vec/exprs/vbitmap_predicate.cpp | 4 +
be/src/vec/exprs/vbloom_predicate.cpp | 4 +
be/src/vec/exprs/vcase_expr.cpp | 9 +-
be/src/vec/exprs/vcast_expr.cpp | 4 +
be/src/vec/exprs/vcolumn_ref.h | 10 +
be/src/vec/exprs/vdirect_in_predicate.h | 19 +-
be/src/vec/exprs/vectorized_fn_call.cpp | 19 +-
be/src/vec/exprs/vexpr.cpp | 62 +-
be/src/vec/exprs/vexpr.h | 63 +-
be/src/vec/exprs/vexpr_context.cpp | 28 +-
be/src/vec/exprs/vexpr_context.h | 15 +-
be/src/vec/exprs/vin_predicate.cpp | 4 +
be/src/vec/exprs/vlambda_function_call_expr.h | 12 +-
be/src/vec/exprs/vlambda_function_expr.h | 15 +
be/src/vec/exprs/vliteral.cpp | 19 +-
be/src/vec/exprs/vliteral.h | 5 +
be/src/vec/exprs/vmatch_predicate.cpp | 5 +-
be/src/vec/exprs/vruntimefilter_wrapper.cpp | 15 +-
be/src/vec/exprs/vslot_ref.cpp | 11 +
be/src/vec/exprs/vslot_ref.h | 5 +-
be/src/vec/exprs/vtuple_is_null_predicate.cpp | 9 +
be/src/vec/exprs/vtuple_is_null_predicate.h | 4 +-
be/src/vec/functions/function.h | 113 --
be/src/vec/functions/function_bit.cpp | 2 +-
be/src/vec/functions/function_bit_count.cpp | 61 +
be/src/vec/functions/function_bit_shift.cpp | 90 ++
be/src/vec/functions/function_cast.h | 4 -
be/src/vec/functions/function_ip.cpp | 31 +-
be/src/vec/functions/function_ip.h | 674 ++++++----
be/src/vec/functions/function_java_udf.h | 13 +-
be/src/vec/functions/function_reverse.h | 2 -
be/src/vec/functions/function_rpc.h | 4 -
be/src/vec/functions/function_string.cpp | 8 +-
be/src/vec/functions/function_string.h | 332 ++++-
be/src/vec/functions/function_string_to_string.h | 10 +-
be/src/vec/functions/function_unary_arithmetic.h | 3 +-
be/src/vec/functions/if.cpp | 5 -
be/src/vec/functions/is_not_null.cpp | 5 -
be/src/vec/functions/is_null.cpp | 4 -
be/src/vec/functions/math.cpp | 12 +-
be/src/vec/functions/simple_function_factory.h | 4 +
be/src/vec/olap/block_reader.cpp | 5 +-
be/src/vec/olap/vertical_block_reader.cpp | 5 +-
be/src/vec/runtime/vdata_stream_recvr.h | 8 +-
be/src/vec/sink/async_writer_sink.h | 38 +-
be/src/vec/sink/load_stream_stub.cpp | 3 -
be/src/vec/sink/load_stream_stub.h | 1 -
be/src/vec/sink/vdata_stream_sender.cpp | 22 +-
be/src/vec/sink/vdata_stream_sender.h | 2 +-
be/src/vec/sink/vresult_file_sink.cpp | 19 +-
be/src/vec/sink/writer/async_result_writer.cpp | 50 +-
be/src/vec/sink/writer/async_result_writer.h | 11 +-
be/src/vec/sink/writer/vjdbc_table_writer.h | 6 +-
be/src/vec/sink/writer/vodbc_table_writer.h | 6 +-
be/src/vec/sink/writer/vtablet_writer.cpp | 79 +-
be/src/vec/sink/writer/vtablet_writer.h | 24 +-
be/src/vec/sink/writer/vtablet_writer_v2.cpp | 21 +-
be/src/vec/sink/writer/vtablet_writer_v2.h | 1 +
be/test/agent/task_worker_pool_test.cpp | 6 +-
be/test/exec/test_data/wal_scanner/wal | Bin 180 -> 220 bytes
.../compaction_delete_bitmap_calculator_test.cpp | 38 +-
be/test/olap/delete_bitmap_calculator_test.cpp | 16 +-
be/test/olap/delete_handler_test.cpp | 12 +-
be/test/olap/delta_writer_test.cpp | 111 +-
.../olap/engine_storage_migration_task_test.cpp | 56 +-
be/test/olap/memtable_flush_executor_test.cpp | 11 +-
be/test/olap/memtable_memory_limiter_test.cpp | 23 +-
be/test/olap/ordered_data_compaction_test.cpp | 24 +-
be/test/olap/path_gc_test.cpp | 1 -
be/test/olap/remote_rowset_gc_test.cpp | 8 +-
be/test/olap/rowid_conversion_test.cpp | 26 +-
be/test/olap/rowset/beta_rowset_test.cpp | 16 -
be/test/olap/rowset/rowset_meta_manager_test.cpp | 13 -
be/test/olap/storage_engine_test.cpp | 3 +-
be/test/olap/tablet_cooldown_test.cpp | 141 +-
be/test/olap/tablet_mgr_test.cpp | 10 +-
be/test/olap/tablet_test.cpp | 18 +-
be/test/olap/txn_manager_test.cpp | 123 +-
be/test/runtime/load_stream_test.cpp | 33 +-
be/test/runtime/snapshot_loader_test.cpp | 18 +-
be/test/testutil/run_all_tests.cpp | 4 +-
be/test/vec/core/block_spill_test.cpp | 6 +-
be/test/vec/core/block_test.cpp | 2 +-
be/test/vec/exec/delta_writer_v2_pool_test.cpp | 18 +-
be/test/vec/jsonb/serialize_test.cpp | 2 +-
be/test/vec/olap/vertical_compaction_test.cpp | 38 +-
cloud/script/start.sh | 2 +-
cloud/src/common/logging.cpp | 5 +-
conf/fe.conf | 2 +-
.../elasticsearch/config/es6/elasticsearch.yml | 62 +-
.../elasticsearch/config/es6/log4j2.properties | 205 +++
.../elasticsearch/config/es7/log4j2.properties | 299 +++++
.../elasticsearch/config/es8/log4j2.properties | 273 ++++
.../docker-compose/elasticsearch/es.yaml.tpl | 28 +-
.../elasticsearch/scripts/index/es6_test1.json | 2 +-
.../elasticsearch/scripts/index/es6_test2.json | 2 +-
docker/thirdparties/run-thirdparties-docker.sh | 18 +-
.../release-and-verify/release-doris-connectors.md | 19 +-
docs/en/docs/admin-manual/config/be-config.md | 2 +-
docs/en/docs/admin-manual/workload-group.md | 114 +-
docs/en/docs/advanced/partition/auto-partition.md | 2 +-
docs/en/docs/advanced/variables.md | 6 +-
docs/en/docs/ecosystem/flink-doris-connector.md | 6 +-
docs/en/docs/install/k8s-deploy/network.md | 377 ++++++
docs/en/docs/install/k8s-deploy/operator-deploy.md | 2 +-
.../docs/install/k8s-deploy/persistent-volume.md | 424 ++++++
.../docs/install/source-install/compilation-arm.md | 505 ++++----
docs/en/docs/lakehouse/multi-catalog/hive.md | 13 +
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 1 -
.../pipeline-x-execution-engine.md | 76 ++
.../sql-functions/bitwise-functions/bitcount.md} | 29 +-
.../bitwise-functions/bitshfitleft.md | 71 ++
.../bitwise-functions/bitshfitright.md | 79 ++
.../sql-functions/ip-functions/inet-aton.md | 20 +-
.../sql-functions/ip-functions/inet6-aton.md | 30 +-
.../ip-functions/ipv4-cidr-to-range.md | 72 ++
.../ip-functions/ipv6-cidr-to-range.md | 72 ++
.../ip-functions/is-ip-address-in-range.md | 65 +
.../{inet-aton.md => is-ipv4-compat.md} | 44 +-
.../sql-functions/ip-functions/is-ipv4-mapped.md | 69 +
.../sql-functions/ip-functions/is-ipv4-string.md | 80 ++
.../sql-functions/ip-functions/is-ipv6-string.md | 80 ++
.../ip-functions/to-ipv4-or-default.md} | 40 +-
.../sql-functions/ip-functions/to-ipv4-or-null.md} | 46 +-
.../sql-functions/ip-functions/to-ipv4.md} | 41 +-
.../ip-functions/to-ipv6-or-default.md} | 40 +-
.../sql-functions/ip-functions/to-ipv6-or-null.md} | 46 +-
.../sql-functions/ip-functions/to-ipv6.md} | 41 +-
.../sql-functions/string-functions/left.md | 47 -
.../sql-functions/string-functions/strleft.md | 23 +-
.../sql-functions/string-functions/strright.md | 23 +-
.../sql-functions/string-functions/substr.md | 50 -
.../sql-functions/string-functions/substring.md | 6 +-
.../Load/CREATE-ROUTINE-LOAD.md | 2 +-
.../sql-reference/Data-Types/DATETIME.md | 7 +-
.../sql-manual/sql-reference/Data-Types/IPV4.md | 85 ++
.../sql-manual/sql-reference/Data-Types/IPV6.md | 83 ++
.../sql-manual/sql-reference/Data-Types/STRUCT.md | 2 +-
.../sql-manual/sql-reference/Data-Types/VARIANT.md | 8 +-
.../ADMIN-SET-REPLICA-STATUS.md | 17 +-
docs/sidebars.json | 32 +-
.../release-and-verify/release-doris-connectors.md | 19 +-
docs/zh-CN/docs/admin-manual/config/be-config.md | 2 +-
docs/zh-CN/docs/admin-manual/workload-group.md | 108 +-
.../docs/advanced/partition/auto-partition.md | 2 +-
docs/zh-CN/docs/advanced/variables.md | 6 +-
docs/zh-CN/docs/ecosystem/flink-doris-connector.md | 6 +-
docs/zh-CN/docs/install/k8s-deploy/network.md | 387 ++++++
.../docs/install/k8s-deploy/operator-deploy.md | 2 +-
.../docs/install/k8s-deploy/persistent-volume.md | 424 ++++++
.../docs/install/source-install/compilation-arm.md | 492 +++----
docs/zh-CN/docs/lakehouse/multi-catalog/hive.md | 13 +
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 1 -
.../pipeline-x-execution-engine.md | 75 ++
.../right.md => bitwise-functions/bitcount.md} | 29 +-
.../bitwise-functions/bitshiftleft.md | 71 ++
.../bitwise-functions/bitshiftright.md | 78 ++
.../sql-functions/ip-functions/inet-aton.md | 19 +-
.../sql-functions/ip-functions/inet6-aton.md | 29 +-
.../ip-functions/ipv4-cidr-to-range.md | 72 ++
.../ip-functions/ipv6-cidr-to-range.md | 72 ++
.../ip-functions/is-ip-address-in-range.md | 65 +
.../{inet-aton.md => is-ipv4-compat.md} | 44 +-
.../sql-functions/ip-functions/is-ipv4-mapped.md | 68 +
.../sql-functions/ip-functions/is-ipv4-string.md | 80 ++
.../sql-functions/ip-functions/is-ipv6-string.md | 80 ++
.../to-ipv4-or-default.md} | 40 +-
.../substr.md => ip-functions/to-ipv4-or-null.md} | 49 +-
.../substr.md => ip-functions/to-ipv4.md} | 44 +-
.../ip-functions/to-ipv6-or-default.md} | 42 +-
.../substr.md => ip-functions/to-ipv6-or-null.md} | 49 +-
.../left.md => ip-functions/to-ipv6.md} | 41 +-
.../sql-functions/string-functions/strleft.md | 23 +-
.../sql-functions/string-functions/strright.md | 23 +-
.../sql-functions/string-functions/substring.md | 8 +-
.../Load/CREATE-ROUTINE-LOAD.md | 2 +-
.../sql-reference/Data-Types/DATETIME.md | 14 +-
.../sql-manual/sql-reference/Data-Types/IPV4.md | 85 ++
.../sql-manual/sql-reference/Data-Types/IPV6.md | 83 ++
.../sql-manual/sql-reference/Data-Types/VARIANT.md | 8 +-
.../ADMIN-SET-REPLICA-STATUS.md | 16 +-
.../java/org/apache/doris/jdbc/JdbcExecutor.java | 52 +-
.../main/java/org/apache/doris/common/Config.java | 23 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 16 +-
.../java/org/apache/doris/alter/AlterJobV2.java | 4 +-
.../org/apache/doris/alter/IndexChangeJob.java | 5 +-
.../doris/alter/MaterializedViewHandler.java | 2 +-
.../java/org/apache/doris/alter/RollupJobV2.java | 8 +-
.../apache/doris/alter/SchemaChangeHandler.java | 75 +-
.../org/apache/doris/alter/SchemaChangeJobV2.java | 27 +-
.../java/org/apache/doris/alter/SystemHandler.java | 15 +-
.../doris/analysis/AdminSetReplicaStatusStmt.java | 4 +-
.../java/org/apache/doris/analysis/Analyzer.java | 2 +
.../org/apache/doris/analysis/BinaryPredicate.java | 8 +
.../java/org/apache/doris/analysis/CaseExpr.java | 14 +-
.../doris/analysis/CreateMaterializedViewStmt.java | 11 +-
.../org/apache/doris/analysis/CreateTableStmt.java | 4 +-
.../org/apache/doris/analysis/DescriptorTable.java | 7 +
.../org/apache/doris/analysis/DropStatsStmt.java | 19 +
.../main/java/org/apache/doris/analysis/Expr.java | 21 +-
.../org/apache/doris/analysis/LiteralExpr.java | 29 +
.../org/apache/doris/analysis/MatchPredicate.java | 10 +-
.../apache/doris/analysis/NativeInsertStmt.java | 2 +-
.../apache/doris/analysis/ShowBuildIndexStmt.java | 9 +-
.../doris/analysis/ShowReplicaStatusStmt.java | 2 +-
.../org/apache/doris/analysis/StmtRewriter.java | 12 +-
.../java/org/apache/doris/analysis/TypeDef.java | 1 -
.../java/org/apache/doris/backup/RestoreJob.java | 2 +-
.../doris/catalog/BuiltinScalarFunctions.java | 36 +-
.../apache/doris/catalog/CatalogRecycleBin.java | 201 +--
.../java/org/apache/doris/catalog/Database.java | 3 +-
.../doris/catalog/DynamicPartitionProperty.java | 50 +-
.../main/java/org/apache/doris/catalog/Env.java | 112 +-
.../java/org/apache/doris/catalog/EnvFactory.java | 107 +-
.../org/apache/doris/catalog/MetadataViewer.java | 4 +
.../java/org/apache/doris/catalog/OlapTable.java | 17 +-
.../java/org/apache/doris/catalog/Partition.java | 2 +-
.../java/org/apache/doris/catalog/Replica.java | 27 +-
.../apache/doris/catalog/ReplicaAllocation.java | 2 +-
.../org/apache/doris/catalog/TableProperty.java | 5 +-
.../main/java/org/apache/doris/catalog/Tablet.java | 6 +-
.../apache/doris/catalog/TabletInvertedIndex.java | 4 +-
.../doris/catalog/external/ExternalDatabase.java | 20 +-
.../catalog/external/HMSExternalDatabase.java | 13 +-
.../catalog/external/IcebergExternalDatabase.java | 4 +-
.../catalog/external/PaimonExternalDatabase.java | 4 +-
.../org/apache/doris/clone/TabletSchedCtx.java | 14 +-
.../org/apache/doris/clone/TabletScheduler.java | 46 +-
.../doris/cloud/catalog/CloudClusterChecker.java | 474 +++++++
...Env.java => CloudDynamicPartitionProperty.java} | 24 +-
.../org/apache/doris/cloud/catalog/CloudEnv.java | 325 +++++
.../doris/cloud/catalog/CloudEnvFactory.java | 108 ++
.../cloud/common/util/CloudPropertyAnalyzer.java | 50 +
.../cloud/datasource/CloudInternalCatalog.java | 2 +-
.../doris/cloud/system/CloudSystemInfoService.java | 252 ++++
.../transaction/CloudGlobalTransactionMgr.java | 836 ++++++++++++
.../apache/doris/cloud/transaction/TxnUtil.java | 357 ++++++
.../apache/doris/common/proc/ReplicasProcNode.java | 2 +
.../apache/doris/common/proc/TransDbProcDir.java | 4 +-
.../doris/common/proc/TransPartitionProcNode.java | 4 +-
.../org/apache/doris/common/proc/TransProcDir.java | 4 +-
.../doris/common/proc/TransStateProcDir.java | 4 +-
.../doris/common/proc/TransTablesProcDir.java | 4 +-
.../doris/common/profile/SummaryProfile.java | 12 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 45 +-
.../org/apache/doris/datasource/CatalogMgr.java | 259 +---
.../apache/doris/datasource/ExternalCatalog.java | 14 +-
.../apache/doris/datasource/ExternalMetaIdMgr.java | 263 ++++
.../doris/datasource/HMSExternalCatalog.java | 107 +-
.../org/apache/doris/datasource/InitTableLog.java | 67 -
.../apache/doris/datasource/InternalCatalog.java | 31 +-
.../apache/doris/datasource/MetaIdMappingsLog.java | 274 ++++
.../datasource/hive/event/AddPartitionEvent.java | 16 +
.../datasource/hive/event/AlterTableEvent.java | 6 +-
.../datasource/hive/event/CreateDatabaseEvent.java | 12 +
.../datasource/hive/event/CreateTableEvent.java | 14 +-
.../datasource/hive/event/DropDatabaseEvent.java | 11 +
.../datasource/hive/event/DropPartitionEvent.java | 17 +-
.../datasource/hive/event/DropTableEvent.java | 15 +-
.../datasource/hive/event/MetastoreEvent.java | 9 +
.../hive/event/MetastoreEventFactory.java | 26 +-
.../hive/event/MetastoreEventsProcessor.java | 225 +++-
.../iceberg/IcebergHadoopExternalCatalog.java | 15 +-
.../datasource/jdbc/client/JdbcClientConfig.java | 12 +-
.../org/apache/doris/deploy/DeployManager.java | 4 +-
.../httpv2/exception/RestApiExceptionHandler.java | 8 +-
.../doris/httpv2/rest/CheckWalSizeAction.java | 3 +-
.../doris/httpv2/rest/GetStreamLoadState.java | 8 +-
.../org/apache/doris/httpv2/rest/LoadAction.java | 23 +-
.../doris/httpv2/rest/manager/NodeAction.java | 2 +-
.../org/apache/doris/job/base/AbstractJob.java | 15 +-
.../doris/job/extensions/insert/InsertJob.java | 46 +-
.../doris/job/extensions/insert/InsertTask.java | 31 +-
.../org/apache/doris/job/manager/JobManager.java | 23 +-
.../apache/doris/job/scheduler/JobScheduler.java | 6 +-
.../java/org/apache/doris/journal/Journal.java | 5 +
.../org/apache/doris/journal/JournalBatch.java | 81 ++
.../org/apache/doris/journal/JournalEntity.java | 20 +-
.../apache/doris/journal/bdbje/BDBJEJournal.java | 78 ++
.../apache/doris/journal/local/LocalJournal.java | 12 +
.../java/org/apache/doris/load/DeleteInfo.java | 16 +-
.../main/java/org/apache/doris/load/DeleteJob.java | 10 +-
.../main/java/org/apache/doris/load/ExportJob.java | 2 +-
.../org/apache/doris/load/GroupCommitManager.java | 25 +-
.../org/apache/doris/load/loadv2/LoadManager.java | 2 +-
.../org/apache/doris/load/loadv2/SparkLoadJob.java | 2 +-
.../doris/load/routineload/KafkaProgress.java | 9 +
.../load/routineload/KafkaRoutineLoadJob.java | 31 +-
.../doris/load/routineload/KafkaTaskInfo.java | 2 +-
.../routineload/RLTaskTxnCommitAttachment.java | 18 +
.../doris/load/routineload/RoutineLoadJob.java | 25 +-
.../load/routineload/RoutineLoadTaskInfo.java | 2 +-
.../load/routineload/RoutineLoadTaskScheduler.java | 16 +-
.../doris/load/sync/canal/CanalSyncChannel.java | 4 +-
.../java/org/apache/doris/metric/MetricRepo.java | 106 ++
.../main/java/org/apache/doris/mtmv/MTMVCache.java | 4 +-
.../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 1 +
.../org/apache/doris/mtmv/MTMVRelationManager.java | 21 +-
.../doris/nereids/analyzer/UnboundRelation.java | 20 +-
.../doris/nereids/analyzer/UnboundResultSink.java | 5 +
.../doris/nereids/analyzer/UnboundTableSink.java | 6 +
.../glue/translator/RuntimeFilterTranslator.java | 32 +-
.../doris/nereids/jobs/executor/Rewriter.java | 11 +-
.../doris/nereids/jobs/joinorder/JoinOrderJob.java | 7 +-
.../jobs/joinorder/hypergraph/HyperGraph.java | 715 +++++------
.../joinorder/hypergraph/bitmap/LongBitmap.java | 9 +
.../joinorder/hypergraph/node/AbstractNode.java | 40 +-
.../joinorder/hypergraph/node/DPhyperNode.java | 2 +-
.../joinorder/hypergraph/node/StructInfoNode.java | 84 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 19 +-
.../processor/post/RuntimeFilterContext.java | 2 +-
.../processor/post/RuntimeFilterGenerator.java | 16 +-
.../org/apache/doris/nereids/rules/RuleType.java | 6 +-
.../AdjustAggregateNullableForEmptySet.java | 3 +-
.../doris/nereids/rules/analysis/BindRelation.java | 19 +-
.../rules/analysis/EliminateGroupByConstant.java | 2 +-
.../doris/nereids/rules/analysis/SlotBinder.java | 10 +-
.../mv/AbstractMaterializedViewAggregateRule.java | 116 +-
.../mv/AbstractMaterializedViewJoinRule.java | 17 +-
.../mv/AbstractMaterializedViewRule.java | 389 +++---
.../rules/exploration/mv/ComparisonResult.java | 36 +-
.../rules/exploration/mv/HyperGraphComparator.java | 87 +-
.../mv/InitMaterializationContextHook.java | 2 +-
.../mv/MaterializedViewAggregateRule.java | 2 +-
.../nereids/rules/exploration/mv/Predicates.java | 28 +-
.../nereids/rules/exploration/mv/StructInfo.java | 173 ++-
.../nereids/rules/expression/rules/OrToIn.java | 51 +-
.../rules/implementation/AggregateStrategies.java | 50 +-
.../LogicalOlapScanToPhysicalOlapScan.java | 10 +-
.../nereids/rules/rewrite/AdjustNullable.java | 8 +
.../rules/rewrite/ConvertOuterJoinToAntiJoin.java | 6 +-
.../nereids/rules/rewrite/EliminateJoinByFK.java | 319 +----
.../rules/rewrite/EliminateJoinByUnique.java | 10 +-
.../rules/rewrite/FindHashConditionForJoin.java | 9 +-
.../nereids/rules/rewrite/ForeignKeyContext.java | 184 +++
.../nereids/rules/rewrite/MergeSetOperations.java | 22 +-
.../rules/rewrite/MergeSetOperationsExcept.java | 64 +
.../nereids/rules/rewrite/NormalizeToSlot.java | 23 +-
.../rules/rewrite/PushDownMinMaxThroughJoin.java | 17 +-
.../rules/rewrite/PushDownSumThroughJoin.java | 4 +-
.../rewrite/PushDownSumThroughJoinOneSide.java | 117 +-
.../rewrite/PushDownTopNDistinctThroughJoin.java | 126 ++
.../rewrite/PushDownTopNDistinctThroughUnion.java | 8 +-
.../mv/AbstractSelectMaterializedIndexRule.java | 3 +-
.../mv/SelectMaterializedIndexWithAggregate.java | 199 +--
.../doris/nereids/stats/ExpressionEstimation.java | 68 +-
.../doris/nereids/stats/StatsCalculator.java | 44 +-
.../org/apache/doris/nereids/trees/TreeNode.java | 17 -
.../nereids/trees/expressions/Expression.java | 22 +-
.../trees/expressions/functions/agg/Sum.java | 6 +
.../trees/expressions/functions/scalar/Array.java | 70 +-
.../scalar/{TimeToSec.java => BitCount.java} | 41 +-
.../scalar/{TimeToSec.java => BitShiftLeft.java} | 41 +-
.../scalar/{TimeToSec.java => BitShiftRight.java} | 41 +-
.../{IPv6CIDRToRange.java => Ipv4CIDRToRange.java} | 32 +-
.../{IPv6CIDRToRange.java => Ipv6CIDRToRange.java} | 12 +-
.../{IsIpv4String.java => IsIpv4Compat.java} | 20 +-
.../{IsIpv4String.java => IsIpv4Mapped.java} | 20 +-
.../expressions/functions/scalar/IsIpv4String.java | 4 +-
.../expressions/functions/scalar/IsIpv6String.java | 4 +-
.../trees/expressions/functions/scalar/Left.java | 7 +-
.../trees/expressions/functions/scalar/Right.java | 7 +-
.../expressions/functions/scalar/StrLeft.java | 7 +-
.../expressions/functions/scalar/StrRight.java | 7 +-
.../expressions/functions/scalar/Substring.java | 5 +-
.../expressions/functions/scalar/TimeToSec.java | 4 +-
.../scalar/{IsIpv4String.java => ToIpv4.java} | 26 +-
.../{IsIpv4String.java => ToIpv4OrDefault.java} | 26 +-
.../{IsIpv4String.java => ToIpv4OrNull.java} | 22 +-
.../scalar/{IsIpv4String.java => ToIpv6.java} | 26 +-
.../{IsIpv4String.java => ToIpv6OrDefault.java} | 26 +-
.../{IsIpv4String.java => ToIpv6OrNull.java} | 22 +-
.../nereids/trees/expressions/literal/Literal.java | 23 +
.../expressions/visitor/ScalarFunctionVisitor.java | 64 +-
.../trees/plans/commands/UpdateCommand.java | 33 +-
.../plans/commands/info/ColumnDefinition.java | 1 -
.../trees/plans/commands/info/CreateTableInfo.java | 13 +-
.../logical/LogicalDeferMaterializeResultSink.java | 7 +
.../trees/plans/logical/LogicalFileSink.java | 2 +-
.../nereids/trees/plans/logical/LogicalJoin.java | 7 +-
.../trees/plans/logical/LogicalOlapScan.java | 35 +-
.../trees/plans/logical/LogicalOlapTableSink.java | 5 +
.../trees/plans/logical/LogicalProject.java | 4 +
.../trees/plans/logical/LogicalResultSink.java | 5 +
.../nereids/trees/plans/logical/LogicalSink.java | 2 +
.../trees/plans/physical/AbstractPhysicalJoin.java | 15 +
.../trees/plans/physical/AbstractPhysicalPlan.java | 26 +-
.../trees/plans/physical/PhysicalProject.java | 11 +-
.../trees/plans/physical/RuntimeFilter.java | 45 +-
.../apache/doris/nereids/util/ExpressionUtils.java | 24 +-
.../org/apache/doris/nereids/util/JoinUtils.java | 55 +
.../org/apache/doris/nereids/util/PlanUtils.java | 10 +
.../doris/nereids/util/TypeCoercionUtils.java | 3 +-
.../java/org/apache/doris/persist/EditLog.java | 29 +-
.../doris/persist/EditLogFileOutputStream.java | 5 +
.../apache/doris/persist/EditLogOutputStream.java | 2 +
.../org/apache/doris/persist/OperationType.java | 9 +
.../java/org/apache/doris/persist/RecoverInfo.java | 9 +-
.../persist/SetReplicaStatusOperationLog.java | 9 +-
.../org/apache/doris/persist/gson/GsonUtils.java | 20 +
.../persist/gson/RuntimeTypeAdapterFactory.java | 25 +
.../apache/doris/planner/GroupCommitPlanner.java | 3 -
.../org/apache/doris/planner/OlapScanNode.java | 41 +-
.../java/org/apache/doris/planner/PlanNode.java | 3 +
.../org/apache/doris/planner/RuntimeFilter.java | 12 +
.../java/org/apache/doris/planner/ScanNode.java | 10 +-
.../apache/doris/planner/SingleNodePlanner.java | 47 +-
.../doris/planner/external/FileQueryScanNode.java | 12 +
.../main/java/org/apache/doris/qe/Coordinator.java | 85 +-
.../java/org/apache/doris/qe/SessionVariable.java | 13 +-
.../java/org/apache/doris/qe/ShowExecutor.java | 4 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 2 +-
.../main/java/org/apache/doris/qe/VariableMgr.java | 20 +
.../org/apache/doris/rewrite/CaseWhenToIf.java | 52 +
.../java/org/apache/doris/rewrite/FEFunctions.java | 6 +-
.../org/apache/doris/rewrite/FunctionAlias.java | 4 +-
.../apache/doris/service/FrontendServiceImpl.java | 36 +-
.../apache/doris/statistics/AnalysisManager.java | 72 +-
.../statistics/ColumnStatisticsCacheLoader.java | 29 +-
.../InvalidateStatsTarget.java} | 34 +-
.../doris/statistics/StatisticsAutoCollector.java | 2 +-
.../apache/doris/statistics/StatisticsCache.java | 17 +-
.../apache/doris/statistics/TableStatsMeta.java | 2 +-
.../main/java/org/apache/doris/system/Backend.java | 8 +
.../java/org/apache/doris/system/Diagnoser.java | 5 +-
.../apache/doris/system/FrontendHbResponse.java | 1 -
.../org/apache/doris/system/SystemInfoService.java | 14 +-
.../org/apache/doris/task/AlterReplicaTask.java | 15 +-
.../transaction/DbUsedDataQuotaInfoCollector.java | 2 +-
.../doris/transaction/GlobalTransactionMgr.java | 430 ++++---
.../transaction/GlobalTransactionMgrIface.java | 184 +++
.../doris/transaction/PublishVersionDaemon.java | 2 +-
.../apache/doris/transaction/TransactionState.java | 19 +-
.../main/resources/retrying_service_config.json | 20 +
.../org/apache/doris/alter/RollupJobV2Test.java | 6 +-
.../apache/doris/alter/SchemaChangeJobV2Test.java | 4 +-
.../apache/doris/analysis/CreateTableStmtTest.java | 12 +-
.../doris/analysis/ShowBuildIndexStmtTest.java | 239 ++++
.../analysis/ShowCreateMaterializedViewTest.java | 93 ++
.../org/apache/doris/binlog/BinlogManagerTest.java | 2 +-
.../org/apache/doris/catalog/AdminStmtTest.java | 3 +-
.../org/apache/doris/catalog/EnvFactoryTest.java | 31 +-
.../doris/cloud/catalog/CloudEnvFactoryTest.java | 52 +
.../transaction/CloudGlobalTransactionMgrTest.java | 407 ++++++
.../doris/datasource/ExternalMetaIdMgrTest.java | 76 ++
.../doris/datasource/MetaIdMappingsLogTest.java | 97 ++
.../external/hms/MetastoreEventFactoryTest.java | 2 +-
.../doris/journal/bdbje/BDBJEJournalTest.java | 97 ++
.../joinorder/hypergraph/CompareOuterJoinTest.java | 29 +-
.../jobs/joinorder/hypergraph/InferJoinTest.java | 16 +-
.../joinorder/hypergraph/InferPredicateTest.java | 4 +-
.../joinorder/hypergraph/PullupExpressionTest.java | 16 +-
.../nereids/postprocess/RuntimeFilterTest.java | 4 +-
.../rules/exploration/mv/BuildStructInfoTest.java | 8 +-
...hComparatorTest.java => EliminateJoinTest.java} | 114 +-
.../rules/exploration/mv/HyperGraphAggTest.java | 6 +-
.../exploration/mv/HyperGraphComparatorTest.java | 16 +-
.../rewrite/ConvertOuterJoinToAntiJoinTest.java | 82 +-
.../doris/nereids/rules/rewrite/OrToInTest.java | 15 +
.../rules/rewrite/mv/SelectMvIndexTest.java | 89 ++
.../rules/rewrite/mv/SelectRollupIndexTest.java | 2 +-
.../nereids/stats/ExpressionEstimationTest.java | 63 +
.../doris/nereids/util/HyperGraphBuilder.java | 4 +-
.../org/apache/doris/rewrite/FEFunctionsTest.java | 19 -
.../doris/statistics/AnalysisManagerTest.java | 7 +-
.../statistics/StatisticsAutoCollectorTest.java | 4 +-
.../transaction/DatabaseTransactionMgrTest.java | 4 +-
.../transaction/GlobalTransactionMgrTest.java | 4 +-
fe/pom.xml | 2 +-
gensrc/proto/internal_service.proto | 6 +-
gensrc/script/doris_builtins_functions.py | 77 +-
gensrc/thrift/FrontendService.thrift | 1 +
gensrc/thrift/PaloInternalService.thrift | 18 +-
gensrc/thrift/PlanNodes.thrift | 5 +
gensrc/thrift/Status.thrift | 1 +
.../data/correctness/test_time_function.out | 9 +-
.../data/correctness_p0/test_bit_function.out | 132 ++
.../data/correctness_p0/test_bit_shift_lagency.out | 345 +++++
.../data/correctness_p0/test_bit_shift_nereids.out | 345 +++++
.../correctness_p0/test_nested_lateral_view.out | 15 +
.../test_predicate_with_implicit_cast.out | 19 +
.../data/external_table_p0/es/test_es_query.out | 42 +-
.../es/test_es_query_no_http_url.out | 2 +-
.../paimon/paimon_timestamp_types.out | 13 +
.../test_index_compaction_null.out | 1345 ++++++++++++++++++++
.../test_index_match_phrase_prefix_1.out} | 13 +-
...ll.out => test_inverted_index_null_ram_dir.out} | 0
.../count_star.out} | 35 +-
regression-test/data/mv_p0/sum_count/sum_count.out | 61 +
.../data/mv_p0/test_casewhen/test_casewhen.out | 11 +
.../testAggQueryOnAggMV1/testAggQueryOnAggMV1.out | 6 +
.../data/nereids_function_p0/ip_functions.out | 68 +-
.../data/nereids_function_p0/scalar_function/R.out | 4 +-
.../data/nereids_function_p0/scalar_function/S.out | 4 +-
.../runtime_filter/expr-target.out} | 5 +-
.../data/nereids_p0/set_operations/except.out | 148 +++
.../nereids_p0/show/test_show_create_catalog.out | 4 -
.../limit_push_down/order_push_down.out | 22 +-
.../mv/agg_with_roll_up/aggregate_with_roll_up.out | 14 +
.../aggregate_without_roll_up.out | 24 +
.../mv/join/left_outer/outer_join.out | 8 +
.../data/nereids_rules_p0/pkfk/eliminate_inner.out | 251 ++--
.../push_down_top_n_distinct_through_join.out | 47 +
.../push_down_top_n_distinct_through_union.out | 138 +-
.../data/nereids_syntax_p0/agg_with_empty_set.out | 3 +
.../shape/query13.out | 54 +-
.../shape/query15.out | 2 +-
.../shape/query48.out | 2 +-
.../shape/query49.out | 162 +--
.../shape/query54.out | 112 +-
.../nereids_tpcds_shape_sf1000_p0/shape/query6.out | 68 +-
.../shape/query85.out | 65 +-
.../noStatsRfPrune/query13.out | 2 +-
.../noStatsRfPrune/query15.out | 2 +-
.../noStatsRfPrune/query48.out | 2 +-
.../noStatsRfPrune/query49.out | 168 +--
.../noStatsRfPrune/query58.out | 21 +-
.../noStatsRfPrune/query64.out | 4 +-
.../noStatsRfPrune/query85.out | 2 +-
.../no_stats_shape/query13.out | 2 +-
.../no_stats_shape/query15.out | 2 +-
.../no_stats_shape/query48.out | 2 +-
.../no_stats_shape/query49.out | 168 +--
.../no_stats_shape/query58.out | 21 +-
.../no_stats_shape/query64.out | 4 +-
.../no_stats_shape/query85.out | 2 +-
.../rf_prune/query13.out | 2 +-
.../rf_prune/query15.out | 2 +-
.../rf_prune/query48.out | 2 +-
.../rf_prune/query49.out | 162 +--
.../rf_prune/query54.out | 120 +-
.../rf_prune/query6.out | 7 +-
.../rf_prune/query64.out | 20 +-
.../rf_prune/query85.out | 58 +-
.../nereids_tpcds_shape_sf100_p0/shape/query13.out | 2 +-
.../nereids_tpcds_shape_sf100_p0/shape/query15.out | 2 +-
.../nereids_tpcds_shape_sf100_p0/shape/query48.out | 2 +-
.../nereids_tpcds_shape_sf100_p0/shape/query49.out | 162 +--
.../nereids_tpcds_shape_sf100_p0/shape/query54.out | 120 +-
.../nereids_tpcds_shape_sf100_p0/shape/query6.out | 9 +-
.../nereids_tpcds_shape_sf100_p0/shape/query64.out | 30 +-
.../nereids_tpcds_shape_sf100_p0/shape/query85.out | 58 +-
.../query_p0/show/test_show_create_catalog.out | 4 -
.../ip_functions/test_ip_functions.out | 68 +-
.../ip_functions/test_ipv4_cidr_to_range.out | 28 +
.../test_ipv6_cidr_to_range_function.out | 4 +
.../ip_functions/test_to_ip_functions.out | 21 +
.../string_functions/test_string_function.out | Bin 3418 -> 4134 bytes
.../data/schema_change_p0/test_partition.out | 14 +
.../test_schema_change_agg_check_all_types.out | 34 +
regression-test/data/statistics/test_select_mv.out | 47 +
.../data/variant_p0/sql/implicit_cast.out | 80 ++
.../doris/regression/action/WaitForAction.groovy | 70 +
.../org/apache/doris/regression/suite/Suite.groovy | 5 +
.../framework/src/main/groovy/suite.gdsl | 4 +-
regression-test/pipeline/common/doris-utils.sh | 15 +-
regression-test/pipeline/common/github-utils.sh | 39 +-
regression-test/pipeline/common/teamcity-utils.sh | 2 +
regression-test/pipeline/performance/run-tpcds.sh | 1 +
regression-test/pipeline/performance/run-tpch.sh | 1 +
.../pipeline/tpch/tpch-sf100/conf/be.conf | 85 --
.../pipeline/tpch/tpch-sf100/conf/external.json | 26 -
.../pipeline/tpch/tpch-sf100/conf/fe.conf | 78 --
.../pipeline/tpch/tpch-sf100/conf/odbcinst.ini | 43 -
regression-test/pipeline/tpch/tpch-sf100/deploy.sh | 119 --
.../pipeline/tpch/tpch-sf100/prepare.sh | 69 -
regression-test/pipeline/tpch/tpch-sf100/run.sh | 131 --
.../suites/correctness/test_time_function.groovy | 9 +
.../suites/correctness_p0/test_bit_function.groovy | 63 +
.../correctness_p0/test_bit_shift_lagency.groovy | 105 ++
.../correctness_p0/test_bit_shift_nereids.groovy | 105 ++
.../correctness_p0/test_nested_lateral_view.groovy | 55 +
.../test_predicate_with_implicit_cast.groovy | 71 ++
.../datatype_p0/string/test_string_basic.groovy | 29 +
.../suites/export/test_struct_export.groovy | 2 +-
.../paimon/paimon_timestamp_types.groovy | 55 +
.../test_index_compaction_fault_injection.groovy | 8 +
.../test_index_compaction_null.groovy} | 192 ++-
.../test_index_match_phrase_prefix_1.groovy | 62 +
.../test_inverted_index_null_ram_dir.groovy | 147 +++
.../suites/inverted_index_p0/test_show_data.groovy | 14 +-
.../suites/job_p0/test_base_insert_job.groovy | 24 +-
.../load_p0/broker_load/ddl/agg_tbl_basic.sql | 4 +-
.../load_p0/broker_load/ddl/dup_tbl_basic.sql | 2 +-
.../load_p0/broker_load/ddl/mow_tbl_basic.sql | 2 +-
.../load_p0/broker_load/ddl/uniq_tbl_basic.sql | 2 +-
.../load_p0/broker_load/test_seq_load.groovy | 2 +-
.../routine_load/data/invalid_json_path.json | 0
.../routine_load/ddl/agg_tbl_basic_create.sql | 4 +-
.../routine_load/ddl/dup_tbl_basic_create.sql | 2 +-
.../routine_load/ddl/mow_tbl_basic_create.sql | 2 +-
.../routine_load/ddl/uniq_tbl_basic_create.sql | 2 +-
.../ddl/uniq_tbl_basic_create_sequence.sql | 2 +-
.../routine_load/test_routine_load_error.groovy | 200 +++
.../stream_load/ddl/agg_tbl_basic_create.sql | 4 +-
.../stream_load/ddl/dup_tbl_basic_create.sql | 2 +-
.../ddl/dup_tbl_basic_create_random_bucket.sql | 2 +-
.../stream_load/ddl/mow_tbl_basic_create.sql | 2 +-
.../stream_load/ddl/uniq_tbl_basic_create.sql | 2 +-
.../ddl/uniq_tbl_basic_create_sequence.sql | 2 +-
.../suites/load_p0/tvf/ddl/agg_tbl_basic_tvf.sql | 4 +-
.../suites/load_p0/tvf/ddl/dup_tbl_basic_tvf.sql | 2 +-
.../suites/load_p0/tvf/ddl/mow_tbl_basic_tvf.sql | 2 +-
.../suites/load_p0/tvf/ddl/uniq_tbl_basic_tvf.sql | 2 +-
.../suites/mv_p0/count_star/count_star.groovy | 77 ++
.../suites/mv_p0/sum_count/sum_count.groovy | 92 ++
.../mv_p0/test_casewhen/test_casewhen.groovy | 41 +
.../testAggQueryOnAggMV1.groovy | 14 +-
.../suites/nereids_function_p0/ip_functions.groovy | 24 +
.../nereids_p0/runtime_filter/expr-target.groovy | 106 ++
.../suites/nereids_p0/set_operations/except.groovy | 126 ++
.../window_functions/test_window_function.groovy | 4 +
.../nereids_p0/update/update_unique_table.groovy | 14 +-
.../limit_push_down/order_push_down.groovy | 1 +
.../agg_with_roll_up/aggregate_with_roll_up.groovy | 177 ++-
.../aggregate_without_roll_up.groovy | 135 +-
.../mv/join/dphyp_outer/outer_join_dphyp.groovy | 2 +-
.../mv/join/left_outer/outer_join.groovy | 118 +-
.../nereids_rules_p0/pkfk/eliminate_inner.groovy | 16 +-
.../push_down_top_n_distinct_through_join.groovy | 68 +
.../nereids_syntax_p0/agg_with_empty_set.groovy | 1 +
.../test_partition_create_tablet_rr.groovy | 162 +++
.../auto_partition/ddl/stress_destination.sql | 20 +
.../auto_partition/ddl/stress_source.sql | 19 +
.../doris_dbgen_conf/stress_test_insert_into.yaml} | 8 +-
.../auto_partition/sql/multi_thread_load.groovy | 3 +-
.../sql/stress_test_insert_into.groovy | 118 ++
.../suites/query_p0/cross_db/cross_db.groovy | 62 +
.../ip_functions/test_ip_functions.groovy | 24 +
.../ip_functions/test_ipv4_cidr_to_range.groovy | 58 +
.../test_ipv6_cidr_to_range_function.groovy | 10 +-
.../ip_functions/test_to_ip_functions.groovy | 49 +
.../string_functions/test_string_function.groovy | 85 +-
.../suites/query_p0/test_row_policy.groovy | 42 +
.../test_alter_table_column.groovy | 99 +-
.../suites/schema_change_p0/test_partition.groovy | 68 +
.../schema_change_p0/test_schema_change_agg.groovy | 28 +
.../test_schema_change_agg_check_all_types.groovy | 215 ++++
.../suites/show_p0/test_show_delete.groovy | 67 +
.../suites/show_p0/test_show_statistic_proc.groovy | 1 +
.../suites/statistics/analyze_stats.groovy | 2 +-
.../suites/statistics/test_select_mv.groovy | 97 ++
.../test_set_replica_drop.groovy | 84 ++
.../suites/variant_p0/sql/implicit_cast.sql | 15 +
thirdparty/build-thirdparty.sh | 3 +-
thirdparty/download-thirdparty.sh | 11 +
thirdparty/patches/ali-sdk-1.36.1586.patch | 62 +
thirdparty/patches/glog-0.6.0.patch | 93 +-
884 files changed, 28161 insertions(+), 10135 deletions(-)
create mode 100644 be/src/cloud/cloud_storage_engine.cpp
create mode 100644 be/src/cloud/cloud_tablet_mgr.cpp
create mode 100644 be/src/cloud/cloud_tablet_mgr.h
copy be/src/vec/common/hash_table/{hash_map.h => join_hash_table.h} (53%)
create mode 100644 be/src/vec/functions/function_bit_count.cpp
create mode 100644 be/src/vec/functions/function_bit_shift.cpp
rename regression-test/pipeline/tpch/tpch-sf100/clean.sh => docker/thirdparties/docker-compose/elasticsearch/config/es6/elasticsearch.yml (51%)
mode change 100644 => 100755
create mode 100755 docker/thirdparties/docker-compose/elasticsearch/config/es6/log4j2.properties
create mode 100755 docker/thirdparties/docker-compose/elasticsearch/config/es7/log4j2.properties
create mode 100755 docker/thirdparties/docker-compose/elasticsearch/config/es8/log4j2.properties
create mode 100644 docs/en/docs/install/k8s-deploy/network.md
create mode 100644 docs/en/docs/install/k8s-deploy/persistent-volume.md
create mode 100644 docs/en/docs/query-acceleration/pipeline-x-execution-engine.md
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/left.md => en/docs/sql-manual/sql-functions/bitwise-functions/bitcount.md} (64%)
create mode 100644 docs/en/docs/sql-manual/sql-functions/bitwise-functions/bitshfitleft.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/bitwise-functions/bitshfitright.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/ip-functions/ipv4-cidr-to-range.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/ip-functions/ipv6-cidr-to-range.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/ip-functions/is-ip-address-in-range.md
copy docs/en/docs/sql-manual/sql-functions/ip-functions/{inet-aton.md => is-ipv4-compat.md} (53%)
create mode 100644 docs/en/docs/sql-manual/sql-functions/ip-functions/is-ipv4-mapped.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/ip-functions/is-ipv4-string.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/ip-functions/is-ipv6-string.md
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/right.md => en/docs/sql-manual/sql-functions/ip-functions/to-ipv4-or-default.md} (59%)
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/strright.md => en/docs/sql-manual/sql-functions/ip-functions/to-ipv4-or-null.md} (57%)
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/left.md => en/docs/sql-manual/sql-functions/ip-functions/to-ipv4.md} (56%)
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/right.md => en/docs/sql-manual/sql-functions/ip-functions/to-ipv6-or-default.md} (59%)
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/strright.md => en/docs/sql-manual/sql-functions/ip-functions/to-ipv6-or-null.md} (57%)
copy docs/{zh-CN/docs/sql-manual/sql-functions/string-functions/left.md => en/docs/sql-manual/sql-functions/ip-functions/to-ipv6.md} (61%)
delete mode 100644 docs/en/docs/sql-manual/sql-functions/string-functions/left.md
delete mode 100644 docs/en/docs/sql-manual/sql-functions/string-functions/substr.md
create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Types/IPV4.md
create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Types/IPV6.md
create mode 100644 docs/zh-CN/docs/install/k8s-deploy/network.md
create mode 100644 docs/zh-CN/docs/install/k8s-deploy/persistent-volume.md
create mode 100644 docs/zh-CN/docs/query-acceleration/pipeline-x-execution-engine.md
rename docs/zh-CN/docs/sql-manual/sql-functions/{string-functions/right.md => bitwise-functions/bitcount.md} (65%)
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/bitwise-functions/bitshiftleft.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/bitwise-functions/bitshiftright.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/ipv4-cidr-to-range.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/ipv6-cidr-to-range.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/is-ip-address-in-range.md
copy docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/{inet-aton.md => is-ipv4-compat.md} (53%)
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/is-ipv4-mapped.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/is-ipv4-string.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/ip-functions/is-ipv6-string.md
copy docs/zh-CN/docs/sql-manual/sql-functions/{string-functions/strright.md => ip-functions/to-ipv4-or-default.md} (59%)
copy docs/zh-CN/docs/sql-manual/sql-functions/{string-functions/substr.md => ip-functions/to-ipv4-or-null.md} (57%)
copy docs/zh-CN/docs/sql-manual/sql-functions/{string-functions/substr.md => ip-functions/to-ipv4.md} (57%)
rename docs/{en/docs/sql-manual/sql-functions/string-functions/right.md => zh-CN/docs/sql-manual/sql-functions/ip-functions/to-ipv6-or-default.md} (59%)
rename docs/zh-CN/docs/sql-manual/sql-functions/{string-functions/substr.md => ip-functions/to-ipv6-or-null.md} (57%)
rename docs/zh-CN/docs/sql-manual/sql-functions/{string-functions/left.md => ip-functions/to-ipv6.md} (59%)
create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Types/IPV4.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Types/IPV6.md
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudClusterChecker.java
copy fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/{CloudEnv.java => CloudDynamicPartitionProperty.java} (55%)
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudEnvFactory.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/common/util/CloudPropertyAnalyzer.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/system/CloudSystemInfoService.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/TxnUtil.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/InitTableLog.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/MetaIdMappingsLog.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/journal/JournalBatch.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ForeignKeyContext.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergeSetOperationsExcept.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownTopNDistinctThroughJoin.java
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{TimeToSec.java => BitCount.java} (67%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{TimeToSec.java => BitShiftLeft.java} (66%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{TimeToSec.java => BitShiftRight.java} (66%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IPv6CIDRToRange.java => Ipv4CIDRToRange.java} (66%)
rename fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IPv6CIDRToRange.java => Ipv6CIDRToRange.java} (89%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => IsIpv4Compat.java} (81%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => IsIpv4Mapped.java} (81%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => ToIpv4.java} (75%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => ToIpv4OrDefault.java} (74%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => ToIpv4OrNull.java} (76%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => ToIpv6.java} (75%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => ToIpv6OrDefault.java} (74%)
copy fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{IsIpv4String.java => ToIpv6OrNull.java} (76%)
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/rewrite/CaseWhenToIf.java
copy fe/fe-core/src/main/java/org/apache/doris/{cloud/catalog/CloudEnv.java => statistics/InvalidateStatsTarget.java} (52%)
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgrIface.java
create mode 100644 fe/fe-core/src/main/resources/retrying_service_config.json
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/analysis/ShowBuildIndexStmtTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/analysis/ShowCreateMaterializedViewTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/cloud/catalog/CloudEnvFactoryTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgrTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaIdMgrTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/MetaIdMappingsLogTest.java
copy fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/mv/{HyperGraphComparatorTest.java => EliminateJoinTest.java} (57%)
create mode 100644 regression-test/data/correctness_p0/test_bit_shift_lagency.out
create mode 100644 regression-test/data/correctness_p0/test_bit_shift_nereids.out
create mode 100644 regression-test/data/correctness_p0/test_nested_lateral_view.out
create mode 100644 regression-test/data/datatype_p0/decimalv3/test_predicate_with_implicit_cast.out
create mode 100644 regression-test/data/external_table_p2/paimon/paimon_timestamp_types.out
create mode 100644 regression-test/data/inverted_index_p0/index_compaction/test_index_compaction_null.out
copy regression-test/data/{correctness_p0/test_bit_function.out => inverted_index_p0/test_index_match_phrase_prefix_1.out} (63%)
copy regression-test/data/inverted_index_p0/{test_inverted_index_null.out => test_inverted_index_null_ram_dir.out} (100%)
copy regression-test/data/mv_p0/{ut/testAggQueryOnAggMV1/testAggQueryOnAggMV1.out => count_star/count_star.out} (55%)
create mode 100644 regression-test/data/mv_p0/sum_count/sum_count.out
create mode 100644 regression-test/data/mv_p0/test_casewhen/test_casewhen.out
copy regression-test/data/{correctness_p0/test_bit_function.out => nereids_p0/runtime_filter/expr-target.out} (71%)
create mode 100644 regression-test/data/nereids_p0/set_operations/except.out
delete mode 100644 regression-test/data/nereids_p0/show/test_show_create_catalog.out
create mode 100644 regression-test/data/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.out
delete mode 100644 regression-test/data/query_p0/show/test_show_create_catalog.out
create mode 100644 regression-test/data/query_p0/sql_functions/ip_functions/test_ipv4_cidr_to_range.out
create mode 100644 regression-test/data/query_p0/sql_functions/ip_functions/test_to_ip_functions.out
create mode 100644 regression-test/data/schema_change_p0/test_partition.out
create mode 100644 regression-test/data/schema_change_p0/test_schema_change_agg_check_all_types.out
create mode 100644 regression-test/data/statistics/test_select_mv.out
create mode 100644 regression-test/data/variant_p0/sql/implicit_cast.out
create mode 100644 regression-test/framework/src/main/groovy/org/apache/doris/regression/action/WaitForAction.groovy
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/conf/be.conf
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/conf/external.json
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/conf/fe.conf
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/conf/odbcinst.ini
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/deploy.sh
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/prepare.sh
delete mode 100644 regression-test/pipeline/tpch/tpch-sf100/run.sh
create mode 100644 regression-test/suites/correctness_p0/test_bit_shift_lagency.groovy
create mode 100644 regression-test/suites/correctness_p0/test_bit_shift_nereids.groovy
create mode 100644 regression-test/suites/correctness_p0/test_nested_lateral_view.groovy
create mode 100644 regression-test/suites/datatype_p0/decimalv3/test_predicate_with_implicit_cast.groovy
create mode 100644 regression-test/suites/external_table_p2/paimon/paimon_timestamp_types.groovy
copy regression-test/suites/{fault_injection_p0/test_index_compaction_fault_injection.groovy => inverted_index_p0/index_compaction/test_index_compaction_null.groovy} (62%)
create mode 100644 regression-test/suites/inverted_index_p0/test_index_match_phrase_prefix_1.groovy
create mode 100644 regression-test/suites/inverted_index_p0/test_inverted_index_null_ram_dir.groovy
copy docker/thirdparties/docker-compose/kafka/scripts/basic_data_json_by_line.json => regression-test/suites/load_p0/routine_load/data/invalid_json_path.json (100%)
create mode 100644 regression-test/suites/mv_p0/count_star/count_star.groovy
create mode 100644 regression-test/suites/mv_p0/sum_count/sum_count.groovy
create mode 100644 regression-test/suites/mv_p0/test_casewhen/test_casewhen.groovy
create mode 100644 regression-test/suites/nereids_p0/runtime_filter/expr-target.groovy
create mode 100644 regression-test/suites/nereids_p0/set_operations/except.groovy
create mode 100644 regression-test/suites/nereids_rules_p0/push_down_top_n/push_down_top_n_distinct_through_join.groovy
create mode 100644 regression-test/suites/partition_p0/test_partition_create_tablet_rr.groovy
create mode 100644 regression-test/suites/partition_p1/auto_partition/ddl/stress_destination.sql
create mode 100644 regression-test/suites/partition_p1/auto_partition/ddl/stress_source.sql
copy regression-test/suites/{correctness_p0/test_bit_function.groovy => partition_p1/auto_partition/doris_dbgen_conf/stress_test_insert_into.yaml} (81%)
create mode 100644 regression-test/suites/partition_p1/auto_partition/sql/stress_test_insert_into.groovy
create mode 100644 regression-test/suites/query_p0/cross_db/cross_db.groovy
create mode 100644 regression-test/suites/query_p0/sql_functions/ip_functions/test_ipv4_cidr_to_range.groovy
create mode 100644 regression-test/suites/query_p0/sql_functions/ip_functions/test_to_ip_functions.groovy
create mode 100644 regression-test/suites/query_p0/test_row_policy.groovy
create mode 100644 regression-test/suites/schema_change_p0/test_partition.groovy
create mode 100644 regression-test/suites/schema_change_p0/test_schema_change_agg_check_all_types.groovy
create mode 100644 regression-test/suites/show_p0/test_show_delete.groovy
create mode 100644 regression-test/suites/statistics/test_select_mv.groovy
create mode 100644 regression-test/suites/tablet_scheduler_p2/test_set_replica_drop.groovy
create mode 100644 regression-test/suites/variant_p0/sql/implicit_cast.sql
create mode 100644 thirdparty/patches/ali-sdk-1.36.1586.patch
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/05: [fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30003)
Posted by zy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch doris-jdk17
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1c2dd126deb5a17a80ca97e6d33bc6ca3dce1711
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Wed Jan 17 12:36:12 2024 +0800
[fix](JDK17) It will report an exception whenwe start BE with JDK17 and query AVRO table : InaccessibleObjectException (#30003)
---
bin/start_be.sh | 7 ++++++-
conf/be.conf | 3 +++
regression-test/pipeline/p0/conf/be.conf | 3 +++
3 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index 046be07e36e..16258dde48d 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -306,7 +306,12 @@ CUR_DATE=$(date +%Y%m%d-%H%M%S)
LOG_PATH="-DlogPath=${DORIS_HOME}/log/jni.log"
COMMON_OPTS="-Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
-if [[ "${java_version}" -gt 8 ]]; then
+if [[ "${java_version}" -gt 16 ]]; then
+ if [[ -z ${JAVA_OPTS_FOR_JDK_17} ]]; then
+ JAVA_OPTS_FOR_JDK_17="-Xmx1024m -XX:+UseZGC ${LOG_PATH} -Xlog:gc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} ${COMMON_OPTS} --add-opens=java.base/java.net=ALL-UNNAMED"
+ fi
+ final_java_opt="${JAVA_OPTS_FOR_JDK_17}"
+elif [[ "${java_version}" -gt 8 ]]; then
if [[ -z ${JAVA_OPTS_FOR_JDK_9} ]]; then
JAVA_OPTS_FOR_JDK_9="-Xmx1024m ${LOG_PATH} -Xlog:gc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} ${COMMON_OPTS}"
fi
diff --git a/conf/be.conf b/conf/be.conf
index 684830f2aa7..a3407ff1021 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -24,6 +24,9 @@ JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/b
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives --add-opens=java.base/java.net=ALL-UNNAMED"
+
# since 1.2, the JAVA_HOME need to be set to run BE process.
# JAVA_HOME=/path/to/jdk/
diff --git a/regression-test/pipeline/p0/conf/be.conf b/regression-test/pipeline/p0/conf/be.conf
index d562e864a84..15f19ec4f42 100644
--- a/regression-test/pipeline/p0/conf/be.conf
+++ b/regression-test/pipeline/p0/conf/be.conf
@@ -17,6 +17,9 @@
PPROF_TMPDIR="$DORIS_HOME/log/"
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives --add-opens=java.base/java.net=ALL-UNNAMED"
+
# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/05: [fix](JDK17) The objects stored in `PriorityQueue` must implement the `Comparable` interface (#30050)
Posted by zy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch doris-jdk17
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f7946fb7a33a5a6f2d7af4da1d5303d012756b04
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Mon Jan 22 20:44:26 2024 +0800
[fix](JDK17) The objects stored in `PriorityQueue` must implement the `Comparable` interface (#30050)
---
.../org/apache/doris/resource/workloadgroup/QueueToken.java | 10 +++++++++-
.../array_functions/test_array_zip_array_enumerate_uniq.groovy | 2 +-
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueueToken.java b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueueToken.java
index 17299d3ea3b..189ba77e8de 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueueToken.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/QueueToken.java
@@ -28,9 +28,14 @@ import java.util.concurrent.locks.ReentrantLock;
// used to mark QueryQueue offer result
// if offer failed, then need to cancel query
// and return failed reason to user client
-public class QueueToken {
+public class QueueToken implements Comparable<QueueToken> {
private static final Logger LOG = LogManager.getLogger(QueueToken.class);
+ @Override
+ public int compareTo(QueueToken other) {
+ return Long.compare(this.tokenId, other.getTokenId());
+ }
+
enum TokenState {
ENQUEUE_SUCCESS,
READY_TO_RUN
@@ -136,4 +141,7 @@ public class QueueToken {
return tokenId == other.tokenId;
}
+ public long getTokenId() {
+ return tokenId;
+ }
}
diff --git a/regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_zip_array_enumerate_uniq.groovy b/regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_zip_array_enumerate_uniq.groovy
index b8207c1ee76..38772921ece 100644
--- a/regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_zip_array_enumerate_uniq.groovy
+++ b/regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_zip_array_enumerate_uniq.groovy
@@ -25,7 +25,7 @@ suite("test_array_zip_array_enumerate_uniq", "p0") {
SELECT array_zip();
"""
} catch (Exception ex) {
- assertTrue(ex.getMessage().contains("errCode = 2, detailMessage = Unexpected exception: 0"))
+ assertTrue(ex.getMessage().contains("errCode = 2, detailMessage = Unexpected exception: "))
}
try {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/05: [Fix](JDK17) Fix start_be.sh (#30300)
Posted by zy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch doris-jdk17
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2afa2cdec351de804e1589d1adcb02c2b399e0cc
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Fri Jan 26 13:03:24 2024 +0800
[Fix](JDK17) Fix start_be.sh (#30300)
---
bin/start_be.sh | 2 +-
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java | 2 +-
regression-test/pipeline/external/conf/regression-conf.groovy | 6 ++++--
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index 16258dde48d..1aed3e746b1 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -308,7 +308,7 @@ COMMON_OPTS="-Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
if [[ "${java_version}" -gt 16 ]]; then
if [[ -z ${JAVA_OPTS_FOR_JDK_17} ]]; then
- JAVA_OPTS_FOR_JDK_17="-Xmx1024m -XX:+UseZGC ${LOG_PATH} -Xlog:gc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} ${COMMON_OPTS} --add-opens=java.base/java.net=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Xmx1024m ${LOG_PATH} -Xlog:gc:${DORIS_HOME}/log/be.gc.log.${CUR_DATE} ${COMMON_OPTS} --add-opens=java.base/java.net=ALL-UNNAMED"
fi
final_java_opt="${JAVA_OPTS_FOR_JDK_17}"
elif [[ "${java_version}" -gt 8 ]]; then
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
index 0ae513d1db6..0f3e30591c0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
@@ -394,7 +394,7 @@ public class ExportJob implements Writable {
}
/**
- * Generate outfile select stmt
+ * Generate select outfile stmtement
* @throws UserException
*/
private void generateQueryStmt() throws UserException {
diff --git a/regression-test/pipeline/external/conf/regression-conf.groovy b/regression-test/pipeline/external/conf/regression-conf.groovy
index 99a8666fbc9..432c836a9e5 100644
--- a/regression-test/pipeline/external/conf/regression-conf.groovy
+++ b/regression-test/pipeline/external/conf/regression-conf.groovy
@@ -55,9 +55,9 @@ testDirectories = ""
// this groups will not be executed
excludeGroups = ""
// this suites will not be executed
-excludeSuites = "test_cast_string_to_array,test_broker_load,test_spark_load,test_analyze_stats_p1,test_refresh_mtmv"
+excludeSuites = "test_cast_string_to_array,test_broker_load,test_spark_load,test_analyze_stats_p1,test_refresh_mtmv,test_catalog_hive_orc,test_catalog_hive_parquet,test_information_schema_external"
// this directories will not be executed
-excludeDirectories = ""
+excludeDirectories = "workload_manager_p1,nereids_rules_p0/subquery"
customConf1 = "test_custom_conf_value"
@@ -80,6 +80,7 @@ pg_14_port=7121
oracle_11_port=1521
sqlserver_2022_port=1433
clickhouse_22_port=8123
+mariadb_10_port=3326
// hive catalog test config
// To enable jdbc test, you need first start hive container.
@@ -87,6 +88,7 @@ clickhouse_22_port=8123
enableHiveTest=true
hms_port=7141
hive_pg_port=5432
+hiveServerPort=10000
enableEsTest=true
es_6_port="19200/"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/05: [Fix](JDK) Start Fe with JDK17 (#29658)
Posted by zy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch doris-jdk17
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2592742fce208dfd656059d4bf16e5e73b4eaa69
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Mon Jan 8 12:10:33 2024 +0800
[Fix](JDK) Start Fe with JDK17 (#29658)
---
conf/fe.conf | 2 +-
fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java | 2 ++
fe/pom.xml | 2 +-
3 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/conf/fe.conf b/conf/fe.conf
index a8d30963890..4bae10ef224 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -39,7 +39,7 @@ JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:
JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time -Dlog4j2.formatMsgNoLookups=true"
# For jdk 16+, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_16="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time --add-opens java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.lang.ref=ALL-UNNAMED --add-opens java.base/java.util.regex=ALL-UNNAMED --add-opens java.base/ [...]
+JAVA_OPTS_FOR_JDK_16="-Djavax.security.auth.useSubjectCredsOnly=false -XX:+UseZGC -Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$DORIS_HOME/log/ -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"
##
## the lowercase properties are read by main program.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
index 95071a43051..607bfe0ee4f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
@@ -126,6 +126,7 @@ import com.google.gson.JsonObject;
import com.google.gson.JsonParseException;
import com.google.gson.JsonSerializationContext;
import com.google.gson.JsonSerializer;
+import com.google.gson.ReflectionAccessFilter;
import com.google.gson.TypeAdapter;
import com.google.gson.TypeAdapterFactory;
import com.google.gson.annotations.SerializedName;
@@ -300,6 +301,7 @@ public class GsonUtils {
// Add any other adapters if necessary.
private static final GsonBuilder GSON_BUILDER = new GsonBuilder().addSerializationExclusionStrategy(
new HiddenAnnotationExclusionStrategy()).enableComplexMapKeySerialization()
+ .addReflectionAccessFilter(ReflectionAccessFilter.BLOCK_INACCESSIBLE_JAVA)
.registerTypeHierarchyAdapter(Table.class, new GuavaTableAdapter())
.registerTypeHierarchyAdapter(Multimap.class, new GuavaMultimapAdapter())
.registerTypeAdapterFactory(new PostProcessTypeAdapterFactory())
diff --git a/fe/pom.xml b/fe/pom.xml
index e0018b8c181..51157e30082 100644
--- a/fe/pom.xml
+++ b/fe/pom.xml
@@ -239,7 +239,7 @@ under the License.
<commons-pool.version>1.5.1</commons-pool.version>
<commons-text.version>1.10.0</commons-text.version>
<commons-validator.version>1.7</commons-validator.version>
- <gson.version>2.8.9</gson.version>
+ <gson.version>2.10.1</gson.version>
<guava.version>32.1.2-jre</guava.version>
<jackson.version>2.15.2</jackson.version>
<java-cup.version>0.11-a-czt02-cdh</java-cup.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/05: [Fix](pre-load)Fix pom (#30287)
Posted by zy...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
zykkk pushed a commit to branch doris-jdk17
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1320b3aabfbb08c1ec72cf451d52337689bc0a8d
Author: Tiewei Fang <43...@users.noreply.github.com>
AuthorDate: Wed Jan 24 14:03:42 2024 +0800
[Fix](pre-load)Fix pom (#30287)
---
fe/be-java-extensions/preload-extensions/pom.xml | 6 ++++++
regression-test/pipeline/external/conf/be.conf | 3 +++
2 files changed, 9 insertions(+)
diff --git a/fe/be-java-extensions/preload-extensions/pom.xml b/fe/be-java-extensions/preload-extensions/pom.xml
index a63c0a9b238..8c6f57f746c 100644
--- a/fe/be-java-extensions/preload-extensions/pom.xml
+++ b/fe/be-java-extensions/preload-extensions/pom.xml
@@ -140,6 +140,12 @@ under the License.
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>hadoop-client-api</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
diff --git a/regression-test/pipeline/external/conf/be.conf b/regression-test/pipeline/external/conf/be.conf
index 9a5b3641b84..c6a894ebf1d 100644
--- a/regression-test/pipeline/external/conf/be.conf
+++ b/regression-test/pipeline/external/conf/be.conf
@@ -17,6 +17,9 @@
PPROF_TMPDIR="$DORIS_HOME/log/"
+# For jdk 17+, this JAVA_OPTS will be used as default JVM options
+JAVA_OPTS_FOR_JDK_17="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives --add-opens=java.base/java.net=ALL-UNNAMED"
+
# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org