You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by xu...@apache.org on 2023/01/13 13:15:47 UTC
[doris] branch struct-type updated (4ab824250a -> c5be8ec48e)
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a change to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
discard 4ab824250a save (#15908)
omit d32456d603 save (#15906)
omit 7f4de925d8 [feature](struct-type) adjust the vexpr for struct type (#15669)
omit e1e8ca2a59 [fix](struct-type) struct create fix and add more struct codes (#15879)
omit 2578bad186 [WIP](struct-type) support struct-type in vectorize engine (#15665)
omit bebab27bc9 [feature](struct-type) add the class ColumnStruct and class DataTypeStruct implement (#14545)
omit 14ec06a8d2 [test] (test) test modify branch (#14375)
omit 2c7694e13b save
omit 51ba9362f9 test
add e651a9bb11 [feature](nereids) add variance function for nereids (#15370)
add 79113b0cd1 [Fix](storage) Fix bug that cooldown time is error (#15444)
add 9b371f6b0b [fix](web ui) fix fe web ui (#14887)
add c54c2f8035 [fix](statistics) fix npe when __internal_schema not created (#15464)
add bb305aa572 [chore](badges) Remove daily test badges for origin engine (#15482)
add dda505487c [fix](nereids) SimplifyArithmeticRuleTest ut failed (#15486)
add 6c847daba0 [Feature](Nereids) Support grouping set for materialized index. (#15383)
add 3ff01ca799 [feature-wip](multi-catalog) support Iceberg time travel in external table (#15418)
add 73f7ccb58f [typo](docs) fix document display error in SHOW-ALTER.md and SHOW-PARTITION-ID.md and SHOW-PARTITIONS.md (#15453)
add 9a517d6a8f [DataType](Deciamlv3) change the avg function scale of decimalv3 (#15445)
add edb9a3b58d [Bug](timediff) Fix wrong result for function `timediff` (#15312)
add 85c7c531f1 [vectorized](jdbc) support array type in jdbc external table (#15303)
add 2c8de30cce [optimize](multi-catalog) use dictionary encode&filter to process delete files (#15441)
add 10be583e52 [chore](pipeline) optimize profile information (#15433)
add 917b266799 [fix](planner) table valued function could not used in subquery (#15496)
add 2339dcda05 [fix](icebergv2)update icebergv2 regression case (#15442)
add 5c5b7a5c6f [Broker](bos) suppoert baidu bos object storage for broker (#15448)
add 5db8b52441 [Fix](SparkLoad): fix the timeout aborted loadtasks are not cleaned up. (#15480)
add 520b6d7910 [Improvement](decimalv3) Add a config to check overflow for DECIMALV3 (#15463)
add 5ec4e5586f [refactor]remove seek block in segmentIterator (#15413)
add 2704651fde [fix](nereids) hll and bitmap type can't be used as order by and group by exprs (#15471)
add 9246e03932 [Enhancement](hdfs) make libhdfs3 compatible with hdfs2 server (#15497)
add 2f572ccc43 [fix](index) fix that the last element of each batch will be read repeatedly for binary prefix page (#15481)
add dec1eb360c [fix](brokerload) be core dump caused by broker load orc format file nullptr pointer (#15460)
add 08d4dcefff [typo](doc)data partition doc including en and zh-CN #15379
add 93a25e1af5 [fix](nereids) the project node is lost when creating PhysicalStorageLayerAggregate node (#15467)
add 34d7eeb571 [doc](session variable) add doc content for adding variables called rewrite_or_to_in_predicate_threshold (#15513)
add a7895ba169 [feature](Nereids): Support variance_samp function. (#15500)
add 084eec87ee [docs](docs)update en docs (#15470)
add 8e58d92e77 [typo](docs) fix document info missing in SHOW-TABLETS.md (#15488)
add ec52907b06 [fix](index) fix wrong dcheck in indexed column writer (#15520)
add aeaa319203 [fix](fe)change session variable group_concat_max_len from int to long (#15515)
add aacd11336a [typo](docs)update java udf demo (#15521)
add b23d068281 [refactor](remove-non-vec) Remove non vec load from memtable and delta writer (#15517)
add edecc2e706 [feature-wip](inverted index) API for inverted index reader and syntax for fulltext match (#14211)
add ad68764977 [enhancement](tablet) Unify redundant `create_rowset_writer` methods (#15519)
add 9c3c9db49b [enhancement](fuzzy test) support fuzzy test of RewriteOrToInPredicateThreshold #15469
add 9bba2f4cde [typo](docs) array function doc fix (#15449)
add cc7a9d92ad [refactor](non-vec) remove non vec code for indexed column reader (#15409)
add 100834df8b [fix](nereids) fix some arrgregate bugs in Nereids (#15326)
add 96518db263 [enhencement](Nereids) remove constant expr constraint on OneRowRelation (#15506)
add 781fa17993 [fix](Nereids) round function return type should be double (#15502)
add c47bdf6606 [vectorized](jdbc) fix external table of oracle have keyworld column (#15487)
add e89adc6e1d [fix](create-table) wrong judgement about partition column type (#15542)
add 50f1931f96 [fix](multi-catalog) get dictionary-encode from parquet metadata (#15525)
add 487d159a3d [improvement](test) add one case for hll (#15543)
add 62f43c3314 [doc](thirdparty) update CHANGELOG, add libhdfs3 2.3.4 (#15539)
add 3c2dee1d10 [fix](typo) Fix typo in variable name (#15538)
add 7a433a19e6 [fix](typo) Fix typo in variable name (#15538)
add ad9a67a76a [Bug](decimalv3) Fix wrong decimalv3 value after insertion (#15505)
add 238ae54620 [fix](merge-on-write) unique key mow tables should require distribution columns be key column (#15535)
add 5d145cf86f [fix](regression-test) fix duplicate columns in yandex_metrica_p2 case (#15489)
add 87110ad3e3 [chore](Sink)remove useless OlapTablePartitionParam-related code (#15549)
add 5062c62ee1 [chore](script) add build-for-release.sh (#15545)
add 365c3eec16 [enhancement](compaction) vertical compaction support unique-key mow (#15353)
add 40c53931e5 [fix](vec) VMergeIterator add key same label for agg table (#14722)
add 14eaf41029 [refactor](remove rowblockv2) remove rowblock v2 structure (#15540)
add 17286861ef [Fix](multi catalog)Skip non-vectorized init code for NewFileScanNode. #15550
add 57620f6f0d [bugfix](datetimev2) fix coredump when load datatime data to doris (#15510)
add 31548cfe2a [fix](nereids) check failed that exchange node under agg must from PhysicalDistribute (#15473)
add 02d035466b [refactor] remove partition pruner v1 (#15552)
add ada72b055f [feature](Nereids): Support any_value/any function. (#15450)
add 77fda4f749 [SpillToDisk](block reader and writer)Support spill to disk: implement interfaces for spill block and read block (#15399)
add 9ab663212b [docs](muti-catalog) update external iceberg system doc (#15556)
add 893f5f9345 [feature-wip](multi-catalog) support automatic sync hive metastore events (#15401)
add 8748f65a1b [fix](nereids)support nulls first/last in order by clause (#15530)
add b50448d5c4 [vectorized](udaf) fix udaf result is null when has multiple aggs (#15554)
add 8a5f1351e2 [typo](doc) add be max jvm heap size config description (#15561)
add caaae28b50 [docs](export) fix export data to object storage docs (#15563)
add 1dabcb0111 [Fix](Nereids) fix except and intersect error for statsCalculator (#15557)
add a365486a25 [fix](Nereids) get datatype for binary arithmetic (#15548)
add 4380f1ec54 [Enhancement](load) reduce memory by memory size of global delta writer (#14491)
add 85fe9d2496 [Bug](filter) fix not in(null) return true (#15466)
add 55dc541c90 [Fix](Nereids) aggregate function except COUNT should nullable without group by expr (#15547)
add 8d0c06c897 [fix](nereids) binding priority in agg-sort, having, group_by_key (#15240)
add 1f07e54178 [typo](docs)fix logstash readme url (#15573)
add 51a042eab3 [doc](yaml) update the label (#15575)
add 18bc354c06 [fix](Nereids) use correct column unique id when read data from non-base index (#15534)
add a97f582b93 [fix](nereids) use DAYS as default unit for DATE_ADD and DATE_SUB function (#15559)
add eef1f432dd [Bug](datetimev2/decimalv3) Fix wrong predicate infer rule (#15574)
add e5397efb67 [docs](releasenotes)release 1.2.1 (#15583)
add f2f06c1acc [feature](nereids) Support select temp partition (#15579)
add 7728794b4a [fix](Nereids) SimplifyArithmeticRule generate wrong expression after process (#15580)
add 36e43c2677 fix 1.2.1 release notes (#15590)
add 73d4070708 [bugfix](compaction) fix missing key_bounds in vertical compaction (#15578)
add e0c56bcd20 [Feature](export) Support cancel export statement (#15128)
add a4af1fbf90 [fix](inbitmap) forbid having clause to include in bitmap. (#15494)
add c42c61dcad [fix](bitmapfilter) fix bitmap filter not pushing down (#15532)
add 7ef3940809 [fix](storage-policy) fix some bug (#15585)
add 4075e3aec6 [fix](csv-reader) fix new csv reader's performance issue (#15581)
add 804ea08825 [add_log](checksum) add cost and size log for checksum (#15599)
add e67ea1ddb7 [fix](doc): catalog use resource doc error (#15607)
add 61d538c713 [improvement](storage-policy) Add check validity when create storage policy. (#14405)
add 5ff5b8fc98 [feature](mark join) Support mark join for hash join node (#15569)
add 93f5e440eb [Bug](execute) fix get next non stop for eos on streaming preagg (#15611)
add 59f34be41f [fix](having-clause) having clause do not works correct with same alias name (#15143)
add 1018657d9d [Enhancement](SparkLoad): avoid BE OOM in push task, fix #15572 (#15620)
add 4f2a36f032 [project] update year in NOTICE.txt (#15632)
add 0dfa143140 [enhancement](Nereids) generate colocate join when property is different with require property (#15479)
add 6523b546ab [chore](vulnerability) fix some high risk vulnerabilities report by bug scanner (#15621)
add 5ee479f45c [Pipeline](load) Support transaction on pipeline engine (#15597)
add 5460c873e8 [Feature] (Nereids) support un equals conjuncts in un scalar sub query (#15591)
add d36b93708c [feature](Nereids): add ExtractFilterFromJoin rule to support more (#14896)
add 9d1f02c580 [Improvement](topn) runtime prune for topn query (#15558)
add 77ffafb766 [vulnerability](CVE-2022-1292) fix CVE-2022-1292 (#15639)
add 6d691edcc7 [fix](Nereids): restrict join reorder project. (#15645)
add 95f2f43c02 [fix](macOS) Failed to run BE UT due to syscall to map cache into shared region failed (#15641)
add b41934864e [enhancement](frontendservice) add retry when create connection to frontend service (#15635)
add 05d72e8919 [fix](join) fix anti join incorrectly outputs null values (#15567)
add b57500d0c3 [Bug](decimalv3) fix wrong result for MOD operation (#15644)
add df2da89b89 [feature](multi-catalog) support postgresql jdbc catalog (#15570)
add ef72b8d859 [Feature](Nereids): add logical operator || && (#15643)
add ae77b582f0 [fix](Nereids) add information function and fix bugs in schemaScan (#15608)
add 7f84db310a [fix](nereids) Convert to datetime when binary expr's left is date and right is int type (#15615)
add f24659c003 [Refactor](pipeline) refactor the code of channel buffer limit and change the default value (#15650)
add 1038093c29 [Pipeline](Exec) disable work steal of hash join build (#15652)
add 9c36278c4a [improvement](pipeline) Support sharing hash table for broadcast join (#15628)
add 53559e2bdc [fix](decimalv2) fix loss of precision when cast to decimalv2 literal (#15629)
add cad47dd9d9 [test](Nereids) add two regression test cases for Nereids (#15598)
add c18bfdc93e [test][regression cases][external]add external table p2 regression cases according doris1.2 docs 20230105 (#15651)
add a6773417ef [Doc] Add sidebars for split_by_string function and delete split_by_char builtins code (#15679)
add 08d439cde7 [feature](Nereids) add keyword rlike (#15647)
add 9c8fcd805c [feature](Nereids) support variable type expression (#15659)
add 76ad599fd7 [enhancement](histogram) optimise aggregate function histogram (#15317)
add 5dfdacd278 [enhancement](histogram) add histogram syntax and perstist histogram statistics (#15490)
add 054af036fe [typo](doc) fix Chinese describe (#15683)
add ae1a77e034 add Q&A to jdbc external table (#15680)
add a1d8177e33 [fix](test) remove unstable regression test (#15689)
add 707eab9a63 [opt](multi-catalog) cache and reuse position delete rows in iceberg v2 (#15670)
add 500c7fb702 [improvement](multi-catalog) support unsupported column type (#15660)
add 90be1a22a9 [bugfix](vertical compaction) fix dcheck failed in MOW tablet (#15638)
add 36590da24b [fix](regression p0) add the alias function hist to histogram and fix p0 (#15708)
add f256bb8d39 [fix](meta) fix priv table load bug when upgrading to 1.2.x (#15706)
add fb1f6bdd82 [doc](export) add docs for cancel-export. (#15682)
add ba54634d55 [refactor] delete non vec load from memtable (#15667)
add 663676ccfe fix(ui): 1. fix component/table can not change pageSize,affect system/query profile/session page etc. (#15533)
add c57fa7c930 [Pipeline] Fix PipScannerContext::can_finish return wrong status (#15259)
add 6829d361cb [Feature](audit) add errorCode and errorMessage in audit log (#14925)
add 97cea9b5c9 [improvement](bdbje) add more log to make bdbje DatabaseNotFoundException problem easily solved (#15715)
add 1514b5ab5c [Feature](Materialized-View) support advanced Materialized-View (#15212)
add 211cc66d02 [fix](multi-catalog) fix image loading failture when create catalog with resource (#15692)
add 699bf972e2 [Bug](bitmap) Fix bitmap_from_string for null constant (#15698)
add 93b941baeb [fix](tvf) use virtual-hosted style when s3('uri'='s3://xxx') (#15617)
add 4f2bea86ee [fix](Nereids) divide operator return type is not same with lagecy planner (#15707)
add 4c50c4906b [fix](Nereids) add implicit casting for arithmetic expression (#15630)
add 2c9c7c48ac [improvement](decimalv3) Java UDF and array type support DECIMALV3 (#15674)
add e2492cf7fc [Bug](DECIMALV3) Fix binary predicate between decimalv3 and float (#15696)
add 7543d677fa [fix](nereids) Fix the bugs of data distribution calculation on OlapScan (#15699)
add 5ceb5441f4 [feature](nereids) let set operation syntax campatible with lagecy planner (#15664)
add 67ceb83294 [enhance](Nereids): polish test format, add more comment. (#15662)
add 2b0e5e42a5 [ehancement](nereids) Support list parttion prune (#15724)
add 05f6e4c48a [fix](predicate) fix be core dump caused by pushing down the double column predicate (#15693)
add ab186a60ce [enhancement](compaction) Optimize judging delete rowset and picking candidate rowsets for compaction #15631
add 9e3a61989b [refactor](es) remove BE generated dsl for es query #15751
add 67a6ad648e [typo](doc) command of manually trigger compaction incorrect (#15709)
add 3990a44aba [typo](doc) add since dev lable to field function doc (#15648)
add 9c0f96883a [fix](hashjoin) Fix right join pull output block memory overflow (#15440)
add c19e391d32 [fix](profile) show query profile for pipeline engine (#15687)
add fd7d13d4c0 [typo](docs)Update dynamic-partition.md #15734
add d0e8f84279 [feature](vectorized) Support MemoryScratchSink on vectorized engine (#15612)
add 601d9af23b [fix](planner) disconjunct in sub-query failed when plan it on hash join (#15653)
add 025623a124 [feature](Nereids) Support lots of aggregate functions (#15671)
add 1888aba301 [fix](MTMV) fix replayReplaceTable error when restart fe (#15564)
add dec79c000b [fix](MTMV) build mode is missing after restart FE (#15551)
add f17d69e450 [feature](file cache)Import `file cache` for remote file reader (#15622)
add ec0a9647f1 [typo](docs)Update sequence-column-manual.md #15727
add 47097a3db8 [fix](having) revert 15143 and fix having clause with multi-conditions (#15745)
add c3da5a687a [fix]fixed dangerous usage of namespace std (#15741)
add 672d11522b [regression](flink)add flink doris connector case (#15676)
add 503b6ee4da [chore](vulnerability) fix fe high risk vulnerability scanned by bug scanner (#15649)
add a67cea2d27 [Enhancement](metric) add current edit log metric (#15657)
add bb28144c76 [fix](schema change) bugfix for light schema change while with rollup (#15681)
add 7767931aca [ehancement](nereids) let parser support utf8 identifier (#15721)
add fec89ad58c [fix](nereids) week should be able to recognized as function name in function call context (#15735)
add 542542a4b2 [fix](nereids) fix bug in estimation of min/max of Year (#15712)
add 90a92f0643 [feature-wip](multi-catalog) add iceberg tvf to read snapshots (#15618)
add 79b24cdb1f [fix](JdbcResource) fix that JdbcResource does not support the jdbcurl of Oracle and SQLServer (#15757)
add ab2e0fd397 [fix](tvf) cancel strict restrictions on tvf parameters (#15764)
add 4be54cfcac [deps](hdfs) update libhdfs3 to v2.3.5 to support KMS (#15770)
add 5dc644769a [mtmv](regression-test) add mtmv write data regression test (#15546)
add f5b0f5e01a [chore](macOS) Don't build useless third-party stuff (#15763)
add 280603b253 [fix](nereids) bind sort key priority problem (#15646)
add c87a9a5949 [fix](Nereids) Add varchar literal compare (#15672)
add bc34a44f06 [Fix](Nereids) fix type coercion for binary arithmetic (#15185)
add 124c8662e8 [Bug](schema scanner) Fix wrong type in schema scanner (#15768)
add 4bbc93b7ce [refactor](hashtable) simplify template args of partitioned hash table (#15736)
add 8f31a36429 [feature] support spill to disk for sort node (#15624)
add 89c21af87d [chore](fe) update fe snapshot to 1.2 and fix auditloader compile error (#15787)
add 5b10116eca [chore](thirdparty) fix bug that GSSAPI of libgsasl is disabled (#15753)
add d857b4af1b [refactor](remove row batch) remove impala rowbatch structure (#15767)
add 3fec5ff0f5 [refactor](scan-pool) move scan pool from env to scanner scheduler (#15604)
add f5948eb4b0 [Build](cmake) Uniform capitalization keyword of cmake (#15728)
add fe5e5d2bf4 [refactor] separate agg and flush in memtable (#15713)
add 5c2a38d2a1 [chore](thirdparty) Fix the md5sum of the package brpc-1.2.0.tar.gz (#15789)
add 870b5c44e6 [fix](compile) compile failed in Mac with clang14 (#15661)
add 3c8c31a5f8 [chore](Session) remove unused codes for enable_lateral_view
add 2587095811 [Bug](mv) fix mv selector check group expr && forbid create dup mv with bitmap/hll && add some case (#15738)
add 94f6380137 [enhance](Nereids): github action forgot some nereids file. (#15746)
add af3416ede0 [docs] Update be-vscode-dev.md (#15800)
add 7f2c433e08 [feature](Nereids) add relation id to unboundTVFRelation to avoid incorrect group expression comparison (#15740)
add 006b3bd61a [fix](nereids) orthogonal_bitmap_intersect's return type should be bitmap (#15784)
add 98d69d1568 [fix](compile) fix vscan node compile error (#15805)
add 4424874237 [fix](Nereids): move parentExpression in moveOwnership() (#15786)
add 18a3b75626 [fix](QueryDetail) fix QueryDetail may be incorrect and null pointer exception (#15765)
add d4e4e18b47 [fix](DOE): Fix query _id error and es properties error (#15792)
add cfb110c905 [fix](nereids) fix some nereids bugs (#15714)
add a2607141a2 [chore](docs) improve docs for ALTER-TABLE-COLUMN #15758
add 330ed9a84c [fix](Nereids) toSql is not work well in non-query statement (#15752)
add ea1493d946 [fix](Nereids) can not parse left and right function (#15655)
add 88a2088c1d [feature](Nereids) parse pipe_concat symbol as concat when sql mode set to PIPES_AS_CONCAT (#15775)
add 99e5ca937a [doc](multi-catalog)hms event sync (#15809)
add 8841988b46 [typo](docs) Optimize FE minor version upgrade document (#15822)
add f3ef3f7e15 [fix](sink) fix memory leak in VNodeChannel (#15834) (#15835)
add 791604ba1f [log](vlog) improve vlog print for query TExecPlanFragmentParams (#15806)
add 640e451b53 [typo](doc) add hot partition caution (#15825)
add ea0ef0d880 [fix](session-variable) repeat_max_num should be forwarded (#15840)
add b86e781727 [testcase](index)change wait timeout from 1m to 2m in index_meta testcase #15838
add 39697bb83e [fix](Nereids) make the type of the first parameter in window_funnel is intergerLike (#15810)
add 92dd7c442a [enhancement](unique key) disable concurrent flush memtable for unique key (#15802)
add b893c0efd8 [typo](doc)any_value add 1.2 label #15827
add 8c47c57264 [regression-test](array) fix abnormal test on function array_intersect (#15848)
add 7441b4dc96 [Feature](function) Support width_bucket function (#14396)
add e48d715a3b [typo](doc) add hive catalog faq related to kerberos and hdfs (#15826)
add ef0e0cf68d [enhancement](load) refine the reduce memory policy when process memory is nearly full (#15685)
add 0fbdf8e3e1 [Refactor](table function) Decouple vectorized table functions from non-vectorized ones (#15772)
add d23646793c [fix](nereids) binding group by key on agg.output if output is slot (#15623)
add be110ffaf6 [thirdparty](clucene) add clucene deps for doris inverted index (#15807)
add 14e3879c4b [regression-test](MTMV) Make the case test_create_mtmv more robust (#15866)
add 9d41994c17 [opt](Nereids) throw exception when aliasedQuery has no alias(#15854)
add a7af869bfd [opt](Nereids) group_concat to support more cases (#15815)
add 174e5e601f [refactor](rpc fn) decouple vectorized remote function from row-based one (#15871)
add 730571e386 [fix](sort spill) fix bug of failed to create spilled file (#15864)
add bae29157aa [fix](olap) dictionary cannot be sorted after inserting some null values (#15829)
add 16862d9b43 [refactor](remove unused code) remove buffer pool and disk io mgr (#15853)
add 688a0bb96a [feature](multi-catalog) support clickhouse jdbc catalog (#15780)
add 9468711f9f [Bug](join) fix bug null aware left anti join not correct result (#15841)
add b1fb1277dd [fix](bitmap) fix bitmap iterator comparison error (#15779)
add 5e59954531 [typo](docs)Update ARRAY.md (#15781)
add 34bb9cd5d3 [fix](parquet-reader) fix coredump when load datatime data to doris from parquet (#15794)
add 67378a2dc3 [fix](nereids) fix bug in SequenceFunction legality check (#15812)
add c1963e799a [fix](nereids)upgrade signature datatype bug (#15867)
add a8dacfbfd9 [opt](planner) return bigint literal when cast date literal to bigint type (#15613)
add 1489e3cfbf [Fix](file system) Make the constructor of `XxxFileSystem` a private method (#15889)
add e979cc444a [improvement](multi-catalog) support hive 1.x (#15886)
add 049f8ad2f9 [Bug](sort)fix merge sorter might div zero when block bytes less than block rows (#15859)
add 514de605b6 [Bug](predicate) add double predicate creator (#15762)
new d2d0960a3d test
new 3d3d92f6c8 save
new e2c95a9098 [test] (test) test modify branch (#14375)
new 3904f1a8d1 [feature](struct-type) add the class ColumnStruct and class DataTypeStruct implement (#14545)
new 550e5b23f0 [WIP](struct-type) support struct-type in vectorize engine (#15665)
new 22b8866753 [fix](struct-type) struct create fix and add more struct codes (#15879)
new 0d4da4f5f7 [feature](struct-type) adjust the vexpr for struct type (#15669)
new 9b43fe8da1 save (#15906)
new c5be8ec48e save (#15908)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4ab824250a)
\
N -- N -- N refs/heads/struct-type (c5be8ec48e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.asf.yaml | 6 -
.github/workflows/labeler/scope-label-conf.yml | 1 +
.github/workflows/scope-label.yml | 1 -
NOTICE.txt | 2 +-
README.md | 3 +-
be/CMakeLists.txt | 30 +-
be/src/agent/task_worker_pool.cpp | 18 +-
be/src/common/config.h | 30 +-
be/src/common/consts.h | 1 +
be/src/common/daemon.cpp | 43 +-
be/src/common/daemon.h | 5 +-
be/src/common/status.h | 3 +
be/src/exec/base_scanner.cpp | 40 +-
be/src/exec/base_scanner.h | 4 +-
be/src/exec/data_sink.cpp | 93 +-
be/src/exec/es/es_predicate.cpp | 2 -
be/src/exec/es/es_scroll_query.cpp | 17 +-
be/src/exec/es/es_scroll_query.h | 1 -
be/src/exec/exec_node.cpp | 307 +--
be/src/exec/exec_node.h | 16 -
be/src/exec/olap_common.h | 91 +-
be/src/exec/olap_utils.h | 86 +
be/src/exec/parquet_scanner.cpp | 1 +
.../exec/schema_scanner/schema_columns_scanner.cpp | 20 +-
be/src/exec/table_connector.cpp | 291 ++-
be/src/exec/table_connector.h | 8 +-
be/src/exec/tablet_info.cpp | 304 ---
be/src/exec/tablet_info.h | 120 +-
be/src/exprs/CMakeLists.txt | 6 +-
be/src/exprs/aggregate_functions.cpp | 4 +-
be/src/exprs/create_predicate_function.h | 11 +-
be/src/exprs/expr.cpp | 9 +
be/src/exprs/hybrid_set.h | 7 +-
be/src/exprs/match_predicate.cpp | 102 +
be/src/exprs/match_predicate.h | 85 +
be/src/exprs/runtime_filter.cpp | 76 +-
.../exprs/table_function/dummy_table_functions.cpp | 66 -
.../exprs/table_function/dummy_table_functions.h | 52 -
be/src/exprs/table_function/explode_bitmap.cpp | 116 -
be/src/exprs/table_function/explode_bitmap.h | 50 -
be/src/exprs/table_function/explode_json_array.cpp | 189 --
be/src/exprs/table_function/explode_json_array.h | 135 -
be/src/exprs/table_function/explode_split.cpp | 93 -
be/src/exprs/table_function/explode_split.h | 49 -
be/src/exprs/table_function/table_function.h | 127 -
.../table_function/table_function_factory.cpp | 108 -
.../exprs/table_function/table_function_factory.h | 50 -
be/src/gutil/hash/hash.h | 12 +-
be/src/gutil/strings/stringpiece.cc | 4 +-
be/src/gutil/strings/stringpiece.h | 4 +-
be/src/http/action/pad_rowset_action.cpp | 8 +-
be/src/io/CMakeLists.txt | 6 +
be/src/io/cache/block/block_file_cache.cpp | 125 +
be/src/io/cache/block/block_file_cache.h | 252 ++
be/src/io/cache/block/block_file_cache_factory.cpp | 94 +
be/src/io/cache/block/block_file_cache_factory.h | 59 +
be/src/io/cache/block/block_file_cache_fwd.h | 37 +
be/src/io/cache/block/block_file_cache_profile.cpp | 202 ++
be/src/io/cache/block/block_file_cache_profile.h | 115 +
be/src/io/cache/block/block_file_cache_settings.h | 40 +
be/src/io/cache/block/block_file_segment.cpp | 317 +++
be/src/io/cache/block/block_file_segment.h | 205 ++
be/src/io/cache/block/block_lru_file_cache.cpp | 844 ++++++
be/src/io/cache/block/block_lru_file_cache.h | 159 ++
.../io/cache/block/cached_remote_file_reader.cpp | 215 ++
be/src/io/cache/block/cached_remote_file_reader.h | 76 +
be/src/io/cache/dummy_file_cache.h | 3 +
be/src/io/cache/file_cache_manager.cpp | 6 +-
be/src/io/cache/file_cache_manager.h | 2 +-
be/src/io/cache/sub_file_cache.cpp | 2 +-
be/src/io/cache/sub_file_cache.h | 3 +
be/src/io/cache/whole_file_cache.cpp | 3 +-
be/src/io/cache/whole_file_cache.h | 3 +
be/src/io/file_factory.cpp | 64 +-
be/src/io/file_factory.h | 24 +-
be/src/io/fs/broker_file_reader.cpp | 27 +-
be/src/io/fs/broker_file_reader.h | 9 +-
be/src/io/fs/broker_file_system.cpp | 14 +-
be/src/io/fs/broker_file_system.h | 10 +-
be/src/io/fs/file_reader.h | 6 +-
be/src/io/fs/file_reader_options.cpp | 10 +-
be/src/io/fs/file_reader_options.h | 20 +-
be/src/io/fs/file_system.h | 12 +-
be/src/io/fs/hdfs_file_reader.cpp | 4 +-
be/src/io/fs/hdfs_file_reader.h | 6 +-
be/src/io/fs/hdfs_file_system.cpp | 20 +-
be/src/io/fs/hdfs_file_system.h | 8 +-
be/src/io/fs/local_file_reader.cpp | 8 +-
be/src/io/fs/local_file_reader.h | 6 +-
be/src/io/fs/local_file_system.cpp | 15 +-
be/src/io/fs/local_file_system.h | 13 +-
be/src/io/fs/remote_file_system.cpp | 21 +-
be/src/io/fs/remote_file_system.h | 4 +-
be/src/io/fs/s3_file_reader.cpp | 4 +-
be/src/io/fs/s3_file_reader.h | 6 +-
be/src/io/fs/s3_file_system.cpp | 13 +-
be/src/io/fs/s3_file_system.h | 7 +-
be/src/io/fs/stream_load_pipe.h | 3 +
be/src/olap/CMakeLists.txt | 2 +-
be/src/olap/aggregate_func.cpp | 9 +-
be/src/olap/base_compaction.cpp | 5 +-
be/src/olap/block_column_predicate.cpp | 1 -
be/src/olap/column_predicate.h | 13 +
be/src/olap/compaction.cpp | 21 +-
be/src/olap/comparison_predicate.h | 46 +
be/src/olap/cumulative_compaction.cpp | 5 +-
be/src/olap/cumulative_compaction_policy.cpp | 18 +-
be/src/olap/cumulative_compaction_policy.h | 9 -
be/src/olap/data_dir.cpp | 2 +-
be/src/olap/delta_writer.cpp | 94 +-
be/src/olap/delta_writer.h | 9 +-
be/src/olap/in_list_predicate.h | 41 +-
be/src/olap/iterators.h | 19 +
be/src/olap/memtable.cpp | 198 +-
be/src/olap/memtable.h | 51 +-
be/src/olap/memtable_flush_executor.cpp | 6 +-
be/src/olap/merger.cpp | 23 +-
be/src/olap/olap_common.h | 19 +
be/src/olap/olap_define.h | 2 +-
be/src/olap/options.cpp | 65 +
be/src/olap/options.h | 17 +
be/src/olap/predicate_creator.h | 24 +-
be/src/olap/push_handler.cpp | 33 +-
be/src/olap/push_handler.h | 1 +
be/src/olap/reader.cpp | 7 +
be/src/olap/reader.h | 2 +
be/src/olap/row_block2.cpp | 682 -----
be/src/olap/row_block2.h | 162 --
be/src/olap/rowset/beta_rowset_reader.cpp | 3 +-
be/src/olap/rowset/beta_rowset_reader.h | 1 -
be/src/olap/rowset/beta_rowset_writer.cpp | 71 +-
be/src/olap/rowset/beta_rowset_writer.h | 4 +-
be/src/olap/rowset/rowset_meta.h | 11 +-
be/src/olap/rowset/rowset_reader_context.h | 2 +
be/src/olap/rowset/rowset_writer.h | 9 +-
be/src/olap/rowset/rowset_writer_context.h | 33 +-
.../olap/rowset/segment_v2/binary_prefix_page.cpp | 25 +-
be/src/olap/rowset/segment_v2/binary_prefix_page.h | 6 +-
.../olap/rowset/segment_v2/bitmap_index_reader.cpp | 13 +-
.../segment_v2/bloom_filter_index_reader.cpp | 19 +-
be/src/olap/rowset/segment_v2/column_reader.cpp | 35 +
be/src/olap/rowset/segment_v2/column_reader.h | 23 +-
.../rowset/segment_v2/empty_segment_iterator.cpp | 2 -
.../rowset/segment_v2/indexed_column_reader.cpp | 35 +
.../olap/rowset/segment_v2/indexed_column_reader.h | 3 +
.../rowset/segment_v2/indexed_column_writer.cpp | 7 +-
.../rowset/segment_v2/inverted_index_reader.cpp | 55 +
.../olap/rowset/segment_v2/inverted_index_reader.h | 103 +
be/src/olap/rowset/segment_v2/page_decoder.h | 4 +-
be/src/olap/rowset/segment_v2/segment.cpp | 55 +-
be/src/olap/rowset/segment_v2/segment.h | 3 +
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 236 +-
be/src/olap/rowset/segment_v2/segment_iterator.h | 82 +-
be/src/olap/rowset/segment_v2/zone_map_index.cpp | 17 +-
be/src/olap/rowset/vertical_beta_rowset_writer.cpp | 4 +-
be/src/olap/rowset/vertical_beta_rowset_writer.h | 2 +-
be/src/olap/schema.cpp | 13 +-
be/src/olap/schema.h | 2 +
be/src/olap/schema_change.cpp | 67 +-
be/src/olap/storage_policy_mgr.cpp | 11 +-
be/src/olap/tablet.cpp | 129 +-
be/src/olap/tablet.h | 29 +-
be/src/olap/task/engine_batch_load_task.h | 2 -
be/src/olap/task/engine_checksum_task.cpp | 8 +-
be/src/olap/utils.cpp | 10 +-
be/src/olap/utils.h | 16 +
be/src/pipeline/CMakeLists.txt | 3 +-
be/src/pipeline/exec/aggregation_source_operator.h | 2 +-
be/src/pipeline/exec/assert_num_rows_operator.h | 2 +-
be/src/pipeline/exec/const_value_operator.h | 2 +-
be/src/pipeline/exec/empty_source_operator.cpp | 25 +
be/src/pipeline/exec/empty_source_operator.h | 62 +
be/src/pipeline/exec/hashjoin_build_sink.h | 2 +-
be/src/pipeline/exec/hashjoin_probe_operator.h | 2 +-
be/src/pipeline/exec/operator.h | 24 +-
be/src/pipeline/exec/scan_operator.cpp | 6 +-
be/src/pipeline/exec/set_probe_sink_operator.h | 2 +-
be/src/pipeline/exec/set_sink_operator.h | 2 +-
be/src/pipeline/exec/set_source_operator.h | 2 +-
.../exec/streaming_aggregation_sink_operator.cpp | 6 +-
be/src/pipeline/exec/union_sink_operator.cpp | 2 +-
be/src/pipeline/exec/union_source_operator.cpp | 2 +-
be/src/pipeline/pipeline.h | 19 +-
be/src/pipeline/pipeline_fragment_context.cpp | 31 +-
be/src/pipeline/pipeline_fragment_context.h | 6 +
be/src/pipeline/pipeline_task.cpp | 61 +-
be/src/pipeline/pipeline_task.h | 32 +-
be/src/pipeline/task_queue.cpp | 23 +-
be/src/pipeline/task_queue.h | 6 +-
be/src/runtime/CMakeLists.txt | 12 +-
be/src/runtime/block_spill_manager.cpp | 132 +
be/src/runtime/block_spill_manager.h | 60 +
be/src/runtime/bufferpool/buffer_allocator.cc | 744 ------
be/src/runtime/bufferpool/buffer_allocator.h | 241 --
be/src/runtime/bufferpool/buffer_pool.cc | 667 -----
be/src/runtime/bufferpool/buffer_pool.h | 466 ----
be/src/runtime/bufferpool/buffer_pool_counters.h | 43 -
be/src/runtime/bufferpool/buffer_pool_internal.h | 299 ---
be/src/runtime/bufferpool/free_list.h | 115 -
be/src/runtime/bufferpool/suballocator.cc | 252 --
be/src/runtime/bufferpool/suballocator.h | 221 --
be/src/runtime/bufferpool/system_allocator.cc | 166 --
be/src/runtime/bufferpool/system_allocator.h | 49 -
be/src/runtime/cache/result_cache.h | 1 -
be/src/runtime/cache/result_node.h | 1 -
be/src/runtime/client_cache.h | 29 +-
be/src/runtime/collection_value.cpp | 50 +
be/src/runtime/datetime_value.h | 4 +-
be/src/runtime/decimalv2_value.h | 4 +-
be/src/runtime/disk_io_mgr.cc | 1195 ---------
be/src/runtime/disk_io_mgr.h | 837 ------
be/src/runtime/disk_io_mgr_internal.h | 455 ----
be/src/runtime/disk_io_mgr_reader_context.cc | 322 ---
be/src/runtime/disk_io_mgr_scan_range.cc | 481 ----
be/src/runtime/exec_env.h | 31 +-
be/src/runtime/exec_env_init.cpp | 79 +-
be/src/runtime/fragment_mgr.cpp | 78 +-
be/src/runtime/fragment_mgr.h | 10 +-
be/src/runtime/load_channel.cpp | 42 +-
be/src/runtime/load_channel.h | 47 +-
be/src/runtime/load_channel_mgr.cpp | 177 +-
be/src/runtime/load_channel_mgr.h | 30 +-
be/src/runtime/plan_fragment_executor.cpp | 22 +-
be/src/runtime/query_fragments_ctx.h | 8 +-
be/src/runtime/row_batch.cpp | 507 ----
be/src/runtime/row_batch.h | 478 ----
be/src/runtime/runtime_predicate.cpp | 153 ++
be/src/runtime/runtime_predicate.h | 178 ++
be/src/runtime/runtime_state.h | 15 +-
be/src/runtime/tablets_channel.cpp | 223 +-
be/src/runtime/tablets_channel.h | 39 +-
be/src/runtime/types.h | 3 +
be/src/service/backend_service.cpp | 2 -
be/src/service/doris_main.cpp | 39 +
be/src/service/internal_service.cpp | 15 +-
be/src/udf/udf_internal.h | 8 +
be/src/util/CMakeLists.txt | 1 +
be/src/util/arrow/block_convertor.cpp | 323 +++
be/src/util/arrow/block_convertor.h | 46 +
be/src/util/arrow/row_batch.cpp | 351 +--
be/src/util/arrow/row_batch.h | 17 -
be/src/util/arrow/row_block.cpp | 155 --
be/src/util/arrow/row_block.h | 11 -
be/src/util/bitmap_value.h | 2 +-
be/src/util/cityhash102/city.cc | 27 +-
be/src/util/date_func.cpp | 21 +
be/src/util/date_func.h | 2 +
be/src/util/filesystem_util.h | 4 +-
be/src/util/hash_util.hpp | 11 +-
be/src/util/mustache/mustache.cc | 82 +-
be/src/util/string_parser.hpp | 2 +-
be/src/util/time.cpp | 44 +-
be/src/util/uid_util.h | 6 +-
be/src/vec/CMakeLists.txt | 11 +
.../aggregate_function_histogram.cpp | 68 +-
.../aggregate_function_histogram.h | 354 +--
.../aggregate_function_java_udaf.h | 4 +-
be/src/vec/columns/column_const.cpp | 3 +-
be/src/vec/columns/column_dictionary.h | 17 +
be/src/vec/common/hash_table/hash_table.h | 8 +-
.../vec/common/hash_table/partitioned_hash_map.h | 37 +-
.../vec/common/hash_table/partitioned_hash_table.h | 33 +-
be/src/vec/common/hex.cpp | 115 +
be/src/vec/common/hex.h | 136 +
be/src/vec/common/sort/heap_sorter.cpp | 11 +
be/src/vec/common/sort/heap_sorter.h | 2 +
be/src/vec/common/sort/sorter.cpp | 246 +-
be/src/vec/common/sort/sorter.h | 102 +-
be/src/vec/common/sort/topn_sorter.cpp | 39 +-
be/src/vec/common/sort/topn_sorter.h | 4 +-
be/src/vec/common/string_ref.h | 4 +-
be/src/vec/common/strong_typedef.h | 4 +-
be/src/vec/common/uint128.h | 46 +-
be/src/vec/core/block.cpp | 14 +-
be/src/vec/core/block.h | 19 +-
be/src/vec/core/block_spill_reader.cpp | 136 +
be/src/vec/core/block_spill_reader.h | 71 +
be/src/vec/core/block_spill_writer.cpp | 147 ++
be/src/vec/core/block_spill_writer.h | 81 +
be/src/vec/core/column_with_type_and_name.cpp | 2 +-
be/src/vec/core/field.h | 3 +
be/src/vec/core/sort_cursor.h | 26 +-
be/src/vec/core/types.h | 8 +-
be/src/vec/data_types/data_type_decimal.h | 33 +-
be/src/vec/data_types/data_type_factory.cpp | 4 +
be/src/vec/data_types/data_type_factory.hpp | 4 +
be/src/vec/data_types/data_type_time.cpp | 52 +
be/src/vec/data_types/data_type_time.h | 53 +
be/src/vec/data_types/data_type_time_v2.cpp | 4 +-
be/src/vec/data_types/data_type_time_v2.h | 8 +-
.../vec/exec/data_gen_functions/vnumbers_tvf.cpp | 1 -
be/src/vec/exec/format/csv/csv_reader.cpp | 79 +-
be/src/vec/exec/format/csv/csv_reader.h | 18 +-
.../file_reader/new_plain_binary_line_reader.cpp | 3 +-
be/src/vec/exec/format/format_common.h | 60 +
be/src/vec/exec/format/json/new_json_reader.cpp | 20 +-
be/src/vec/exec/format/json/new_json_reader.h | 11 +-
be/src/vec/exec/format/orc/vorc_reader.cpp | 21 +-
be/src/vec/exec/format/orc/vorc_reader.h | 9 +-
be/src/vec/exec/format/parquet/parquet_common.cpp | 56 +
be/src/vec/exec/format/parquet/parquet_common.h | 21 +-
be/src/vec/exec/format/parquet/schema_desc.cpp | 63 +-
.../parquet/vparquet_column_chunk_reader.cpp | 4 +
.../format/parquet/vparquet_column_chunk_reader.h | 1 +
.../exec/format/parquet/vparquet_group_reader.cpp | 10 +-
be/src/vec/exec/format/parquet/vparquet_reader.cpp | 35 +-
be/src/vec/exec/format/parquet/vparquet_reader.h | 16 +-
be/src/vec/exec/format/table/iceberg_reader.cpp | 331 ++-
be/src/vec/exec/format/table/iceberg_reader.h | 43 +-
be/src/vec/exec/join/join_op.h | 4 +-
be/src/vec/exec/join/process_hash_table_probe.h | 6 +-
.../vec/exec/join/process_hash_table_probe_impl.h | 121 +-
be/src/vec/exec/join/vhash_join_node.cpp | 25 +-
be/src/vec/exec/join/vhash_join_node.h | 12 +
be/src/vec/exec/join/vjoin_node_base.cpp | 3 +-
be/src/vec/exec/join/vnested_loop_join_node.cpp | 13 +-
be/src/vec/exec/join/vnested_loop_join_node.h | 20 +-
be/src/vec/exec/scan/new_es_scan_node.cpp | 90 +-
be/src/vec/exec/scan/new_es_scan_node.h | 7 -
be/src/vec/exec/scan/new_es_scanner.cpp | 4 +-
be/src/vec/exec/scan/new_es_scanner.h | 3 +-
be/src/vec/exec/scan/new_file_scan_node.cpp | 4 +-
be/src/vec/exec/scan/new_file_scan_node.h | 2 +
be/src/vec/exec/scan/new_jdbc_scan_node.cpp | 5 +-
be/src/vec/exec/scan/new_jdbc_scanner.cpp | 6 +-
be/src/vec/exec/scan/new_jdbc_scanner.h | 2 +-
be/src/vec/exec/scan/new_odbc_scan_node.cpp | 3 +-
be/src/vec/exec/scan/new_odbc_scanner.cpp | 4 +-
be/src/vec/exec/scan/new_odbc_scanner.h | 2 +-
be/src/vec/exec/scan/new_olap_scan_node.cpp | 6 +
be/src/vec/exec/scan/new_olap_scan_node.h | 6 +
be/src/vec/exec/scan/new_olap_scanner.cpp | 25 +-
be/src/vec/exec/scan/new_olap_scanner.h | 1 -
be/src/vec/exec/scan/pip_scanner_context.h | 6 +-
be/src/vec/exec/scan/scanner_context.cpp | 14 +-
be/src/vec/exec/scan/scanner_context.h | 4 +-
be/src/vec/exec/scan/scanner_scheduler.cpp | 11 +
be/src/vec/exec/scan/scanner_scheduler.h | 6 +
be/src/vec/exec/scan/vfile_scanner.cpp | 37 +-
be/src/vec/exec/scan/vfile_scanner.h | 15 +-
be/src/vec/exec/scan/vmeta_scan_node.cpp | 68 +
be/src/vec/exec/scan/vmeta_scan_node.h | 45 +
be/src/vec/exec/scan/vmeta_scanner.cpp | 210 ++
be/src/vec/exec/scan/vmeta_scanner.h | 48 +
be/src/vec/exec/scan/vscan_node.cpp | 63 +-
be/src/vec/exec/scan/vscan_node.h | 5 +
be/src/vec/exec/scan/vscanner.cpp | 3 +-
be/src/vec/exec/scan/vscanner.h | 6 +-
be/src/vec/exec/vaggregation_node.cpp | 13 +-
be/src/vec/exec/vaggregation_node.h | 1 +
be/src/vec/exec/vanalytic_eval_node.cpp | 1 -
be/src/vec/exec/vbroker_scan_node.h | 2 +-
be/src/vec/exec/vdata_gen_scan_node.cpp | 1 -
be/src/vec/exec/vexchange_node.cpp | 5 +-
be/src/vec/exec/vjdbc_connector.cpp | 217 +-
be/src/vec/exec/vjdbc_connector.h | 36 +-
be/src/vec/exec/vmysql_scan_node.cpp | 22 +-
be/src/vec/exec/vrepeat_node.cpp | 8 +-
be/src/vec/exec/vschema_scan_node.cpp | 17 +-
be/src/vec/exec/vselect_node.cpp | 1 -
be/src/vec/exec/vset_operation_node.h | 1 -
be/src/vec/exec/vsort_node.cpp | 49 +-
be/src/vec/exec/vsort_node.h | 4 +
be/src/vec/exec/vtable_function_node.cpp | 8 +-
be/src/vec/exec/vtable_function_node.h | 7 +-
be/src/vec/exprs/table_function/table_function.h | 109 +
.../table_function/table_function_factory.cpp | 82 +
.../exprs/table_function/table_function_factory.h | 46 +
be/src/vec/exprs/table_function/vexplode.h | 2 +-
.../vec/exprs/table_function/vexplode_bitmap.cpp | 54 +-
be/src/vec/exprs/table_function/vexplode_bitmap.h | 18 +-
.../exprs/table_function/vexplode_json_array.cpp | 108 +-
.../vec/exprs/table_function/vexplode_json_array.h | 91 +-
be/src/vec/exprs/table_function/vexplode_numbers.h | 2 +-
be/src/vec/exprs/table_function/vexplode_split.cpp | 8 +
be/src/vec/exprs/table_function/vexplode_split.h | 21 +-
be/src/vec/exprs/vcase_expr.cpp | 1 -
be/src/vec/exprs/vectorized_fn_call.cpp | 4 +-
be/src/vec/exprs/vexpr.cpp | 10 +-
be/src/vec/exprs/vexpr.h | 4 +
be/src/vec/exprs/vexpr_context.cpp | 2 +
be/src/vec/functions/function_binary_arithmetic.h | 142 +-
be/src/vec/functions/function_bitmap.cpp | 16 +-
be/src/vec/functions/function_cast.h | 124 +-
.../function_date_or_datetime_computation.h | 5 +-
be/src/vec/functions/function_rpc.cpp | 505 +++-
be/src/vec/functions/function_rpc.h | 42 +-
be/src/vec/functions/function_running_difference.h | 3 +-
be/src/vec/functions/function_string.h | 97 +-
be/src/vec/functions/function_totype.h | 4 +-
be/src/vec/functions/function_width_bucket.cpp | 142 +
.../functions/functions_multi_string_position.cpp | 12 +-
be/src/vec/functions/in.h | 22 +-
be/src/vec/functions/match.cpp | 141 +
be/src/vec/functions/simple_function_factory.h | 6 +-
be/src/vec/olap/block_reader.cpp | 11 +-
be/src/vec/olap/block_reader.h | 2 +
be/src/vec/olap/vcollect_iterator.cpp | 5 +
be/src/vec/olap/vertical_block_reader.cpp | 30 +-
be/src/vec/olap/vertical_block_reader.h | 6 +-
be/src/vec/olap/vertical_merge_iterator.cpp | 37 +-
be/src/vec/olap/vertical_merge_iterator.h | 28 +-
be/src/vec/olap/vgeneric_iterators.cpp | 9 +-
be/src/vec/olap/vgeneric_iterators.h | 19 +-
.../vec/runtime/shared_hash_table_controller.cpp | 18 +
be/src/vec/runtime/shared_hash_table_controller.h | 6 +
be/src/vec/runtime/vdata_stream_mgr.cpp | 6 +-
be/src/vec/runtime/vdata_stream_mgr.h | 2 +-
be/src/vec/runtime/vdata_stream_recvr.cpp | 3 +-
be/src/vec/runtime/vdata_stream_recvr.h | 13 +-
be/src/vec/runtime/vdatetime_value.h | 8 +-
be/src/vec/runtime/vsorted_run_merger.cpp | 25 +-
be/src/vec/runtime/vsorted_run_merger.h | 16 +-
be/src/vec/sink/vjdbc_table_sink.cpp | 4 +-
be/src/vec/sink/vjdbc_table_sink.h | 2 -
be/src/vec/sink/vmemory_scratch_sink.cpp | 95 +
be/src/vec/sink/vmemory_scratch_sink.h | 85 +
be/src/vec/sink/vresult_file_sink.cpp | 1 -
be/src/vec/sink/vtablet_sink.cpp | 14 +-
be/src/vec/sink/vtablet_sink.h | 3 +-
be/src/vec/utils/histogram_helpers.hpp | 270 ++
be/test/CMakeLists.txt | 16 +-
be/test/exprs/binary_predicate_test.cpp | 1 -
be/test/exprs/in_op_test.cpp | 1 -
be/test/io/cache/file_block_cache_test.cpp | 544 ++++
be/test/io/cache/remote_file_cache_test.cpp | 2 +-
be/test/olap/block_column_predicate_test.cpp | 1 -
be/test/olap/cumulative_compaction_policy_test.cpp | 45 +-
be/test/olap/delta_writer_test.cpp | 202 +-
.../olap/engine_storage_migration_task_test.cpp | 20 +-
be/test/olap/primary_key_index_test.cpp | 2 +-
be/test/olap/remote_rowset_gc_test.cpp | 18 +-
be/test/olap/rowid_conversion_test.cpp | 62 +-
be/test/olap/rowset/beta_rowset_test.cpp | 2 +-
.../rowset/segment_v2/binary_prefix_page_test.cpp | 138 +
.../olap/rowset/segment_v2/bitmap_index_test.cpp | 3 +-
.../bloom_filter_index_reader_writer_test.cpp | 2 +-
.../segment_v2/column_reader_writer_test.cpp | 4 +-
.../rowset/segment_v2/ordinal_page_index_test.cpp | 2 +-
.../olap/rowset/segment_v2/zone_map_index_test.cpp | 6 +-
be/test/olap/segcompaction_test.cpp | 1 +
be/test/olap/tablet_cooldown_test.cpp | 18 +-
be/test/runtime/array_test.cpp | 54 +-
be/test/runtime/data_spliter_test.cpp | 1 -
be/test/runtime/disk_io_mgr_test.cpp | 1069 --------
be/test/runtime/fragment_mgr_test.cpp | 1 -
be/test/runtime/load_channel_mgr_test.cpp | 240 --
be/test/runtime/test_env.cc | 11 -
be/test/runtime/test_env.h | 3 -
be/test/tools/benchmark_tool.cpp | 7 +-
be/test/util/bitmap_value_test.cpp | 10 +-
.../vec/aggregate_functions/agg_histogram_test.cpp | 153 +-
be/test/vec/core/block_spill_test.cpp | 469 ++++
be/test/vec/core/block_test.cpp | 1 -
be/test/vec/exec/parquet/parquet_reader_test.cpp | 7 +-
be/test/vec/exec/parquet/parquet_thrift_test.cpp | 20 +-
be/test/vec/exec/vgeneric_iterators_test.cpp | 1 -
be/test/vec/exec/vjson_scanner_test.cpp | 1 -
be/test/vec/exec/vorc_scanner_test.cpp | 1 -
be/test/vec/exec/vparquet_scanner_test.cpp | 1 -
be/test/vec/exprs/vexpr_test.cpp | 27 -
.../function/function_running_difference_test.cpp | 2 +-
be/test/vec/function/function_test_util.h | 9 +-
be/test/vec/function/function_time_test.cpp | 22 +-
be/test/vec/olap/vertical_compaction_test.cpp | 7 +-
be/test/vec/runtime/vdata_stream_test.cpp | 5 +-
build-for-release.sh | 162 ++
build.sh | 12 +-
dist/NOTICE-dist.txt | 2 +-
.../hive/scripts/create_preinstalled_table.hql | 12 +
.../postgresql/init/01-create-schema.sql | 1 +
.../postgresql/init/02-create-table.sql | 28 +
.../docker-compose/postgresql/init/03-insert.sql | 33 +
docs/en/community/developer-guide/be-vscode-dev.md | 10 +-
.../cluster-management/elastic-expansion.md | 6 +-
.../admin-manual/cluster-management/upgrade.md | 2 +-
docs/en/docs/admin-manual/config/be-config.md | 16 +-
docs/en/docs/admin-manual/config/fe-config.md | 10 -
.../docs/advanced/partition/dynamic-partition.md | 2 +
docs/en/docs/advanced/variables.md | 14 +-
docs/en/docs/data-operate/export/export-manual.md | 41 +-
docs/en/docs/data-table/data-partition.md | 2 +-
docs/en/docs/ecosystem/audit-plugin.md | 4 +
.../docs/ecosystem/external-table/doris-on-es.md | 2 -
.../docs/ecosystem/external-table/jdbc-of-doris.md | 40 +-
.../docs/ecosystem/external-table/multi-catalog.md | 237 +-
.../ecosystem/udf/java-user-defined-function.md | 41 +-
.../construct-docker/construct-docker-compose.md | 4 +-
.../construct-docker/construct-docker-image.md | 100 +-
docs/en/docs/install/install-deploy.md | 258 +-
.../docs/install/source-install/compilation-arm.md | 235 +-
.../docs/install/source-install/compilation-mac.md | 31 +-
.../docs/install/source-install/compilation-win.md | 26 +-
.../compilation-with-ldb-toolchain.md | 41 +-
docs/en/docs/install/source-install/compilation.md | 131 +-
docs/en/docs/releasenotes/release-1.2.1.md | 196 ++
.../sql-functions/aggregate-functions/any_value.md | 8 +
.../sql-functions/aggregate-functions/histogram.md | 92 +-
.../sql-functions/array-functions/array_join.md | 2 +-
.../sql-functions/array-functions/array_sort.md | 2 +-
.../sql-functions/string-functions/field.md | 7 +
.../sql-manual/sql-functions/table-functions/s3.md | 43 +-
.../docs/sql-manual/sql-functions/width-bucket.md | 154 ++
.../Alter/ALTER-TABLE-COLUMN.md | 2 +-
.../Create/CREATE-CATALOG.md | 66 +-
.../Manipulation/CANCEL-EXPORT.md | 80 +
.../sql-manual/sql-reference/Data-Types/ARRAY.md | 4 +-
.../sql-reference/Data-Types/DECIMALV3.md | 2 +-
.../sql-reference/Show-Statements/SHOW-TABLETS.md | 7 +-
docs/sidebars.json | 8 +-
.../admin-manual/cluster-management/upgrade.md | 2 +-
docs/zh-CN/docs/admin-manual/config/be-config.md | 16 +-
docs/zh-CN/docs/admin-manual/config/fe-config.md | 10 -
.../admin-manual/http-actions/compaction-action.md | 2 +-
.../maint-monitor/monitor-metrics/metrics.md | 4 +-
.../docs/advanced/partition/dynamic-partition.md | 4 +-
docs/zh-CN/docs/advanced/variables.md | 12 +-
.../docs/data-operate/export/export-manual.md | 42 +-
.../update-delete/sequence-column-manual.md | 2 +-
docs/zh-CN/docs/data-table/data-partition.md | 2 +-
docs/zh-CN/docs/ecosystem/audit-plugin.md | 4 +
.../docs/ecosystem/external-table/doris-on-es.md | 2 -
.../docs/ecosystem/external-table/jdbc-of-doris.md | 28 +
.../docs/ecosystem/external-table/multi-catalog.md | 240 +-
.../ecosystem/udf/java-user-defined-function.md | 43 +-
.../compilation-with-ldb-toolchain.md | 13 +-
docs/zh-CN/docs/releasenotes/release-1.2.1.md | 193 ++
.../sql-functions/aggregate-functions/any_value.md | 8 +
.../sql-functions/aggregate-functions/histogram.md | 91 +-
.../sql-functions/array-functions/array.md | 2 +-
.../array-functions/array_difference.md | 2 +-
.../sql-functions/array-functions/array_join.md | 2 +-
.../sql-functions/array-functions/array_sort.md | 4 +-
.../array-functions/array_with_constant.md | 2 +-
.../sql-functions/string-functions/field.md | 7 +
.../string-functions/split_by_string.md | 4 +-
.../sql-manual/sql-functions/table-functions/s3.md | 45 +-
.../docs/sql-manual/sql-functions/width-bucket.md | 153 ++
.../Alter/ALTER-TABLE-COLUMN.md | 4 +-
.../Create/CREATE-CATALOG.md | 68 +-
.../Manipulation/CANCEL-EXPORT.md | 80 +
.../sql-manual/sql-reference/Data-Types/ARRAY.md | 4 +-
.../sql-reference/Data-Types/DECIMALV3.md | 2 +-
.../sql-reference/Show-Statements/SHOW-ALTER.md | 2 +-
.../Show-Statements/SHOW-PARTITION-ID.md | 10 +-
.../Show-Statements/SHOW-PARTITIONS.md | 20 +-
.../sql-reference/Show-Statements/SHOW-TABLETS.md | 7 +-
extension/logstash/README.md | 4 +-
.../org/apache/doris/common/io/BitmapValue.java | 14 +-
.../doris/common/jmockit/FieldReflection.java | 2 +-
.../doris/common/jmockit/MethodReflection.java | 2 +-
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 11 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 48 +-
fe/fe-core/src/main/cup/sql_parser.cup | 87 +-
.../main/java/org/apache/doris/alter/Alter.java | 12 +-
.../doris/alter/MaterializedViewHandler.java | 30 +-
.../java/org/apache/doris/alter/RollupJobV2.java | 29 +-
.../apache/doris/alter/SchemaChangeHandler.java | 14 +-
.../doris/analysis/AlterColumnStatsStmt.java | 1 -
.../org/apache/doris/analysis/AlterPolicyStmt.java | 6 +-
.../org/apache/doris/analysis/AnalyzeStmt.java | 14 +-
.../java/org/apache/doris/analysis/Analyzer.java | 13 +-
.../org/apache/doris/analysis/ArithmeticExpr.java | 25 +-
.../org/apache/doris/analysis/BaseTableRef.java | 1 +
.../org/apache/doris/analysis/BinaryPredicate.java | 13 +-
.../apache/doris/analysis/CancelExportStmt.java | 176 ++
.../org/apache/doris/analysis/CancelLoadStmt.java | 3 +
.../java/org/apache/doris/analysis/ColumnDef.java | 2 +-
.../apache/doris/analysis/ColumnPartitionDesc.java | 3 +
.../apache/doris/analysis/CreateCatalogStmt.java | 22 +-
.../doris/analysis/CreateMaterializedViewStmt.java | 82 +-
.../org/apache/doris/analysis/CreateTableStmt.java | 6 +-
.../org/apache/doris/analysis/DateLiteral.java | 9 +-
.../org/apache/doris/analysis/DecimalLiteral.java | 1 +
.../apache/doris/analysis/DistributionDesc.java | 2 +-
.../java/org/apache/doris/analysis/ExportStmt.java | 8 +-
.../main/java/org/apache/doris/analysis/Expr.java | 49 +
.../apache/doris/analysis/ExprSubstitutionMap.java | 10 +
.../org/apache/doris/analysis/FloatLiteral.java | 8 +-
.../apache/doris/analysis/FunctionCallExpr.java | 84 +-
.../doris/analysis/HashDistributionDesc.java | 17 +-
.../java/org/apache/doris/analysis/IntLiteral.java | 4 +-
.../org/apache/doris/analysis/JoinOperator.java | 5 +
.../org/apache/doris/analysis/LargeIntLiteral.java | 4 +-
.../doris/analysis/MVColumnBitmapUnionPattern.java | 6 +-
.../doris/analysis/MVColumnHLLUnionPattern.java | 2 +-
.../org/apache/doris/analysis/MVColumnItem.java | 78 +-
.../org/apache/doris/analysis/MatchPredicate.java | 243 ++
.../org/apache/doris/analysis/PartitionDesc.java | 4 +
.../doris/analysis/RandomDistributionDesc.java | 2 +-
.../apache/doris/analysis/RangePartitionDesc.java | 8 +-
.../java/org/apache/doris/analysis/SelectList.java | 9 +
.../java/org/apache/doris/analysis/SelectStmt.java | 88 +-
.../apache/doris/analysis/SetOperationStmt.java | 6 +
.../java/org/apache/doris/analysis/SetVar.java | 9 -
.../doris/analysis/ShowCreateCatalogStmt.java | 2 +-
.../apache/doris/analysis/SinglePartitionDesc.java | 4 +-
.../org/apache/doris/analysis/SlotDescriptor.java | 16 +-
.../java/org/apache/doris/analysis/SlotRef.java | 6 +-
.../org/apache/doris/analysis/StringLiteral.java | 5 +-
.../java/org/apache/doris/analysis/TableRef.java | 37 +
.../org/apache/doris/analysis/TableSnapshot.java | 70 +
.../doris/analysis/TableValuedFunctionRef.java | 8 +-
.../java/org/apache/doris/backup/S3Storage.java | 11 +-
.../doris/catalog/BuiltinAggregateFunctions.java | 88 +-
.../doris/catalog/BuiltinScalarFunctions.java | 111 +-
.../main/java/org/apache/doris/catalog/Column.java | 9 +-
.../org/apache/doris/catalog/DataProperty.java | 5 +-
.../java/org/apache/doris/catalog/DatabaseIf.java | 12 +
.../main/java/org/apache/doris/catalog/Env.java | 7 +
.../java/org/apache/doris/catalog/EsResource.java | 4 +
.../java/org/apache/doris/catalog/EsTable.java | 36 +-
.../java/org/apache/doris/catalog/FunctionSet.java | 9 +
.../apache/doris/catalog/FunctionSignature.java | 57 +-
.../java/org/apache/doris/catalog/HMSResource.java | 3 +-
.../doris/catalog/HiveMetaStoreClientHelper.java | 2 +-
.../doris/catalog/InternalSchemaInitializer.java | 35 +-
.../org/apache/doris/catalog/JdbcResource.java | 102 +-
.../doris/catalog/MaterializedIndexMeta.java | 10 +-
.../org/apache/doris/catalog/MaterializedView.java | 1 +
.../java/org/apache/doris/catalog/OdbcTable.java | 7 +
.../java/org/apache/doris/catalog/OlapTable.java | 6 +-
.../org/apache/doris/catalog/PrimitiveType.java | 1 +
.../java/org/apache/doris/catalog/S3Resource.java | 48 +
.../java/org/apache/doris/catalog/ScalarType.java | 27 +-
.../main/java/org/apache/doris/catalog/Type.java | 42 +-
.../doris/catalog/external/EsExternalTable.java | 8 +-
.../doris/catalog/external/ExternalDatabase.java | 5 +
.../catalog/external/HMSExternalDatabase.java | 10 +
.../doris/catalog/external/HMSExternalTable.java | 2 +-
.../doris/clone/DynamicPartitionScheduler.java | 4 +-
.../main/java/org/apache/doris/common/Config.java | 24 +-
.../java/org/apache/doris/common/ErrorCode.java | 5 +-
.../doris/common/profile/ProfileTreeBuilder.java | 20 +-
.../doris/common/util/DynamicPartitionUtil.java | 5 +-
.../org/apache/doris/common/util/TimeUtils.java | 2 +-
.../apache/doris/datasource/CatalogFactory.java | 63 +-
.../org/apache/doris/datasource/CatalogIf.java | 6 +-
.../org/apache/doris/datasource/CatalogMgr.java | 68 +-
.../apache/doris/datasource/CatalogProperty.java | 43 +-
.../apache/doris/datasource/EsExternalCatalog.java | 125 +-
.../apache/doris/datasource/ExternalCatalog.java | 9 +
.../doris/datasource/HMSClientException.java | 5 +-
.../doris/datasource/HMSExternalCatalog.java | 71 +-
.../apache/doris/datasource/InternalCatalog.java | 6 +-
.../doris/datasource/JdbcExternalCatalog.java | 107 +-
.../datasource/PooledHiveMetaStoreClient.java | 190 --
.../doris/datasource/hive/HiveMetaStoreClient.java | 2758 ++++++++++++++++++++
.../doris/datasource/hive/HiveVersionUtil.java | 75 +
.../datasource/hive/PooledHiveMetaStoreClient.java | 217 ++
.../datasource/hive/event/DropTableEvent.java | 89 +
.../doris/datasource/hive/event/EventFactory.java | 32 +
.../doris/datasource/hive/event/IgnoredEvent.java | 43 +
.../datasource/hive/event/MetastoreEvent.java | 203 ++
.../hive/event/MetastoreEventFactory.java | 81 +
.../datasource/hive/event/MetastoreEventType.java | 68 +
.../hive/event/MetastoreEventsProcessor.java | 151 ++
.../hive/event/MetastoreNotificationException.java | 37 +
.../event/MetastoreNotificationFetchException.java | 37 +
.../datasource/hive/event/MetastoreTableEvent.java | 50 +
.../doris/deploy/impl/LocalFileDeployManager.java | 2 +-
.../doris/external/elasticsearch/EsUtil.java | 3 +-
.../doris/external/elasticsearch/MappingPhase.java | 22 +-
.../iceberg/IcebergTableCreationRecordMgr.java | 2 +-
.../org/apache/doris/external/jdbc/JdbcClient.java | 255 +-
.../apache/doris/httpv2/rest/GetLogFileAction.java | 2 +-
.../java/org/apache/doris/journal/Journal.java | 3 +
.../org/apache/doris/journal/JournalEntity.java | 1 +
.../apache/doris/journal/bdbje/BDBJEJournal.java | 12 +
.../apache/doris/journal/local/LocalJournal.java | 5 +
.../main/java/org/apache/doris/load/DppConfig.java | 43 +
.../java/org/apache/doris/load/EtlJobInfo.java | 12 +-
.../main/java/org/apache/doris/load/ExportJob.java | 14 +-
.../main/java/org/apache/doris/load/ExportMgr.java | 61 +-
.../org/apache/doris/load/loadv2/SparkLoadJob.java | 9 +
.../doris/load/sync/canal/CanalSyncChannel.java | 5 +-
.../java/org/apache/doris/master/Checkpoint.java | 2 +
.../org/apache/doris/metric/LongCounterMetric.java | 9 +
.../java/org/apache/doris/metric/MetricRepo.java | 15 +-
.../apache/doris/mysql/MysqlHandshakePacket.java | 4 +-
.../doris/mysql/privilege/UserPrivTable.java | 1 +
.../doris/nereids/analyzer/NereidsAnalyzer.java | 2 +
.../nereids/analyzer/UnboundOneRowRelation.java | 4 +
.../doris/nereids/analyzer/UnboundRelation.java | 21 +-
.../doris/nereids/analyzer/UnboundTVFRelation.java | 18 +-
.../glue/translator/ExpressionTranslator.java | 94 +-
.../glue/translator/PhysicalPlanTranslator.java | 77 +-
.../glue/translator/PlanTranslatorContext.java | 17 +-
.../AdjustAggregateNullableForEmptySetJob.java | 34 +
.../AdjustApplyFromCorrelateToUnCorrelateJob.java | 2 +
.../doris/nereids/jobs/batch/AnalyzeRulesJob.java | 14 +-
.../doris/nereids/jobs/batch/BatchRulesJob.java | 9 +
.../jobs/batch/CheckLegalityAfterRewrite.java | 34 +
.../batch/CheckLegalityBeforeTypeCoercion.java | 34 +
.../jobs/batch/NereidsRewriteJobExecutor.java | 20 +-
.../doris/nereids/jobs/batch/TypeCoercionJob.java | 5 +
.../nereids/jobs/cascades/CostAndEnforcerJob.java | 42 +-
.../java/org/apache/doris/nereids/memo/Group.java | 45 +-
.../java/org/apache/doris/nereids/memo/Memo.java | 17 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 121 +-
.../nereids/processor/post/PlanPostProcessors.java | 5 +-
.../properties/ChildrenPropertiesRegulator.java | 85 +-
.../nereids/properties/DistributionSpecHash.java | 19 +-
.../apache/doris/nereids/properties/OrderKey.java | 6 +-
.../org/apache/doris/nereids/rules/RuleType.java | 9 +
.../AdjustAggregateNullableForEmptySet.java | 80 +
.../rules/analysis/AvgDistinctToSumDivCount.java | 70 +
.../doris/nereids/rules/analysis/BindFunction.java | 65 +-
.../doris/nereids/rules/analysis/BindRelation.java | 19 +-
.../nereids/rules/analysis/BindSlotReference.java | 260 +-
.../nereids/rules/analysis/CheckAfterRewrite.java | 19 +
.../nereids/rules/analysis/CheckAnalysis.java | 28 +-
.../nereids/rules/analysis/FillUpMissingSlots.java | 3 +-
.../nereids/rules/analysis/NormalizeRepeat.java | 11 +-
.../ResolveOrdinalInOrderByAndGroupBy.java | 10 +-
.../exploration/join/InnerJoinLAsscomProject.java | 1 +
.../rules/exploration/join/JoinReorderCommon.java | 54 +
.../exploration/join/OuterJoinLAsscomProject.java | 81 +-
.../join/SemiJoinLogicalJoinTranspose.java | 16 +-
.../join/SemiJoinLogicalJoinTransposeProject.java | 20 +-
.../join/SemiJoinSemiJoinTransposeProject.java | 1 +
.../rewrite/ExpressionNormalization.java | 4 +-
.../rules/CharacterLiteralTypeCoercion.java | 3 +
.../rewrite/rules/FoldConstantRuleOnFE.java | 61 +-
.../rewrite/rules/SimplifyArithmeticRule.java | 9 +-
.../rewrite/rules/SupportJavaDateFormatter.java | 87 +
.../expression/rewrite/rules/TypeCoercion.java | 47 +-
.../LogicalOlapScanToPhysicalOlapScan.java | 17 +-
.../mv/AbstractSelectMaterializedIndexRule.java | 11 +-
.../mv/SelectMaterializedIndexWithAggregate.java | 230 +-
.../nereids/rules/rewrite/AggregateStrategies.java | 24 +-
.../logical/ApplyPullFilterOnProjectUnderAgg.java | 4 +-
.../rewrite/logical/CountDistinctRewrite.java | 4 +-
.../logical/EliminateFilterUnderApplyProject.java | 97 +
.../rewrite/logical/EliminateGroupByConstant.java | 2 +-
.../logical/ExtractFilterFromCrossJoin.java | 51 +
.../rules/rewrite/logical/InApplyToJoin.java | 14 +-
.../rules/rewrite/logical/NormalizeAggregate.java | 9 +-
.../rules/rewrite/logical/NormalizeToSlot.java | 8 +-
.../rewrite/logical/PruneOlapScanPartition.java | 9 +-
.../rewrite/logical/PushApplyUnderProject.java | 29 +-
.../rules/rewrite/logical/ScalarApplyToJoin.java | 12 +
.../doris/nereids/stats/ExpressionEstimation.java | 9 +-
.../doris/nereids/stats/StatsCalculator.java | 32 +-
.../trees/expressions/AggregateExpression.java | 8 +-
.../trees/expressions/BinaryArithmetic.java | 16 +-
.../doris/nereids/trees/expressions/Cast.java | 4 +-
.../doris/nereids/trees/expressions/Divide.java | 13 +
.../nereids/trees/expressions/Expression.java | 2 +-
.../nereids/trees/expressions/OrderExpression.java | 100 +
.../nereids/trees/expressions/UnaryArithmetic.java | 3 +-
.../nereids/trees/expressions/VariableDesc.java | 104 +
.../functions/BitmapIntersectFunction.java | 45 +
.../trees/expressions/functions/BoundFunction.java | 151 +-
.../expressions/functions/ComputePrecision.java | 25 +
.../functions/ComputePrecisionForRound.java | 56 +
.../functions/ComputePrecisionForSum.java | 37 +
.../expressions/functions/ComputeSignature.java | 35 +-
.../functions/ComputeSignatureHelper.java | 151 ++
.../functions/DateTimeWithPrecision.java | 2 +-
.../functions/ExplicitlyCastableSignature.java | 16 +-
.../expressions/functions/ExpressionTrait.java | 13 +
.../expressions/functions/IdenticalSignature.java | 10 +-
.../functions/ImplicitlyCastableSignature.java | 24 +-
.../functions/NullOrIdenticalSignature.java | 12 +-
.../functions/agg/AggregateFunction.java | 41 +-
.../trees/expressions/functions/agg/AnyValue.java | 75 +
.../functions/agg/ApproxCountDistinct.java | 76 -
.../trees/expressions/functions/agg/Avg.java | 110 +-
.../expressions/functions/agg/AvgWeighted.java | 87 +
.../expressions/functions/agg/BitmapFunction.java | 34 +
.../expressions/functions/agg/BitmapIntersect.java | 35 +-
.../expressions/functions/agg/BitmapUnion.java | 35 +-
.../functions/agg/BitmapUnionCount.java | 39 +-
.../expressions/functions/agg/BitmapUnionInt.java | 45 +-
.../expressions/functions/agg/CollectList.java | 110 +
.../expressions/functions/agg/CollectSet.java | 110 +
.../trees/expressions/functions/agg/Count.java | 66 +-
.../expressions/functions/agg/GroupBitAnd.java | 47 +-
.../expressions/functions/agg/GroupBitOr.java | 45 +-
.../expressions/functions/agg/GroupBitXor.java | 44 +-
.../expressions/functions/agg/GroupBitmapXor.java | 48 +-
.../expressions/functions/agg/GroupConcat.java | 144 +
.../trees/expressions/functions/agg/Histogram.java | 104 +
.../expressions/functions/agg/HllFunction.java | 32 +
.../trees/expressions/functions/agg/HllUnion.java | 39 +-
.../expressions/functions/agg/HllUnionAgg.java | 39 +-
.../expressions/functions/agg/IntersectCount.java | 81 +
.../trees/expressions/functions/agg/Max.java | 31 +-
.../trees/expressions/functions/agg/MaxBy.java | 92 +
.../trees/expressions/functions/agg/Min.java | 31 +-
.../trees/expressions/functions/agg/MinBy.java | 92 +
.../functions/agg/MultiDistinctCount.java | 48 +-
.../functions/agg/MultiDistinctSum.java | 32 +-
.../trees/expressions/functions/agg/Ndv.java | 83 +-
.../functions/agg/NullableAggregateFunction.java | 75 +
.../functions/agg/OrthogonalBitmapFunction.java | 54 +
.../functions/agg/OrthogonalBitmapIntersect.java | 78 +
.../agg/OrthogonalBitmapIntersectCount.java | 66 +
.../functions/agg/OrthogonalBitmapUnionCount.java | 76 +
.../expressions/functions/agg/Percentile.java | 85 +
.../functions/agg/PercentileApprox.java | 110 +
.../expressions/functions/agg/PercentileArray.java | 78 +
.../expressions/functions/agg/QuantileUnion.java | 86 +
.../trees/expressions/functions/agg/Retention.java | 94 +
.../expressions/functions/agg/SequenceCount.java | 87 +
.../functions/agg/SequenceFunction.java | 54 +
.../expressions/functions/agg/SequenceMatch.java | 86 +
.../functions/agg/StdDevOrVarianceFunction.java | 31 +
.../trees/expressions/functions/agg/Stddev.java | 97 +
.../expressions/functions/agg/StddevSamp.java | 89 +
.../trees/expressions/functions/agg/Sum.java | 86 +-
.../trees/expressions/functions/agg/TopN.java | 115 +
.../trees/expressions/functions/agg/TopNArray.java | 158 ++
.../expressions/functions/agg/TopNWeighted.java | 174 ++
.../trees/expressions/functions/agg/Variance.java | 97 +
.../expressions/functions/agg/VarianceSamp.java | 88 +
.../expressions/functions/agg/WindowFunnel.java | 128 +
.../trees/expressions/functions/scalar/Abs.java | 2 +-
.../trees/expressions/functions/scalar/Acos.java | 2 +-
.../functions/scalar/AesCryptoFunction.java | 75 +
.../expressions/functions/scalar/AesDecrypt.java | 34 +-
.../expressions/functions/scalar/AesEncrypt.java | 37 +-
.../scalar/AppendTrailingCharIfAbsent.java | 2 +-
.../trees/expressions/functions/scalar/Array.java | 126 +
.../expressions/functions/scalar/ArrayAvg.java | 84 +
.../expressions/functions/scalar/ArrayCompact.java | 98 +
.../functions/scalar/ArrayContains.java | 103 +
.../functions/scalar/ArrayDifference.java | 84 +
.../functions/scalar/ArrayDistinct.java | 99 +
.../functions/scalar/ArrayEnumerate.java | 97 +
.../expressions/functions/scalar/ArrayExcept.java | 111 +
.../functions/scalar/ArrayIntersect.java | 111 +
.../expressions/functions/scalar/ArrayJoin.java | 159 ++
.../expressions/functions/scalar/ArrayMax.java | 92 +
.../expressions/functions/scalar/ArrayMin.java | 92 +
.../expressions/functions/scalar/ArrayPopback.java | 95 +
.../functions/scalar/ArrayPosition.java | 100 +
.../expressions/functions/scalar/ArrayProduct.java | 84 +
.../expressions/functions/scalar/ArrayRange.java | 93 +
.../expressions/functions/scalar/ArrayRemove.java | 112 +
.../expressions/functions/scalar/ArraySize.java | 69 +
.../expressions/functions/scalar/ArraySlice.java | 143 +
.../expressions/functions/scalar/ArraySort.java | 100 +
.../expressions/functions/scalar/ArraySum.java | 84 +
.../expressions/functions/scalar/ArrayUnion.java | 111 +
.../functions/scalar/ArrayWithConstant.java | 95 +
.../functions/scalar/ArraysOverlap.java | 111 +
.../trees/expressions/functions/scalar/Ascii.java | 2 +-
.../trees/expressions/functions/scalar/Asin.java | 2 +-
.../trees/expressions/functions/scalar/Atan.java | 2 +-
.../trees/expressions/functions/scalar/Bin.java | 2 +-
.../expressions/functions/scalar/BitLength.java | 2 +-
.../expressions/functions/scalar/BitmapAnd.java | 2 +-
.../functions/scalar/BitmapAndCount.java | 2 +-
.../expressions/functions/scalar/BitmapAndNot.java | 2 +-
.../functions/scalar/BitmapAndNotCount.java | 2 +-
.../functions/scalar/BitmapContains.java | 2 +-
.../expressions/functions/scalar/BitmapCount.java | 5 +-
.../expressions/functions/scalar/BitmapEmpty.java | 2 +-
.../functions/scalar/BitmapFromArray.java | 76 +
.../functions/scalar/BitmapFromString.java | 2 +-
.../expressions/functions/scalar/BitmapHasAll.java | 2 +-
.../expressions/functions/scalar/BitmapHasAny.java | 2 +-
.../expressions/functions/scalar/BitmapHash.java | 2 +-
.../expressions/functions/scalar/BitmapHash64.java | 2 +-
.../expressions/functions/scalar/BitmapMax.java | 2 +-
.../expressions/functions/scalar/BitmapMin.java | 2 +-
.../expressions/functions/scalar/BitmapNot.java | 2 +-
.../expressions/functions/scalar/BitmapOr.java | 2 +-
.../functions/scalar/BitmapOrCount.java | 2 +-
.../functions/scalar/BitmapSubsetInRange.java | 2 +-
.../functions/scalar/BitmapSubsetLimit.java | 2 +-
.../functions/scalar/BitmapToArray.java | 70 +
.../functions/scalar/BitmapToString.java | 2 +-
.../expressions/functions/scalar/BitmapXor.java | 2 +-
.../functions/scalar/BitmapXorCount.java | 2 +-
.../expressions/functions/scalar/Cardinality.java | 69 +
.../trees/expressions/functions/scalar/Cbrt.java | 2 +-
.../trees/expressions/functions/scalar/Ceil.java | 8 +-
.../expressions/functions/scalar/Ceiling.java | 2 +-
.../expressions/functions/scalar/CharLength.java | 71 -
.../functions/scalar/CharacterLength.java | 2 +-
.../expressions/functions/scalar/Coalesce.java | 2 +-
.../scalar/CompatibleTypeArrayFunction.java | 65 +
.../trees/expressions/functions/scalar/Concat.java | 2 +-
.../expressions/functions/scalar/ConcatWs.java | 13 +-
.../expressions/functions/scalar/ConnectionId.java | 54 +
.../trees/expressions/functions/scalar/Conv.java | 2 +-
.../expressions/functions/scalar/ConvertTo.java | 78 +
.../expressions/functions/scalar/ConvertTz.java | 2 +-
.../trees/expressions/functions/scalar/Cos.java | 2 +-
.../expressions/functions/scalar/CountEqual.java | 100 +
.../functions/scalar/CryptoFunction.java | 93 +
.../expressions/functions/scalar/CurrentDate.java | 2 +-
.../expressions/functions/scalar/CurrentTime.java | 2 +-
.../functions/scalar/CurrentTimestamp.java | 2 +-
.../expressions/functions/scalar/CurrentUser.java | 54 +
.../expressions/functions/scalar/Curtime.java | 2 +-
.../expressions/functions/scalar/Database.java | 54 +
.../trees/expressions/functions/scalar/Date.java | 2 +-
.../expressions/functions/scalar/DateDiff.java | 2 +-
.../expressions/functions/scalar/DateFormat.java | 2 +-
.../expressions/functions/scalar/DateTrunc.java | 2 +-
.../trees/expressions/functions/scalar/DateV2.java | 2 +-
.../trees/expressions/functions/scalar/Day.java | 2 +-
.../expressions/functions/scalar/DayCeil.java | 2 +-
.../expressions/functions/scalar/DayFloor.java | 2 +-
.../expressions/functions/scalar/DayName.java | 2 +-
.../expressions/functions/scalar/DayOfMonth.java | 2 +-
.../expressions/functions/scalar/DayOfWeek.java | 2 +-
.../expressions/functions/scalar/DayOfYear.java | 2 +-
.../expressions/functions/scalar/DaysAdd.java | 13 +-
.../expressions/functions/scalar/DaysDiff.java | 2 +-
.../expressions/functions/scalar/DaysSub.java | 13 +-
.../trees/expressions/functions/scalar/Dceil.java | 2 +-
.../expressions/functions/scalar/Degrees.java | 2 +-
.../trees/expressions/functions/scalar/Dexp.java | 2 +-
.../trees/expressions/functions/scalar/Dfloor.java | 2 +-
.../trees/expressions/functions/scalar/Dlog1.java | 2 +-
.../trees/expressions/functions/scalar/Dlog10.java | 2 +-
.../trees/expressions/functions/scalar/Domain.java | 68 +
.../functions/scalar/DomainWithoutWww.java | 68 +
.../trees/expressions/functions/scalar/Dpow.java | 2 +-
.../trees/expressions/functions/scalar/Dround.java | 5 +-
.../trees/expressions/functions/scalar/Dsqrt.java | 2 +-
.../trees/expressions/functions/scalar/E.java | 2 +-
.../expressions/functions/scalar/ElementAt.java | 100 +
.../functions/scalar/ElementExtract.java | 100 +
.../expressions/functions/scalar/ElementSlice.java | 143 +
.../trees/expressions/functions/scalar/Elt.java | 2 +-
.../expressions/functions/scalar/EndsWith.java | 2 +-
.../expressions/functions/scalar/EsQuery.java | 2 +-
.../trees/expressions/functions/scalar/Exp.java | 2 +-
.../functions/scalar/ExtractUrlParameter.java | 2 +-
.../trees/expressions/functions/scalar/Field.java | 102 +
.../expressions/functions/scalar/FindInSet.java | 2 +-
.../trees/expressions/functions/scalar/Floor.java | 8 +-
.../trees/expressions/functions/scalar/Fmod.java | 2 +-
.../trees/expressions/functions/scalar/Fpow.java | 2 +-
.../expressions/functions/scalar/FromBase64.java | 2 +-
.../expressions/functions/scalar/FromDays.java | 2 +-
.../expressions/functions/scalar/FromUnixtime.java | 2 +-
.../functions/scalar/GetJsonDouble.java | 2 +-
.../expressions/functions/scalar/GetJsonInt.java | 2 +-
.../functions/scalar/GetJsonString.java | 2 +-
.../expressions/functions/scalar/Greatest.java | 2 +-
.../trees/expressions/functions/scalar/Hex.java | 2 +-
.../functions/scalar/HllCardinality.java | 5 +-
.../expressions/functions/scalar/HllEmpty.java | 2 +-
.../expressions/functions/scalar/HllHash.java | 2 +-
.../trees/expressions/functions/scalar/Hour.java | 2 +-
.../expressions/functions/scalar/HourCeil.java | 2 +-
.../expressions/functions/scalar/HourFloor.java | 2 +-
.../expressions/functions/scalar/HoursDiff.java | 2 +-
.../trees/expressions/functions/scalar/If.java | 102 +-
.../expressions/functions/scalar/Initcap.java | 2 +-
.../trees/expressions/functions/scalar/Instr.java | 2 +-
.../expressions/functions/scalar/JsonArray.java | 2 +-
.../expressions/functions/scalar/JsonObject.java | 12 +-
.../expressions/functions/scalar/JsonQuote.java | 2 +-
.../functions/scalar/JsonbExistsPath.java | 2 +-
.../expressions/functions/scalar/JsonbExtract.java | 2 +-
.../functions/scalar/JsonbExtractBigint.java | 2 +-
.../functions/scalar/JsonbExtractBool.java | 2 +-
.../functions/scalar/JsonbExtractDouble.java | 2 +-
.../functions/scalar/JsonbExtractInt.java | 2 +-
.../functions/scalar/JsonbExtractIsnull.java | 2 +-
.../functions/scalar/JsonbExtractString.java | 2 +-
.../expressions/functions/scalar/JsonbParse.java | 2 +-
.../functions/scalar/JsonbParseErrorToInvalid.java | 2 +-
.../functions/scalar/JsonbParseErrorToNull.java | 2 +-
.../functions/scalar/JsonbParseErrorToValue.java | 2 +-
.../functions/scalar/JsonbParseNotnull.java | 2 +-
.../scalar/JsonbParseNotnullErrorToInvalid.java | 2 +-
.../scalar/JsonbParseNotnullErrorToValue.java | 2 +-
.../functions/scalar/JsonbParseNullable.java | 2 +-
.../scalar/JsonbParseNullableErrorToInvalid.java | 2 +-
.../scalar/JsonbParseNullableErrorToNull.java | 2 +-
.../scalar/JsonbParseNullableErrorToValue.java | 2 +-
.../expressions/functions/scalar/JsonbType.java | 2 +-
.../expressions/functions/scalar/LastDay.java | 74 +
.../trees/expressions/functions/scalar/Least.java | 2 +-
.../trees/expressions/functions/scalar/Left.java | 2 +-
.../trees/expressions/functions/scalar/Length.java | 2 +-
.../trees/expressions/functions/scalar/Ln.java | 2 +-
.../expressions/functions/scalar/LocalTime.java | 2 +-
.../functions/scalar/LocalTimestamp.java | 2 +-
.../trees/expressions/functions/scalar/Locate.java | 2 +-
.../trees/expressions/functions/scalar/Log.java | 2 +-
.../trees/expressions/functions/scalar/Log10.java | 2 +-
.../trees/expressions/functions/scalar/Log2.java | 2 +-
.../trees/expressions/functions/scalar/Lower.java | 2 +-
.../trees/expressions/functions/scalar/Lpad.java | 2 +-
.../trees/expressions/functions/scalar/Ltrim.java | 2 +-
.../expressions/functions/scalar/MakeDate.java | 2 +-
.../trees/expressions/functions/scalar/Md5.java | 2 +-
.../trees/expressions/functions/scalar/Md5Sum.java | 2 +-
.../trees/expressions/functions/scalar/Minute.java | 2 +-
.../expressions/functions/scalar/MinuteCeil.java | 2 +-
.../expressions/functions/scalar/MinuteFloor.java | 2 +-
.../expressions/functions/scalar/MinutesDiff.java | 2 +-
.../expressions/functions/scalar/MoneyFormat.java | 2 +-
.../trees/expressions/functions/scalar/Month.java | 2 +-
.../expressions/functions/scalar/MonthCeil.java | 2 +-
.../expressions/functions/scalar/MonthFloor.java | 2 +-
.../expressions/functions/scalar/MonthName.java | 2 +-
.../expressions/functions/scalar/MonthsDiff.java | 2 +-
.../functions/scalar/MultiMatchAny.java | 70 +
.../functions/scalar/MultiSearchAllPositions.java | 71 +
.../functions/scalar/MurmurHash332.java | 2 +-
.../functions/scalar/MurmurHash364.java | 2 +-
.../expressions/functions/scalar/Negative.java | 2 +-
.../functions/scalar/NotNullOrEmpty.java | 2 +-
.../trees/expressions/functions/scalar/Now.java | 2 +-
.../trees/expressions/functions/scalar/NullIf.java | 2 +-
.../expressions/functions/scalar/NullOrEmpty.java | 2 +-
.../trees/expressions/functions/scalar/Nvl.java | 2 +-
.../expressions/functions/scalar/ParseUrl.java | 2 +-
.../trees/expressions/functions/scalar/Pi.java | 2 +-
.../trees/expressions/functions/scalar/Pmod.java | 2 +-
.../expressions/functions/scalar/Positive.java | 2 +-
.../trees/expressions/functions/scalar/Pow.java | 2 +-
.../trees/expressions/functions/scalar/Power.java | 2 +-
.../expressions/functions/scalar/Protocol.java | 68 +
.../functions/scalar/QuantilePercent.java | 2 +-
.../expressions/functions/scalar/Quarter.java | 2 +-
.../expressions/functions/scalar/Radians.java | 2 +-
.../trees/expressions/functions/scalar/Random.java | 2 +-
.../functions/scalar/RegexpExtract.java | 2 +-
.../functions/scalar/RegexpExtractAll.java | 71 +
.../functions/scalar/RegexpReplace.java | 2 +-
.../functions/scalar/RegexpReplaceOne.java | 72 +
.../trees/expressions/functions/scalar/Repeat.java | 2 +-
.../expressions/functions/scalar/Replace.java | 2 +-
.../expressions/functions/scalar/Reverse.java | 37 +-
.../trees/expressions/functions/scalar/Right.java | 2 +-
.../trees/expressions/functions/scalar/Round.java | 8 +-
.../expressions/functions/scalar/RoundBankers.java | 8 +-
.../trees/expressions/functions/scalar/Rpad.java | 2 +-
.../trees/expressions/functions/scalar/Rtrim.java | 2 +-
.../functions/scalar/RunningDifference.java | 94 +
.../trees/expressions/functions/scalar/Second.java | 2 +-
.../expressions/functions/scalar/SecondCeil.java | 2 +-
.../expressions/functions/scalar/SecondFloor.java | 2 +-
.../expressions/functions/scalar/SecondsDiff.java | 2 +-
.../trees/expressions/functions/scalar/Sign.java | 2 +-
.../trees/expressions/functions/scalar/Sin.java | 2 +-
.../trees/expressions/functions/scalar/Size.java | 69 +
.../trees/expressions/functions/scalar/Sleep.java | 2 +-
.../trees/expressions/functions/scalar/Sm3.java | 2 +-
.../trees/expressions/functions/scalar/Sm3sum.java | 2 +-
.../functions/scalar/Sm4CryptoFunction.java | 57 +
.../expressions/functions/scalar/Sm4Decrypt.java | 39 +-
.../expressions/functions/scalar/Sm4Encrypt.java | 34 +-
.../trees/expressions/functions/scalar/Space.java | 2 +-
.../expressions/functions/scalar/SplitByChar.java | 71 +
.../functions/scalar/SplitByString.java | 71 +
.../expressions/functions/scalar/SplitPart.java | 2 +-
.../trees/expressions/functions/scalar/Sqrt.java | 2 +-
.../expressions/functions/scalar/StAstext.java | 2 +-
.../expressions/functions/scalar/StAswkt.java | 2 +-
.../expressions/functions/scalar/StCircle.java | 2 +-
.../expressions/functions/scalar/StContains.java | 2 +-
.../functions/scalar/StDistanceSphere.java | 2 +-
.../functions/scalar/StGeometryfromtext.java | 2 +-
.../functions/scalar/StGeomfromtext.java | 2 +-
.../functions/scalar/StLinefromtext.java | 2 +-
.../functions/scalar/StLinestringfromtext.java | 2 +-
.../expressions/functions/scalar/StPoint.java | 2 +-
.../functions/scalar/StPolyfromtext.java | 2 +-
.../expressions/functions/scalar/StPolygon.java | 2 +-
.../functions/scalar/StPolygonfromtext.java | 2 +-
.../trees/expressions/functions/scalar/StX.java | 2 +-
.../trees/expressions/functions/scalar/StY.java | 2 +-
.../expressions/functions/scalar/StartsWith.java | 2 +-
.../expressions/functions/scalar/StrLeft.java | 2 +-
.../expressions/functions/scalar/StrRight.java | 2 +-
.../expressions/functions/scalar/StrToDate.java | 4 +-
.../expressions/functions/scalar/SubBitmap.java | 2 +-
.../expressions/functions/scalar/Substring.java | 4 +-
.../functions/scalar/SubstringIndex.java | 73 +
.../trees/expressions/functions/scalar/Tan.java | 2 +-
.../expressions/functions/scalar/TimeDiff.java | 2 +-
.../expressions/functions/scalar/Timestamp.java | 11 +-
.../expressions/functions/scalar/ToBase64.java | 2 +-
.../expressions/functions/scalar/ToBitmap.java | 2 +-
.../functions/scalar/ToBitmapWithCheck.java | 2 +-
.../trees/expressions/functions/scalar/ToDate.java | 2 +-
.../expressions/functions/scalar/ToDateV2.java | 2 +-
.../trees/expressions/functions/scalar/ToDays.java | 2 +-
.../expressions/functions/scalar/ToMonday.java | 74 +
.../functions/scalar/ToQuantileState.java | 10 +-
.../trees/expressions/functions/scalar/Trim.java | 2 +-
.../expressions/functions/scalar/Truncate.java | 5 +-
.../trees/expressions/functions/scalar/Unhex.java | 2 +-
.../functions/scalar/UnixTimestamp.java | 2 +-
.../trees/expressions/functions/scalar/Upper.java | 2 +-
.../trees/expressions/functions/scalar/User.java | 55 +
.../expressions/functions/scalar/UtcTimestamp.java | 2 +-
.../trees/expressions/functions/scalar/Uuid.java | 58 +
.../expressions/functions/scalar/Version.java | 2 +-
.../trees/expressions/functions/scalar/Week.java | 2 +-
.../expressions/functions/scalar/WeekCeil.java | 2 +-
.../expressions/functions/scalar/WeekFloor.java | 2 +-
.../expressions/functions/scalar/WeekOfYear.java | 2 +-
.../expressions/functions/scalar/Weekday.java | 2 +-
.../expressions/functions/scalar/WeeksDiff.java | 2 +-
.../trees/expressions/functions/scalar/Year.java | 2 +-
.../expressions/functions/scalar/YearCeil.java | 2 +-
.../expressions/functions/scalar/YearFloor.java | 2 +-
.../expressions/functions/scalar/YearWeek.java | 2 +-
.../expressions/functions/scalar/YearsDiff.java | 2 +-
.../trees/expressions/functions/table/Numbers.java | 2 +-
.../trees/expressions/literal/ArrayLiteral.java | 98 +
.../expressions/literal/IntegerLikeLiteral.java | 4 +
.../nereids/trees/expressions/literal/Literal.java | 34 +-
.../visitor/AggregateFunctionVisitor.java | 200 +-
.../expressions/visitor/ExpressionVisitor.java | 15 +
.../expressions/visitor/ScalarFunctionVisitor.java | 289 +-
.../trees/plans/logical/LogicalAggregate.java | 33 +-
.../nereids/trees/plans/logical/LogicalApply.java | 4 +
.../trees/plans/logical/LogicalSetOperation.java | 2 +-
.../trees/plans/logical/LogicalSubQueryAlias.java | 37 +-
.../plans/physical/PhysicalOneRowRelation.java | 9 +-
.../org/apache/doris/nereids/types/ArrayType.java | 10 +-
.../org/apache/doris/nereids/types/BigIntType.java | 3 +-
.../org/apache/doris/nereids/types/DataType.java | 140 +-
.../apache/doris/nereids/types/IntegerType.java | 3 +-
.../apache/doris/nereids/types/SmallIntType.java | 3 +-
.../apache/doris/nereids/types/TinyIntType.java | 3 +-
.../types/coercion/FollowToArgumentType.java | 55 +
.../nereids/types/coercion/Int16OrLessType.java | 22 +
.../nereids/types/coercion/Int32OrLessType.java | 22 +
.../nereids/types/coercion/Int64OrLessType.java | 22 +
.../apache/doris/nereids/util/ExpressionUtils.java | 14 +
.../org/apache/doris/nereids/util/JoinUtils.java | 11 +-
.../doris/nereids/util/TypeCoercionUtils.java | 51 +-
.../java/org/apache/doris/nereids/util/Utils.java | 41 +-
.../java/org/apache/doris/persist/EditLog.java | 19 +-
.../org/apache/doris/persist/OperationType.java | 2 +
.../doris/persist/TableAddOrDropColumnsInfo.java | 2 +-
.../doris/persist/TableRenameColumnInfo.java | 2 +-
.../planner/BackendPartitionedSchemaScanNode.java | 11 +-
.../apache/doris/planner/DistributedPlanner.java | 1 +
.../java/org/apache/doris/planner/EsScanNode.java | 13 +-
.../org/apache/doris/planner/HashJoinNode.java | 1 +
.../org/apache/doris/planner/JdbcScanNode.java | 6 -
.../org/apache/doris/planner/JoinNodeBase.java | 8 +
.../apache/doris/planner/ListPartitionPruner.java | 252 --
.../doris/planner/MaterializedViewSelector.java | 195 +-
.../apache/doris/planner/NestedLoopJoinNode.java | 2 +-
.../org/apache/doris/planner/OlapScanNode.java | 43 +-
.../org/apache/doris/planner/OriginalPlanner.java | 32 +-
.../org/apache/doris/planner/PlanFragment.java | 10 +
.../apache/doris/planner/RangePartitionPruner.java | 194 --
.../org/apache/doris/planner/RuntimeFilter.java | 4 +
.../doris/planner/RuntimeFilterGenerator.java | 2 +-
.../java/org/apache/doris/planner/ScanNode.java | 23 +-
.../org/apache/doris/planner/SchemaScanNode.java | 7 +
.../org/apache/doris/planner/SetOperationNode.java | 3 +
.../apache/doris/planner/SingleNodePlanner.java | 6 +-
.../java/org/apache/doris/planner/SortNode.java | 22 +-
.../apache/doris/planner/StreamLoadPlanner.java | 19 +-
.../planner/external/IcebergScanProvider.java | 109 +-
.../doris/planner/external/IcebergSplit.java | 2 -
.../doris/planner/external/MetadataScanNode.java | 114 +
.../java/org/apache/doris/plugin/AuditEvent.java | 15 +-
.../org/apache/doris/policy/StoragePolicy.java | 2 +-
.../java/org/apache/doris/qe/ConnectProcessor.java | 19 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 49 +-
.../main/java/org/apache/doris/qe/DdlExecutor.java | 3 +
.../main/java/org/apache/doris/qe/QueryState.java | 11 +
.../java/org/apache/doris/qe/SessionVariable.java | 66 +-
.../java/org/apache/doris/qe/ShowExecutor.java | 3 +
.../java/org/apache/doris/qe/SqlModeHelper.java | 14 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 15 +-
.../main/java/org/apache/doris/qe/VariableMgr.java | 63 +-
.../apache/doris/rewrite/MatchPredicateRule.java | 41 +
.../RoundLiteralInBinaryPredicatesRule.java | 5 +-
.../doris/rewrite/mvrewrite/CountFieldToSum.java | 2 +-
.../doris/rewrite/mvrewrite/ExprToSlotRefRule.java | 125 +
.../rewrite/mvrewrite/HLLHashToSlotRefRule.java | 5 +-
.../doris/rewrite/mvrewrite/SlotRefEqualRule.java | 3 +-
.../rewrite/mvrewrite/ToBitmapToSlotRefRule.java | 4 +-
.../apache/doris/service/FrontendServiceImpl.java | 96 +-
.../apache/doris/statistics/AnalysisManager.java | 7 +-
.../apache/doris/statistics/AnalysisTaskInfo.java | 7 +-
.../apache/doris/statistics/BaseAnalysisTask.java | 7 +-
.../java/org/apache/doris/statistics/Bucket.java | 25 +
.../apache/doris/statistics/ColumnStatistic.java | 42 +-
.../doris/statistics/ColumnStatisticBuilder.java | 13 +-
.../org/apache/doris/statistics/Histogram.java | 187 +-
.../apache/doris/statistics/HistogramBuilder.java | 100 +
.../org/apache/doris/statistics/HistogramTask.java | 90 +
.../apache/doris/statistics/HiveAnalysisTask.java | 4 +-
.../doris/statistics/IcebergAnalysisTask.java | 2 +-
.../org/apache/doris/statistics/Statistic.java | 55 +
.../doris/statistics/StatisticConstants.java | 2 +
.../apache/doris/statistics/StatisticalType.java | 1 +
.../apache/doris/statistics/StatisticsCache.java | 36 +-
.../doris/statistics/StatisticsCacheLoader.java | 40 +-
.../doris/statistics/StatisticsRepository.java | 20 +-
.../doris/statistics/util/StatisticsUtil.java | 35 +
.../tablefunction/HdfsTableValuedFunction.java | 6 +-
.../tablefunction/IcebergTableValuedFunction.java | 123 +
.../tablefunction/MetadataTableValuedFunction.java | 46 +
.../doris/tablefunction/S3TableValuedFunction.java | 4 +-
.../doris/tablefunction/TableValuedFunctionIf.java | 2 +
.../org/apache/doris/task/ExportExportingTask.java | 26 +-
.../doris/task/NotifyUpdateStoragePolicyTask.java | 4 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 12 +
.../java/org/apache/doris/alter/AlterTest.java | 4 +-
.../doris/alter/MaterializedViewHandlerTest.java | 74 +-
.../doris/analysis/CancelExportStmtTest.java | 190 ++
.../doris/analysis/CreateCatalogStmtTest.java | 8 +-
.../analysis/CreateMaterializedViewStmtTest.java | 99 +-
.../analysis/CreateTableAsSelectStmtTest.java | 17 +-
.../apache/doris/analysis/CreateTableStmtTest.java | 14 +-
.../java/org/apache/doris/analysis/ExprTest.java | 4 -
.../doris/analysis/ListPartitionPrunerTest.java | 56 +-
.../analysis/MVColumnBitmapUnionPatternTest.java | 5 +
.../analysis/MVColumnHLLUnionPatternTest.java | 5 +
.../doris/analysis/PartitionPruneTestBase.java | 21 +-
.../org/apache/doris/analysis/QueryStmtTest.java | 6 +-
.../doris/analysis/RangePartitionPruneTest.java | 95 +-
.../org/apache/doris/catalog/CatalogTestUtil.java | 6 +-
.../org/apache/doris/catalog/CreateTableTest.java | 41 +
.../org/apache/doris/catalog/DataPropertyTest.java | 2 +-
.../org/apache/doris/catalog/ResourceMgrTest.java | 1 +
.../org/apache/doris/catalog/S3ResourceTest.java | 3 +
.../org/apache/doris/common/ExceptionChecker.java | 2 +-
.../org/apache/doris/common/util/UnitTestUtil.java | 2 +-
.../apache/doris/datasource/CatalogMgrTest.java | 14 +-
.../org/apache/doris/http/DorisHttpTestCase.java | 5 +-
.../doris/mysql/MysqlHandshakePacketTest.java | 2 +-
.../translator/PhysicalPlanTranslatorTest.java | 2 +-
.../properties/DistributionSpecHashTest.java | 6 +-
.../rules/analysis/AnalyzeWhereSubqueryTest.java | 61 +-
.../rules/analysis/BindSlotReferenceTest.java | 57 +-
.../nereids/rules/analysis/CheckAnalysisTest.java | 13 +
.../analysis/CheckExpressionLegalityTest.java | 74 +
.../rules/analysis/FillUpMissingSlotsTest.java | 4 +-
.../nereids/rules/analysis/GenerateFunction.java | 1645 ++++++++++++
.../rules/analysis/GenerateScalarFunction.java | 1393 ----------
.../join/OuterJoinLAsscomProjectTest.java | 39 +-
.../rewrite/ExpressionRewriteTestHelper.java | 1 -
.../rewrite/SimplifyArithmeticRuleTest.java | 2 +
.../rules/expression/rewrite/TypeCoercionTest.java | 20 +-
.../doris/nereids/rules/mv/SelectMvIndexTest.java | 6 +-
.../rewrite/logical/AggregateStrategiesTest.java | 2 +-
.../logical/EliminateUnnecessaryProjectTest.java | 34 +-
.../logical/ExtractFilterFromCrossJoinTest.java | 50 +
.../rules/rewrite/logical/LimitPushDownTest.java | 16 +-
.../logical/PhysicalStorageLayerAggregateTest.java | 12 +-
.../org/apache/doris/nereids/sqltest/JoinTest.java | 25 +
.../doris/nereids/sqltest/MultiJoinTest.java | 28 +-
.../apache/doris/nereids/sqltest/SqlTestBase.java | 17 +-
.../nereids/trees/expressions/GetDataTypeTest.java | 39 +-
.../nereids/trees/plans/PlanToStringTest.java | 2 +-
.../doris/nereids/util/HyperGraphBuilder.java | 2 +-
.../org/apache/doris/nereids/util/PlanChecker.java | 34 +
.../doris/nereids/util/TypeCoercionUtilsTest.java | 3 +-
.../persist/BatchModifyPartitionsInfoTest.java | 5 +-
.../planner/MaterializedViewFunctionTest.java | 2 +-
.../planner/MaterializedViewSelectorTest.java | 67 +-
.../org/apache/doris/planner/QueryPlanTest.java | 4 +-
.../doris/planner/TableFunctionPlanTest.java | 34 +-
.../doris/rewrite/RewriteDateLiteralRuleTest.java | 24 +-
.../apache/doris/statistics/AnalysisJobTest.java | 2 +-
.../org/apache/doris/statistics/CacheTest.java | 78 +-
.../org/apache/doris/statistics/HistogramTest.java | 16 +-
.../apache/doris/statistics/MVStatisticsTest.java | 9 +-
.../doris/statistics/StatsDeriveResultTest.java | 2 +-
.../java/org/apache/doris/utframe/DorisAssert.java | 2 +-
.../java/org/apache/doris/udf/JdbcExecutor.java | 33 +-
.../java/org/apache/doris/udf/UdafExecutor.java | 69 +-
.../java/org/apache/doris/udf/UdfExecutor.java | 73 +-
.../main/java/org/apache/doris/udf/UdfUtils.java | 43 +-
fe/pom.xml | 4 +-
.../doris/plugin/audit/AuditLoaderPlugin.java | 2 +
.../doris/plugin/audit/DorisStreamLoader.java | 14 +-
fe_plugins/pom.xml | 8 +-
.../doris/broker/hdfs/FileSystemIdentity.java | 17 +
.../doris/broker/hdfs/FileSystemManager.java | 119 +-
gensrc/script/doris_builtins_functions.py | 100 +-
gensrc/thrift/Exprs.thrift | 3 +
gensrc/thrift/FrontendService.thrift | 9 +
gensrc/thrift/Opcodes.thrift | 8 +
gensrc/thrift/PaloInternalService.thrift | 9 +
gensrc/thrift/PlanNodes.thrift | 26 +-
gensrc/thrift/Types.thrift | 1 +
.../correctness/test_subquery_in_disjunction.out | 11 +
.../table_valued_function/test_numbers.out | 10 +
.../correctness_p0/test_group_having_alias.out | 16 +
.../test_null_aware_left_anti_join.out | 1 -
.../storage/test_dup_tab_datetime_nullable.out | 1 -
.../duplicate/storage/test_short_key_index.out | 344 +++
.../data/datatype_p0/bitmap/test_bitmap_const.out | 4 +
.../data/datatype_p0/decimalv3/test_load.out | 6 +
.../data/datatype_p0/decimalv3/test_overflow.out | 19 +
.../data/datatype_p0/decimalv3/test_predicate.out | 9 +
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q07.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q09.out | 2 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q13.out | 2 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q18.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q26.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q27.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q28.out | 2 +-
.../datev2/tpcds_sf1_p1/sql/pipeline_q39_1.out | 88 +-
.../datev2/tpcds_sf1_p1/sql/pipeline_q39_2.out | 8 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q47.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q53.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q57.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q63.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q75.out | 182 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q85.out | 12 +-
.../data/datev2/tpcds_sf1_p1/sql/pipeline_q89.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q07.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q09.out | 2 +-
.../data/datev2/tpcds_sf1_p1/sql/q13.out | 2 +-
.../data/datev2/tpcds_sf1_p1/sql/q18.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q26.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q27.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q28.out | 2 +-
.../data/datev2/tpcds_sf1_p1/sql/q39_1.out | 88 +-
.../data/datev2/tpcds_sf1_p1/sql/q39_2.out | 8 +-
.../data/datev2/tpcds_sf1_p1/sql/q47.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q53.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q57.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q63.out | 200 +-
.../data/datev2/tpcds_sf1_p1/sql/q75.out | 180 +-
.../data/datev2/tpcds_sf1_p1/sql/q85.out | 12 +-
.../data/datev2/tpcds_sf1_p1/sql/q89.out | 200 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q01.out | 8 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q02.out | 24 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q03.out | 20 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q05.out | 10 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q06.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q07.out | 8 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q08.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q09.out | 350 +--
.../data/datev2/tpch_sf1_p2/sql/pipeline_q10.out | 40 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q11.out | 2096 +++++++--------
.../data/datev2/tpch_sf1_p2/sql/pipeline_q15.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q18.out | 114 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q19.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/pipeline_q22.out | 14 +-
.../data/datev2/tpch_sf1_p2/sql/q01.out | 8 +-
.../data/datev2/tpch_sf1_p2/sql/q02.out | 24 +-
.../data/datev2/tpch_sf1_p2/sql/q03.out | 20 +-
.../data/datev2/tpch_sf1_p2/sql/q05.out | 10 +-
.../data/datev2/tpch_sf1_p2/sql/q06.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/q07.out | 8 +-
.../data/datev2/tpch_sf1_p2/sql/q08.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/q09.out | 350 +--
.../data/datev2/tpch_sf1_p2/sql/q10.out | 40 +-
.../data/datev2/tpch_sf1_p2/sql/q11.out | 2096 +++++++--------
.../data/datev2/tpch_sf1_p2/sql/q15.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/q18.out | 114 +-
.../data/datev2/tpch_sf1_p2/sql/q19.out | 2 +-
.../data/datev2/tpch_sf1_p2/sql/q22.out | 14 +-
.../data/decimalv3/tpch_sf0.1_p1/sql/q01.out | 8 +-
.../data/decimalv3/tpch_sf1_p2/sql/q01.out | 8 +-
regression-test/data/demo_p0/streamLoad_action.out | 8 +
.../hive/test_catalog_with_resource.out | 31 +
.../external_catalog_p0/hive/test_hive_other.out | 12 +
.../iceberg/test_external_catalog_icebergv2.out | 44 +-
.../data/flink_connector_p0/flink_connector.out | 9 +
.../data/javaudf_p0/test_javaudaf_mysum_int.out | 12 +
.../jdbc_catalog_p0/test_mysql_jdbc_catalog.out | 16 +-
.../data/jdbc_catalog_p0/test_pg_jdbc_catalog.out | 2143 +++++++++++++++
.../data/jdbc_p0/test_jdbc_query_mysql.out | 2092 +++++++--------
.../data/jdbc_p0/test_jdbc_query_pg.out | 2080 +++++++--------
.../test_dup_mv_abs/test_dup_mv_abs.out | 43 +
.../test_dup_mv_bin/test_dup_mv_bin.out | 43 +
.../test_dup_mv_bitmap_hash.out | 18 +
.../test_dup_mv_plus/test_dup_mv_plus.out | 49 +
.../test_dup_mv_year/test_dup_mv_year.out | 18 +
regression-test/data/mtmv_p0/test_create_mtmv.out | 6 +
.../data/nereids_syntax_p0/agg_with_empty_set.out | 20 +
.../agg_with_sort.out} | 0
.../data/nereids_syntax_p0/bind_priority.out | 28 +
regression-test/data/nereids_syntax_p0/cast.out | 268 ++
.../data/nereids_syntax_p0/function.out | 36 +
.../data/nereids_syntax_p0/group_by_constant.out | 17 +
.../join_order.out} | 0
.../null_aware_left_anti_join.out | 1 -
regression-test/data/nereids_syntax_p0/rlike.out | 15 +
regression-test/data/nereids_syntax_p0/rollup.out | 9 +
.../data/nereids_syntax_p0/scan_partition.out | 4 +
.../data/nereids_syntax_p0/select_partition.out | 9 +
.../data/nereids_syntax_p0/set_operation.out | 121 +
.../nereids_syntax_p0/sub_query_correlated.out | 73 +-
.../sub_query_count_with_const.out} | 0
.../sub_query_diff_old_optimize.out | 31 +
.../test_bitmap_function_nereids.out} | 0
.../data/nereids_syntax_p0/test_convert.out | 4 +
.../nereids_syntax_p0/test_nereids_function.out | 7 +
.../test_time_lut_nereids.out} | 0
.../nereids_syntax_p0/test_variance_nereids.out | 4 +
.../data/nereids_syntax_p0/type_cast.out | 7 +
.../data/nereids_syntax_p0/utf8_id_test.out | 16 +
.../data/query_p0/aggregate/aggregate.out | 6 +
.../data/query_p0/join/test_bitmap_filter.out | 7 +
regression-test/data/query_p0/join/test_join.out | 91 +
.../query_p0/join/test_runtimefilter_on_datev2.out | 10 +
.../data/query_p0/keyword/order_group.out | 57 +
.../data/query_p0/keyword/test_keyword.out | 12 +
.../data/query_p0/limit/sql/withOrderBy.out | 13 +
.../query_p0/show/test_show_create_catalog.out | 2 +-
.../data/query_p0/show/test_show_where.out | 84 +
.../test_aggregate_histogram.out | 14 +
.../test_conditional_function.out | 201 ++
.../sql_functions/math_functions/test_round.out | 12 +
.../math_functions/test_running_difference.out | 12 +-
.../string_functions/test_split_part.out | 5 +
.../data/query_p0/sql_functions/test_in_expr.out | 10 +
.../test_width_bucket_function.out | 95 +
regression-test/data/query_p0/union/test_union.out | 41 +-
.../data/rollup/test_materialized_view_hll.out | Bin 451 -> 450 bytes
.../test_materialized_view_hll_with_light_sc.out | Bin 466 -> 465 bytes
.../data/rollup_p0/test_materialized_view.out | Bin 1399 -> 1398 bytes
.../data/rollup_p0/test_rollup_add_column.out | 7 +
.../cold_heat_separation/policy/alter.groovy | 11 +-
.../cold_heat_separation/policy/create.groovy | 63 +-
.../suites/cold_heat_separation/policy/drop.groovy | 10 +-
.../suites/cold_heat_separation/policy/show.groovy | 5 +-
.../use_policy/alter_table_add_policy.groovy | 10 +-
.../create_table_use_partition_policy.groovy | 10 +-
.../use_policy/create_table_use_policy.groovy | 5 +-
.../use_policy/modify_partition_add_policy.groovy | 11 +-
.../use_policy/use_default_storage_policy.groovy | 5 +-
.../test_vertical_compaction_uniq_keys.groovy | 2 +-
.../test_subquery_in_disjunction.groovy | 16 +
.../table_valued_function/test_numbers.groovy | 4 +
.../correctness_p0/test_group_having_alias.groovy | 27 +-
.../duplicate/storage/test_short_key_index.groovy | 545 ++++
.../datatype_p0/bitmap/test_bitmap_const.groovy | 22 +
.../datatype_p0/decimalv3/test_data/test.csv | 3 +
.../suites/datatype_p0/decimalv3/test_load.groovy | 58 +
.../datatype_p0/decimalv3/test_overflow.groovy | 56 +
.../datatype_p0/decimalv3/test_predicate.groovy | 46 +
.../suites/demo_p0/streamLoad_action.groovy | 26 +
.../hive/test_catalog_with_resource.groovy | 98 +
.../hive/test_hive_other.groovy | 20 +-
.../es/test_external_catalog_es.groovy | 59 +
.../es/test_external_es.groovy | 5 +-
.../iceberg/test_external_catalog_icebergv2.groovy | 22 +-
.../mysql/test_external_catalog_mysql.groovy | 102 +
.../mysql/test_external_mysql.groovy | 70 -
.../mysql/test_external_resource_mysql.groovy | 108 +-
.../pg/test_external_pg.groovy | 87 +-
.../suites/flink_connector_p0/ddl/create.sql | 12 +
.../flink_connector_p0/flink_connector.groovy | 41 +
.../test_decimal_bitmap_index_multi_page.groovy | 8 +
.../suites/index_p0/test_index_meta.groovy | 7 +-
.../javaudf_p0/test_javaudaf_mysum_int.groovy | 4 +
.../jdbc_catalog_p0/test_mysql_jdbc_catalog.groovy | 12 +-
.../jdbc_catalog_p0/test_pg_jdbc_catalog.groovy | 94 +
.../suites/jdbc_p0/test_jdbc_query_mysql.groovy | 4 +-
.../test_dup_mv_abs/test_dup_mv_abs.groovy | 86 +
.../test_dup_mv_bin/test_dup_mv_bin.groovy | 86 +
.../test_dup_mv_bitmap_hash.groovy | 79 +
.../test_dup_mv_plus/test_dup_mv_plus.groovy | 92 +
.../test_dup_mv_year/test_dup_mv_year.groovy | 79 +
.../suites/mtmv_p0/test_create_mtmv.groovy | 86 +
.../nereids_syntax_p0/agg_with_empty_set.groovy | 32 +
.../suites/nereids_syntax_p0/agg_with_sort.groovy | 38 +
.../suites/nereids_syntax_p0/any_value.groovy | 26 +
.../suites/nereids_syntax_p0/array_function.groovy | 44 +
.../suites/nereids_syntax_p0/bind_priority.groovy | 84 +
.../suites/nereids_syntax_p0/cast.groovy | 196 ++
.../suites/nereids_syntax_p0/expression.groovy | 37 +
.../suites/nereids_syntax_p0/function.groovy | 81 +-
.../nereids_syntax_p0/group_by_constant.groovy | 19 +
.../suites/nereids_syntax_p0/group_concat.groovy | 47 +
.../nereids_syntax_p0/information_schema.groovy | 9 +
.../suites/nereids_syntax_p0/join.groovy | 155 ++
.../suites/nereids_syntax_p0/join_order.groovy | 96 +
.../suites/nereids_syntax_p0/rlike.groovy | 27 +
.../suites/nereids_syntax_p0/rollup.groovy | 6 +-
.../suites/nereids_syntax_p0/scan_partition.groovy | 51 +
.../nereids_syntax_p0/select_partition.groovy | 12 +-
.../suites/nereids_syntax_p0/set_operation.groovy | 67 +-
.../suites/nereids_syntax_p0/set_var.groovy | 43 -
.../nereids_syntax_p0/sub_query_alias.groovy | 13 +
.../nereids_syntax_p0/sub_query_correlated.groovy | 146 +-
.../sub_query_count_with_const.groovy | 53 +
.../sub_query_diff_old_optimize.groovy | 160 ++
.../suites/nereids_syntax_p0/system_var.groovy | 49 +
.../test_bitmap_function_nereids.groovy | 194 ++
.../suites/nereids_syntax_p0/test_convert.groovy | 50 +
.../nereids_syntax_p0/test_nereids_function.groovy | 58 +
.../nereids_syntax_p0/test_time_lut_nereids.groovy | 29 +
.../nereids_syntax_p0/test_variance_nereids.groovy | 23 +
.../suites/nereids_syntax_p0/type_cast.groovy | 41 +
.../suites/nereids_syntax_p0/utf8_id_test.groovy | 34 +
.../performance_p0/redundant_conjuncts.groovy | 5 +-
.../suites/query_p0/aggregate/aggregate.groovy | 6 +
.../query_p0/aggregate/aggregate_count1.groovy | 102 +-
.../aggregate_group_by_metric_type.groovy | 40 +
.../suites/query_p0/join/test_bitmap_filter.groovy | 9 +
.../suites/query_p0/join/test_join.groovy | 26 +-
.../join/test_runtimefilter_on_datev2.groovy | 7 +
.../suites/query_p0/keyword/order_group.groovy | 10 +
.../suites/query_p0/keyword/test_keyword.groovy | 4 +
.../suites/query_p0/limit/sql/withOrderBy.sql | 4 +
.../test_aggregate_histogram.groovy | 214 ++
.../test_array_functions_by_literal.groovy | 5 +-
.../test_conditional_function.groovy | 57 +
.../sql_functions/math_functions/test_round.groovy | 10 +
.../string_functions/test_split_part.groovy | 2 +
.../query_p0/sql_functions/test_in_expr.groovy | 8 +
.../test_width_bucket_function.groovy | 88 +
.../system/test_query_sys_data_type.groovy | 2 +-
.../suites/query_p0/test_dict_with_null.groovy | 46 +
.../suites/rollup_p0/test_rollup_add_column.groovy | 81 +
.../test_agg_keys_schema_change.groovy | 1 +
.../suites/statistics/alter_col_stats.groovy | 3 +-
.../suites/yandex_metrica_p2/load.groovy | 2 +-
run-be-ut.sh | 5 +-
thirdparty/CHANGELOG.md | 6 +
thirdparty/build-thirdparty.sh | 47 +-
thirdparty/vars.sh | 31 +-
ui/src/api/api.ts | 64 +-
ui/src/components/table/index.less | 11 +
ui/src/components/table/index.tsx | 50 +-
ui/src/i18n.tsx | 15 +-
ui/src/pages/configuration/index.tsx | 37 +-
ui/src/pages/logs/index.tsx | 56 +-
.../playground/content/components/data-prev.tsx | 85 +-
ui/src/pages/playground/page-side/index.tsx | 20 +-
ui/src/pages/playground/tree/index.tsx | 164 +-
ui/src/pages/query-profile/index.tsx | 80 +-
ui/src/pages/session/index.tsx | 45 +-
ui/src/pages/system/index.tsx | 62 +-
ui/src/utils/lazy.tsx | 17 +-
ui/src/utils/utils.ts | 37 +-
ui/tsconfig.json | 94 +-
1540 files changed, 55554 insertions(+), 29211 deletions(-)
create mode 100644 be/src/exprs/match_predicate.cpp
create mode 100644 be/src/exprs/match_predicate.h
delete mode 100644 be/src/exprs/table_function/dummy_table_functions.cpp
delete mode 100644 be/src/exprs/table_function/dummy_table_functions.h
delete mode 100644 be/src/exprs/table_function/explode_bitmap.cpp
delete mode 100644 be/src/exprs/table_function/explode_bitmap.h
delete mode 100644 be/src/exprs/table_function/explode_json_array.cpp
delete mode 100644 be/src/exprs/table_function/explode_json_array.h
delete mode 100644 be/src/exprs/table_function/explode_split.cpp
delete mode 100644 be/src/exprs/table_function/explode_split.h
delete mode 100644 be/src/exprs/table_function/table_function.h
delete mode 100644 be/src/exprs/table_function/table_function_factory.cpp
delete mode 100644 be/src/exprs/table_function/table_function_factory.h
create mode 100644 be/src/io/cache/block/block_file_cache.cpp
create mode 100644 be/src/io/cache/block/block_file_cache.h
create mode 100644 be/src/io/cache/block/block_file_cache_factory.cpp
create mode 100644 be/src/io/cache/block/block_file_cache_factory.h
create mode 100644 be/src/io/cache/block/block_file_cache_fwd.h
create mode 100644 be/src/io/cache/block/block_file_cache_profile.cpp
create mode 100644 be/src/io/cache/block/block_file_cache_profile.h
create mode 100644 be/src/io/cache/block/block_file_cache_settings.h
create mode 100644 be/src/io/cache/block/block_file_segment.cpp
create mode 100644 be/src/io/cache/block/block_file_segment.h
create mode 100644 be/src/io/cache/block/block_lru_file_cache.cpp
create mode 100644 be/src/io/cache/block/block_lru_file_cache.h
create mode 100644 be/src/io/cache/block/cached_remote_file_reader.cpp
create mode 100644 be/src/io/cache/block/cached_remote_file_reader.h
delete mode 100644 be/src/olap/row_block2.cpp
delete mode 100644 be/src/olap/row_block2.h
create mode 100644 be/src/olap/rowset/segment_v2/inverted_index_reader.cpp
create mode 100644 be/src/olap/rowset/segment_v2/inverted_index_reader.h
create mode 100644 be/src/pipeline/exec/empty_source_operator.cpp
create mode 100644 be/src/pipeline/exec/empty_source_operator.h
create mode 100644 be/src/runtime/block_spill_manager.cpp
create mode 100644 be/src/runtime/block_spill_manager.h
delete mode 100644 be/src/runtime/bufferpool/buffer_allocator.cc
delete mode 100644 be/src/runtime/bufferpool/buffer_allocator.h
delete mode 100644 be/src/runtime/bufferpool/buffer_pool.cc
delete mode 100644 be/src/runtime/bufferpool/buffer_pool.h
delete mode 100644 be/src/runtime/bufferpool/buffer_pool_counters.h
delete mode 100644 be/src/runtime/bufferpool/buffer_pool_internal.h
delete mode 100644 be/src/runtime/bufferpool/free_list.h
delete mode 100644 be/src/runtime/bufferpool/suballocator.cc
delete mode 100644 be/src/runtime/bufferpool/suballocator.h
delete mode 100644 be/src/runtime/bufferpool/system_allocator.cc
delete mode 100644 be/src/runtime/bufferpool/system_allocator.h
delete mode 100644 be/src/runtime/disk_io_mgr.cc
delete mode 100644 be/src/runtime/disk_io_mgr.h
delete mode 100644 be/src/runtime/disk_io_mgr_internal.h
delete mode 100644 be/src/runtime/disk_io_mgr_reader_context.cc
delete mode 100644 be/src/runtime/disk_io_mgr_scan_range.cc
delete mode 100644 be/src/runtime/row_batch.cpp
delete mode 100644 be/src/runtime/row_batch.h
create mode 100644 be/src/runtime/runtime_predicate.cpp
create mode 100644 be/src/runtime/runtime_predicate.h
create mode 100644 be/src/util/arrow/block_convertor.cpp
create mode 100644 be/src/util/arrow/block_convertor.h
create mode 100644 be/src/vec/common/hex.cpp
create mode 100644 be/src/vec/common/hex.h
create mode 100644 be/src/vec/core/block_spill_reader.cpp
create mode 100644 be/src/vec/core/block_spill_reader.h
create mode 100644 be/src/vec/core/block_spill_writer.cpp
create mode 100644 be/src/vec/core/block_spill_writer.h
create mode 100644 be/src/vec/data_types/data_type_time.cpp
create mode 100644 be/src/vec/data_types/data_type_time.h
create mode 100644 be/src/vec/exec/scan/vmeta_scan_node.cpp
create mode 100644 be/src/vec/exec/scan/vmeta_scan_node.h
create mode 100644 be/src/vec/exec/scan/vmeta_scanner.cpp
create mode 100644 be/src/vec/exec/scan/vmeta_scanner.h
create mode 100644 be/src/vec/exprs/table_function/table_function.h
create mode 100644 be/src/vec/exprs/table_function/table_function_factory.cpp
create mode 100644 be/src/vec/exprs/table_function/table_function_factory.h
create mode 100644 be/src/vec/functions/function_width_bucket.cpp
create mode 100644 be/src/vec/functions/match.cpp
create mode 100644 be/src/vec/sink/vmemory_scratch_sink.cpp
create mode 100644 be/src/vec/sink/vmemory_scratch_sink.h
create mode 100644 be/src/vec/utils/histogram_helpers.hpp
create mode 100644 be/test/io/cache/file_block_cache_test.cpp
delete mode 100644 be/test/runtime/disk_io_mgr_test.cpp
delete mode 100644 be/test/runtime/load_channel_mgr_test.cpp
create mode 100644 be/test/vec/core/block_spill_test.cpp
create mode 100755 build-for-release.sh
create mode 100644 docs/en/docs/releasenotes/release-1.2.1.md
create mode 100644 docs/en/docs/sql-manual/sql-functions/width-bucket.md
create mode 100644 docs/en/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/CANCEL-EXPORT.md
create mode 100644 docs/zh-CN/docs/releasenotes/release-1.2.1.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/width-bucket.md
create mode 100644 docs/zh-CN/docs/sql-manual/sql-reference/Data-Manipulation-Statements/Manipulation/CANCEL-EXPORT.md
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/analysis/CancelExportStmt.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/analysis/MatchPredicate.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/analysis/TableSnapshot.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/PooledHiveMetaStoreClient.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreClient.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveVersionUtil.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/PooledHiveMetaStoreClient.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/DropTableEvent.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/EventFactory.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/IgnoredEvent.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEvent.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventFactory.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventType.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventsProcessor.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreNotificationException.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreNotificationFetchException.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreTableEvent.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/AdjustAggregateNullableForEmptySetJob.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/CheckLegalityAfterRewrite.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/CheckLegalityBeforeTypeCoercion.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AdjustAggregateNullableForEmptySet.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/AvgDistinctToSumDivCount.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/JoinReorderCommon.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/rules/SupportJavaDateFormatter.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateFilterUnderApplyProject.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/ExtractFilterFromCrossJoin.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/OrderExpression.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/VariableDesc.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/BitmapIntersectFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecision.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecisionForRound.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputePrecisionForSum.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ComputeSignatureHelper.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AnyValue.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/ApproxCountDistinct.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/AvgWeighted.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/BitmapFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/CollectList.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/CollectSet.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/GroupConcat.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Histogram.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/HllFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/IntersectCount.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/MaxBy.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/MinBy.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/NullableAggregateFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/OrthogonalBitmapFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/OrthogonalBitmapIntersect.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/OrthogonalBitmapIntersectCount.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/OrthogonalBitmapUnionCount.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Percentile.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/PercentileApprox.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/PercentileArray.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/QuantileUnion.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Retention.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/SequenceCount.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/SequenceFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/SequenceMatch.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/StdDevOrVarianceFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Stddev.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/StddevSamp.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/TopN.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/TopNArray.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/TopNWeighted.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Variance.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/VarianceSamp.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/WindowFunnel.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/AesCryptoFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Array.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayAvg.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayCompact.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayContains.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayDifference.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayDistinct.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayEnumerate.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayExcept.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayIntersect.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayJoin.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayMax.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayMin.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayPopback.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayPosition.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayProduct.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRange.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayRemove.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySize.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySlice.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySort.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySum.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayUnion.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayWithConstant.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraysOverlap.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/BitmapFromArray.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/BitmapToArray.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Cardinality.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CharLength.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CompatibleTypeArrayFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ConnectionId.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ConvertTo.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CountEqual.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CryptoFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CurrentUser.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Database.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Domain.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/DomainWithoutWww.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ElementAt.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ElementExtract.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ElementSlice.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Field.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/LastDay.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/MultiMatchAny.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/MultiSearchAllPositions.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Protocol.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/RegexpExtractAll.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/RegexpReplaceOne.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/RunningDifference.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Size.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sm4CryptoFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SplitByChar.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SplitByString.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SubstringIndex.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ToMonday.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/User.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Uuid.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/ArrayLiteral.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/FollowToArgumentType.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/Int16OrLessType.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/Int32OrLessType.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/Int64OrLessType.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/ListPartitionPruner.java
delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/RangePartitionPruner.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/planner/external/MetadataScanNode.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/rewrite/MatchPredicateRule.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/rewrite/mvrewrite/ExprToSlotRefRule.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/statistics/HistogramBuilder.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/statistics/HistogramTask.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/statistics/Statistic.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/tablefunction/IcebergTableValuedFunction.java
create mode 100644 fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataTableValuedFunction.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/analysis/CancelExportStmtTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/CheckExpressionLegalityTest.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/GenerateFunction.java
delete mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/GenerateScalarFunction.java
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/ExtractFilterFromCrossJoinTest.java
create mode 100644 regression-test/data/data_model_p0/duplicate/storage/test_short_key_index.out
create mode 100644 regression-test/data/datatype_p0/bitmap/test_bitmap_const.out
create mode 100644 regression-test/data/datatype_p0/decimalv3/test_load.out
create mode 100644 regression-test/data/datatype_p0/decimalv3/test_overflow.out
create mode 100644 regression-test/data/datatype_p0/decimalv3/test_predicate.out
create mode 100644 regression-test/data/demo_p0/streamLoad_action.out
create mode 100644 regression-test/data/external_catalog_p0/hive/test_catalog_with_resource.out
create mode 100644 regression-test/data/flink_connector_p0/flink_connector.out
create mode 100644 regression-test/data/jdbc_catalog_p0/test_pg_jdbc_catalog.out
create mode 100644 regression-test/data/materialized_view_p0/test_dup_mv_abs/test_dup_mv_abs.out
create mode 100644 regression-test/data/materialized_view_p0/test_dup_mv_bin/test_dup_mv_bin.out
create mode 100644 regression-test/data/materialized_view_p0/test_dup_mv_bitmap_hash/test_dup_mv_bitmap_hash.out
create mode 100644 regression-test/data/materialized_view_p0/test_dup_mv_plus/test_dup_mv_plus.out
create mode 100644 regression-test/data/materialized_view_p0/test_dup_mv_year/test_dup_mv_year.out
create mode 100644 regression-test/data/mtmv_p0/test_create_mtmv.out
create mode 100644 regression-test/data/nereids_syntax_p0/agg_with_empty_set.out
copy regression-test/data/{tpch_sf1_p1/function/test_find_in_set.out => nereids_syntax_p0/agg_with_sort.out} (100%)
create mode 100644 regression-test/data/nereids_syntax_p0/bind_priority.out
create mode 100644 regression-test/data/nereids_syntax_p0/cast.out
copy regression-test/data/{correctness/test_crossjoin_inlineview_slot.out => nereids_syntax_p0/join_order.out} (100%)
create mode 100644 regression-test/data/nereids_syntax_p0/rlike.out
create mode 100644 regression-test/data/nereids_syntax_p0/rollup.out
create mode 100644 regression-test/data/nereids_syntax_p0/scan_partition.out
copy regression-test/data/{correctness_p0/test_join_with_projection.out => nereids_syntax_p0/sub_query_count_with_const.out} (100%)
create mode 100644 regression-test/data/nereids_syntax_p0/sub_query_diff_old_optimize.out
copy regression-test/data/{query_p0/sql_functions/bitmap_functions/test_bitmap_function.out => nereids_syntax_p0/test_bitmap_function_nereids.out} (100%)
create mode 100644 regression-test/data/nereids_syntax_p0/test_convert.out
create mode 100644 regression-test/data/nereids_syntax_p0/test_nereids_function.out
copy regression-test/data/{query_p0/sql_functions/datetime_functions/test_time_lut.out => nereids_syntax_p0/test_time_lut_nereids.out} (100%)
create mode 100644 regression-test/data/nereids_syntax_p0/test_variance_nereids.out
create mode 100644 regression-test/data/nereids_syntax_p0/type_cast.out
create mode 100644 regression-test/data/nereids_syntax_p0/utf8_id_test.out
create mode 100644 regression-test/data/query_p0/limit/sql/withOrderBy.out
create mode 100644 regression-test/data/query_p0/sql_functions/aggregate_functions/test_aggregate_histogram.out
create mode 100644 regression-test/data/query_p0/sql_functions/string_functions/test_split_part.out
create mode 100644 regression-test/data/query_p0/sql_functions/width_bucket_fuctions/test_width_bucket_function.out
create mode 100644 regression-test/data/rollup_p0/test_rollup_add_column.out
create mode 100644 regression-test/suites/data_model_p0/duplicate/storage/test_short_key_index.groovy
create mode 100644 regression-test/suites/datatype_p0/bitmap/test_bitmap_const.groovy
create mode 100644 regression-test/suites/datatype_p0/decimalv3/test_data/test.csv
create mode 100644 regression-test/suites/datatype_p0/decimalv3/test_load.groovy
create mode 100644 regression-test/suites/datatype_p0/decimalv3/test_overflow.groovy
create mode 100644 regression-test/suites/datatype_p0/decimalv3/test_predicate.groovy
create mode 100644 regression-test/suites/external_catalog_p0/hive/test_catalog_with_resource.groovy
create mode 100644 regression-test/suites/external_table_emr_p2/es/test_external_catalog_es.groovy
create mode 100644 regression-test/suites/external_table_emr_p2/mysql/test_external_catalog_mysql.groovy
delete mode 100644 regression-test/suites/external_table_emr_p2/mysql/test_external_mysql.groovy
create mode 100644 regression-test/suites/flink_connector_p0/ddl/create.sql
create mode 100644 regression-test/suites/flink_connector_p0/flink_connector.groovy
create mode 100644 regression-test/suites/jdbc_catalog_p0/test_pg_jdbc_catalog.groovy
create mode 100644 regression-test/suites/materialized_view_p0/test_dup_mv_abs/test_dup_mv_abs.groovy
create mode 100644 regression-test/suites/materialized_view_p0/test_dup_mv_bin/test_dup_mv_bin.groovy
create mode 100644 regression-test/suites/materialized_view_p0/test_dup_mv_bitmap_hash/test_dup_mv_bitmap_hash.groovy
create mode 100644 regression-test/suites/materialized_view_p0/test_dup_mv_plus/test_dup_mv_plus.groovy
create mode 100644 regression-test/suites/materialized_view_p0/test_dup_mv_year/test_dup_mv_year.groovy
create mode 100644 regression-test/suites/mtmv_p0/test_create_mtmv.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/agg_with_empty_set.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/agg_with_sort.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/any_value.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/array_function.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/bind_priority.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/cast.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/expression.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/group_concat.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/join_order.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/rlike.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/scan_partition.groovy
delete mode 100644 regression-test/suites/nereids_syntax_p0/set_var.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/sub_query_count_with_const.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/sub_query_diff_old_optimize.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/system_var.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/test_bitmap_function_nereids.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/test_convert.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/test_nereids_function.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/test_time_lut_nereids.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/test_variance_nereids.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/type_cast.groovy
create mode 100644 regression-test/suites/nereids_syntax_p0/utf8_id_test.groovy
create mode 100644 regression-test/suites/query_p0/limit/sql/withOrderBy.sql
create mode 100644 regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_histogram.groovy
create mode 100644 regression-test/suites/query_p0/sql_functions/width_bucket_fuctions/test_width_bucket_function.groovy
create mode 100644 regression-test/suites/query_p0/test_dict_with_null.groovy
create mode 100644 regression-test/suites/rollup_p0/test_rollup_add_column.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/09: [feature](struct-type) add the class ColumnStruct and class DataTypeStruct implement (#14545)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3904f1a8d187e5fc4af5721c867ac80351d78704
Author: carlvinhust2012 <hu...@126.com>
AuthorDate: Thu Dec 8 09:26:12 2022 +0800
[feature](struct-type) add the class ColumnStruct and class DataTypeStruct implement (#14545)
Co-authored-by: hucheng01 <hu...@baidu.com>
---
be/src/vec/CMakeLists.txt | 2 +
be/src/vec/columns/column_struct.cpp | 618 +++++++++++++++++++++++++++++
be/src/vec/columns/column_struct.h | 232 +++++++++++
be/src/vec/core/types.h | 3 +
be/src/vec/data_types/data_type_struct.cpp | 361 +++++++++++++++++
be/src/vec/data_types/data_type_struct.h | 115 ++++++
6 files changed, 1331 insertions(+)
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index 794562d194..be5b60c6dd 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -49,6 +49,7 @@ set(VEC_FILES
aggregate_functions/aggregate_function_histogram.cpp
columns/column.cpp
columns/column_array.cpp
+ columns/column_struct.cpp
columns/column_const.cpp
columns/column_decimal.cpp
columns/column_nullable.cpp
@@ -75,6 +76,7 @@ set(VEC_FILES
core/materialize_block.cpp
data_types/data_type.cpp
data_types/data_type_array.cpp
+ data_types/data_type_struct.cpp
data_types/data_type_bitmap.cpp
data_types/data_type_factory.cpp
data_types/data_type_fixed_length_object.cpp
diff --git a/be/src/vec/columns/column_struct.cpp b/be/src/vec/columns/column_struct.cpp
new file mode 100644
index 0000000000..05d5df0fd4
--- /dev/null
+++ b/be/src/vec/columns/column_struct.cpp
@@ -0,0 +1,618 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnStruct.cpp
+// and modified by Doris
+
+#include "vec/columns/column_struct.h"
+
+namespace doris::vectorized {
+
+namespace ErrorCodes {
+extern const int ILLEGAL_COLUMN;
+extern const int NOT_IMPLEMENTED;
+extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
+extern const int LOGICAL_ERROR;
+} // namespace ErrorCodes
+
+std::string ColumnStruct::get_name() const {
+ std::stringstream res;
+ res << "Struct(";
+ bool is_first = true;
+ for (const auto& column : columns) {
+ if (!is_first) {
+ res << ", ";
+ }
+ is_first = false;
+ res << column->get_name();
+ }
+ res << ")";
+ return res.str();
+}
+
+ColumnStruct::ColumnStruct(MutableColumns&& mutable_columns) {
+ columns.reserve(mutable_columns.size());
+ for (auto& column : mutable_columns) {
+ if (is_column_const(*column)) {
+ throw Exception {"ColumnStruct cannot have ColumnConst as its element",
+ ErrorCodes::ILLEGAL_COLUMN};
+ }
+ columns.push_back(std::move(column));
+ }
+}
+
+ColumnStruct::ColumnStruct(Columns&& columns) {
+ columns.reserve(columns.size());
+ for (auto& column : columns) {
+ if (is_column_const(*column)) {
+ throw Exception {"ColumnStruct cannot have ColumnConst as its element",
+ ErrorCodes::ILLEGAL_COLUMN};
+ }
+ columns.push_back(std::move(column));
+ }
+}
+
+ColumnStruct::ColumnStruct(TupleColumns&& tuple_columns) {
+ columns.reserve(tuple_columns.size());
+ for (auto& column : tuple_columns) {
+ if (is_column_const(*column)) {
+ throw Exception {"ColumnStruct cannot have ColumnConst as its element",
+ ErrorCodes::ILLEGAL_COLUMN};
+ }
+ columns.push_back(std::move(column));
+ }
+}
+
+ColumnStruct::Ptr ColumnStruct::create(Columns& columns) {
+ for (const auto& column : columns) {
+ if (is_column_const(*column))
+ throw Exception {"ColumnStruct cannot have ColumnConst as its element",
+ ErrorCodes::ILLEGAL_COLUMN};
+ }
+ auto column_struct = ColumnStruct::create(columns);
+ return column_struct;
+}
+
+ColumnStruct::Ptr ColumnStruct::create(TupleColumns& tuple_columns) {
+ for (const auto& column : tuple_columns) {
+ if (is_column_const(*column)) {
+ throw Exception {"ColumnStruct cannot have ColumnConst as its element",
+ ErrorCodes::ILLEGAL_COLUMN};
+ }
+ }
+ auto column_struct = ColumnStruct::create(tuple_columns);
+ return column_struct;
+}
+
+MutableColumnPtr ColumnStruct::clone_empty() const {
+ const size_t tuple_size = columns.size();
+ MutableColumns new_columns(tuple_size);
+ for (size_t i = 0; i < tuple_size; ++i) {
+ new_columns[i] = columns[i]->clone_empty();
+ }
+ return ColumnStruct::create(std::move(new_columns));
+}
+
+MutableColumnPtr ColumnStruct::clone_resized(size_t new_size) const {
+ const size_t tuple_size = columns.size();
+ MutableColumns new_columns(tuple_size);
+ for (size_t i = 0; i < tuple_size; ++i) {
+ new_columns[i] = columns[i]->clone_resized(new_size);
+ }
+ return ColumnStruct::create(std::move(new_columns));
+}
+
+Field ColumnStruct::operator[](size_t n) const {
+ Field res;
+ get(n, res);
+ return res;
+}
+
+void ColumnStruct::get(size_t n, Field& res) const {
+ const size_t tuple_size = columns.size();
+
+ res = Tuple();
+ Tuple& res_tuple = res.get<Tuple&>();
+ res_tuple.reserve(tuple_size);
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ res_tuple.push_back((*columns[i])[n]);
+ }
+}
+
+bool ColumnStruct::is_default_at(size_t n) const {
+ const size_t tuple_size = columns.size();
+ for (size_t i = 0; i < tuple_size; ++i) {
+ if (!columns[i]->is_default_at(n)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+StringRef ColumnStruct::get_data_at(size_t) const {
+ throw Exception("Method get_data_at is not supported for " + get_name(),
+ ErrorCodes::NOT_IMPLEMENTED);
+}
+
+void ColumnStruct::insert_data(const char*, size_t) {
+ throw Exception("Method insert_data is not supported for " + get_name(),
+ ErrorCodes::NOT_IMPLEMENTED);
+}
+
+void ColumnStruct::insert(const Field& x) {
+ const auto& tuple = x.get<const Tuple&>();
+ const size_t tuple_size = columns.size();
+ if (tuple.size() != tuple_size) {
+ throw Exception("Cannot insert value of different size into tuple",
+ ErrorCodes::CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE);
+ }
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ columns[i]->insert(tuple[i]);
+ }
+}
+
+void ColumnStruct::insert_from(const IColumn& src_, size_t n) {
+ const ColumnStruct& src = assert_cast<const ColumnStruct&>(src_);
+
+ const size_t tuple_size = columns.size();
+ if (src.columns.size() != tuple_size) {
+ throw Exception("Cannot insert value of different size into tuple",
+ ErrorCodes::CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE);
+ }
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ columns[i]->insert_from(*src.columns[i], n);
+ }
+}
+
+void ColumnStruct::insert_default() {
+ for (auto& column : columns) {
+ column->insert_default();
+ }
+}
+
+void ColumnStruct::pop_back(size_t n) {
+ for (auto& column : columns) {
+ column->pop_back(n);
+ }
+}
+
+StringRef ColumnStruct::serialize_value_into_arena(size_t n, Arena& arena,
+ char const*& begin) const {
+ StringRef res(begin, 0);
+ for (const auto& column : columns) {
+ auto value_ref = column->serialize_value_into_arena(n, arena, begin);
+ res.data = value_ref.data - res.size;
+ res.size += value_ref.size;
+ }
+
+ return res;
+}
+
+const char* ColumnStruct::deserialize_and_insert_from_arena(const char* pos) {
+ for (auto& column : columns) {
+ pos = column->deserialize_and_insert_from_arena(pos);
+ }
+
+ return pos;
+}
+
+void ColumnStruct::update_hash_with_value(size_t n, SipHash& hash) const {
+ for (const auto& column : columns) {
+ column->update_hash_with_value(n, hash);
+ }
+}
+
+// void ColumnStruct::update_weak_hash32(WeakHash32 & hash) const {
+// auto s = size();
+// if (hash.get_data().size() != s) {
+// throw Exception("Size of WeakHash32 does not match size of column: column size is " + std::to_string(s) +
+// ", hash size is " + std::to_string(hash.getData().size()), ErrorCodes::LOGICAL_ERROR);
+// }
+
+// for (const auto & column : columns) {
+// column->update_weak_hash32(hash);
+// }
+// }
+
+// void ColumnStruct::update_hash_fast(SipHash & hash) const {
+// for (const auto & column : columns) {
+// column->update_hash_fast(hash);
+// }
+// }
+
+// const char * ColumnStruct::skip_serialized_in_arena(const char * pos) const {
+// for (const auto & column : columns) {
+// pos = column->skip_serialized_in_arena(pos);
+// }
+// return pos;
+// }
+
+// void ColumnStruct::expand(const Filter & mask, bool inverted)
+// {
+// for (auto & column : columns) {
+// column->expand(mask, inverted);
+// }
+// }
+
+// ColumnPtr ColumnStruct::index(const IColumn & indexes, size_t limit) const
+// {
+// const size_t tuple_size = columns.size();
+// Columns new_columns(tuple_size);
+
+// for (size_t i = 0; i < tuple_size; ++i) {
+// new_columns[i] = columns[i]->index(indexes, limit);
+// }
+
+// return ColumnStruct::create(new_columns);
+// }
+
+void ColumnStruct::insert_range_from(const IColumn& src, size_t start, size_t length) {
+ const size_t tuple_size = columns.size();
+ for (size_t i = 0; i < tuple_size; ++i) {
+ columns[i]->insert_range_from(*assert_cast<const ColumnStruct&>(src).columns[i], start,
+ length);
+ }
+}
+
+ColumnPtr ColumnStruct::filter(const Filter& filt, ssize_t result_size_hint) const {
+ const size_t tuple_size = columns.size();
+ Columns new_columns(tuple_size);
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ new_columns[i] = columns[i]->filter(filt, result_size_hint);
+ }
+ return ColumnStruct::create(new_columns);
+}
+
+ColumnPtr ColumnStruct::permute(const Permutation& perm, size_t limit) const {
+ const size_t tuple_size = columns.size();
+ Columns new_columns(tuple_size);
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ new_columns[i] = columns[i]->permute(perm, limit);
+ }
+
+ return ColumnStruct::create(new_columns);
+}
+
+ColumnPtr ColumnStruct::replicate(const Offsets& offsets) const {
+ const size_t tuple_size = columns.size();
+ Columns new_columns(tuple_size);
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ new_columns[i] = columns[i]->replicate(offsets);
+ }
+
+ return ColumnStruct::create(new_columns);
+}
+
+MutableColumns ColumnStruct::scatter(ColumnIndex num_columns, const Selector& selector) const {
+ const size_t tuple_size = columns.size();
+ std::vector<MutableColumns> scattered_tuple_elements(tuple_size);
+
+ for (size_t tuple_element_idx = 0; tuple_element_idx < tuple_size; ++tuple_element_idx) {
+ scattered_tuple_elements[tuple_element_idx] =
+ columns[tuple_element_idx]->scatter(num_columns, selector);
+ }
+
+ MutableColumns res(num_columns);
+
+ for (size_t scattered_idx = 0; scattered_idx < num_columns; ++scattered_idx) {
+ MutableColumns new_columns(tuple_size);
+ for (size_t tuple_element_idx = 0; tuple_element_idx < tuple_size; ++tuple_element_idx) {
+ new_columns[tuple_element_idx] =
+ std::move(scattered_tuple_elements[tuple_element_idx][scattered_idx]);
+ }
+ res[scattered_idx] = ColumnStruct::create(std::move(new_columns));
+ }
+
+ return res;
+}
+
+// int ColumnStruct::compare_at_impl(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint,
+// const Collator* collator) const {
+// const size_t tuple_size = columns.size();
+// for (size_t i = 0; i < tuple_size; ++i) {
+// int res = 0;
+// if (collator && columns[i]->is_collation_supported()) {
+// res = columns[i]->compare_at_with_collation(
+// n, m, *assert_cast<const ColumnStruct&>(rhs).columns[i], nan_direction_hint,
+// *collator);
+// } else {
+// res = columns[i]->compare_at(n, m, *assert_cast<const ColumnStruct&>(rhs).columns[i],
+// nan_direction_hint);
+// }
+
+// if (res) {
+// return res;
+// }
+// }
+// return 0;
+// }
+
+// int ColumnStruct::compare_at(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) const {
+// return compare_at_impl(n, m, rhs, nan_direction_hint);
+// }
+
+// void ColumnStruct::compare_column(const IColumn& rhs, size_t rhs_row_num,
+// PaddedPODArray<UInt64>* row_indexes,
+// PaddedPODArray<Int8>& compare_results, int direction,
+// int nan_direction_hint) const {
+// return do_compare_column<ColumnStruct>(assert_cast<const ColumnStruct&>(rhs), rhs_row_num,
+// row_indexes, compare_results, direction,
+// nan_direction_hint);
+// }
+
+// int ColumnStruct::compare_at_with_collation(size_t n, size_t m, const IColumn& rhs,
+// int nan_direction_hint,
+// const Collator& collator) const {
+// return compare_at_impl(n, m, rhs, nan_direction_hint, &collator);
+// }
+
+// bool ColumnStruct::has_equal_values() const {
+// return has_equal_values_impl<ColumnStruct>();
+// }
+
+// template <bool positive>
+// struct ColumnStruct::Less {
+// TupleColumns columns;
+// int nan_direction_hint;
+// const Collator* collator;
+
+// Less(const TupleColumns& columns_, int nan_direction_hint_, const Collator* collator_ = nullptr)
+// : columns(columns_), nan_direction_hint(nan_direction_hint_), collator(collator_) {}
+
+// bool operator()(size_t a, size_t b) const {
+// for (const auto& column : columns) {
+// int res;
+// if (collator && column->isCollationSupported()) {
+// res = column->compareAtWithCollation(a, b, *column, nan_direction_hint, *collator);
+// } else {
+// res = column->compareAt(a, b, *column, nan_direction_hint);
+// }
+// if (res < 0) {
+// return positive;
+// } else if (res > 0) {
+// return !positive;
+// }
+// }
+// return false;
+// }
+// };
+
+// void ColumnStruct::get_permutation_impl(IColumn::PermutationSortDirection direction,
+// IColumn::PermutationSortStability stability, size_t limit,
+// int nan_direction_hint, Permutation& res,
+// const Collator* collator) const {
+// size_t rows = size();
+// res.resize(rows);
+// for (size_t i = 0; i < rows; ++i) {
+// res[i] = i;
+// }
+
+// if (limit >= rows) {
+// limit = 0;
+// }
+
+// EqualRange ranges;
+// ranges.emplace_back(0, rows);
+// update_permutation_impl(direction, stability, limit, nan_direction_hint, res, ranges, collator);
+// }
+
+// void ColumnStruct::update_permutation_impl(IColumn::PermutationSortDirection direction,
+// IColumn::PermutationSortStability stability,
+// size_t limit, int nan_direction_hint,
+// IColumn::Permutation& res, EqualRanges& equal_ranges,
+// const Collator* collator) const {
+// if (equal_ranges.empty()) {
+// return;
+// }
+
+// for (const auto& column : columns) {
+// while (!equal_ranges.empty() && limit && limit <= equal_ranges.back().first) {
+// equal_ranges.pop_back();
+// }
+
+// if (collator && column->isCollationSupported()) {
+// column->update_permutation_with_collation(*collator, direction, stability, limit,
+// nan_direction_hint, res, equal_ranges);
+// } else {
+// column->update_permutation(direction, stability, limit, nan_direction_hint, res,
+// equal_ranges);
+// }
+// if (equal_ranges.empty()) {
+// break;
+// }
+// }
+// }
+
+// void ColumnStruct::get_permutation(IColumn::PermutationSortDirection direction,
+// IColumn::PermutationSortStability stability, size_t limit,
+// int nan_direction_hint, Permutation& res) const {
+// get_permutation_impl(direction, stability, limit, nan_direction_hint, res, nullptr);
+// }
+
+// void ColumnStruct::update_permutation(IColumn::PermutationSortDirection direction,
+// IColumn::PermutationSortStability stability, size_t limit,
+// int nan_direction_hint, IColumn::Permutation& res,
+// EqualRanges& equal_ranges) const {
+// update_permutation_impl(direction, stability, limit, nan_direction_hint, res, equal_ranges);
+// }
+
+// void ColumnStruct::get_permutation_with_collation(const Collator& collator,
+// IColumn::PermutationSortDirection direction,
+// IColumn::PermutationSortStability stability,
+// size_t limit, int nan_direction_hint,
+// Permutation& res) const {
+// get_permutation_impl(direction, stability, limit, nan_direction_hint, res, &collator);
+// }
+
+// void ColumnStruct::update_permutation_with_collation(const Collator& collator,
+// IColumn::PermutationSortDirection direction,
+// IColumn::PermutationSortStability stability,
+// size_t limit, int nan_direction_hint,
+// Permutation& res,
+// EqualRanges& equal_ranges) const {
+// update_permutation_impl(direction, stability, limit, nan_direction_hint, res, equal_ranges,
+// &collator);
+// }
+
+// void ColumnStruct::gather(ColumnGathererStream& gatherer) {
+// gatherer.gather(*this);
+// }
+
+void ColumnStruct::reserve(size_t n) {
+ const size_t tuple_size = columns.size();
+ for (size_t i = 0; i < tuple_size; ++i) {
+ get_column(i).reserve(n);
+ }
+}
+
+size_t ColumnStruct::byte_size() const {
+ size_t res = 0;
+ for (const auto& column : columns) {
+ res += column->byte_size();
+ }
+ return res;
+}
+
+// size_t ColumnStruct::byte_size_at(size_t n) const {
+// size_t res = 0;
+// for (const auto& column : columns) {
+// res += column->byte_size_at(n);
+// }
+// return res;
+// }
+
+// void ColumnStruct::ensure_ownership() {
+// const size_t tuple_size = columns.size();
+// for (size_t i = 0; i < tuple_size; ++i) {
+// get_column(i).ensure_ownership();
+// }
+// }
+
+size_t ColumnStruct::allocated_bytes() const {
+ size_t res = 0;
+ for (const auto& column : columns) {
+ res += column->allocated_bytes();
+ }
+ return res;
+}
+
+void ColumnStruct::protect() {
+ for (auto& column : columns) {
+ column->protect();
+ }
+}
+
+void ColumnStruct::get_extremes(Field& min, Field& max) const {
+ const size_t tuple_size = columns.size();
+
+ Tuple min_tuple(tuple_size);
+ Tuple max_tuple(tuple_size);
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ columns[i]->get_extremes(min_tuple[i], max_tuple[i]);
+ }
+
+ min = min_tuple;
+ max = max_tuple;
+}
+
+void ColumnStruct::for_each_subcolumn(ColumnCallback callback) {
+ for (auto& column : columns) {
+ callback(column);
+ }
+}
+
+bool ColumnStruct::structure_equals(const IColumn& rhs) const {
+ if (const auto* rhs_tuple = typeid_cast<const ColumnStruct*>(&rhs)) {
+ const size_t tuple_size = columns.size();
+ if (tuple_size != rhs_tuple->columns.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < tuple_size; ++i) {
+ if (!columns[i]->structure_equals(*rhs_tuple->columns[i])) {
+ return false;
+ }
+ }
+ return true;
+ } else {
+ return false;
+ }
+}
+
+// void ColumnStruct::for_each_subcolumn_recursively(ColumnCallback callback) {
+// for (auto& column : columns) {
+// callback(column);
+// column->for_each_subcolumn_recursively(callback);
+// }
+// }
+
+// bool ColumnStruct::is_collation_supported() const {
+// for (const auto& column : columns) {
+// if (column->is_collation_supported()) {
+// return true;
+// }
+// }
+// return false;
+// }
+
+// ColumnPtr ColumnStruct::compress() const {
+// size_t byte_size = 0;
+// Columns compressed;
+// compressed.reserve(columns.size());
+// for (const auto& column : columns) {
+// auto compressed_column = column->compress();
+// byte_size += compressed_column->byteSize();
+// compressed.emplace_back(std::move(compressed_column));
+// }
+
+// return ColumnCompressed::create(size(), byte_size,
+// [compressed = std::move(compressed)]() mutable {
+// for (auto& column : compressed) {
+// column = column->decompress();
+// }
+// return ColumnStruct::create(compressed);
+// });
+// }
+
+// double ColumnStruct::get_ratio_of_default_rows(double sample_ratio) const {
+// return get_ratio_of_default_rows_impl<ColumnStruct>(sample_ratio);
+// }
+
+// void ColumnStruct::get_indices_of_nondefault_rows(Offsets& indices, size_t from,
+// size_t limit) const {
+// return get_indices_of_nondefault_rows_impl<ColumnStruct>(indices, from, limit);
+// }
+
+// void ColumnStruct::finalize() {
+// for (auto& column : columns) {
+// column->finalize();
+// }
+// }
+
+// bool ColumnStruct::is_finalized() const {
+// return std::all_of(columns.begin(), columns.end(),
+// [](const auto& column) { return column->is_finalized(); });
+// }
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/columns/column_struct.h b/be/src/vec/columns/column_struct.h
new file mode 100644
index 0000000000..895a2796b7
--- /dev/null
+++ b/be/src/vec/columns/column_struct.h
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/Columns/ColumnTuple.h
+// and modified by Doris
+
+/********************************************************************************
+// doris/core/be/src/vec/core/field.h
+class Field;
+using FieldVector = std::vector<Field>;
+
+/// Array and Tuple use the same storage type -- FieldVector, but we declare
+/// distinct types for them, so that the caller can choose whether it wants to
+/// construct a Field of Array or a Tuple type. An alternative approach would be
+/// to construct both of these types from FieldVector, and have the caller
+/// specify the desired Field type explicitly.
+
+#define DEFINE_FIELD_VECTOR(X) \
+ struct X : public FieldVector { \
+ using FieldVector::FieldVector; \
+ }
+
+DEFINE_FIELD_VECTOR(Array);
+DEFINE_FIELD_VECTOR(Tuple);
+
+#undef DEFINE_FIELD_VECTOR
+
+// defination of some pointer
+using WrappedPtr = chameleon_ptr<Derived>;
+
+using Ptr = immutable_ptr<Derived>;
+using ColumnPtr = IColumn::Ptr;
+using Columns = std::vector<ColumnPtr>;
+using MutablePtr = mutable_ptr<Derived>;
+using MutableColumnPtr = IColumn::MutablePtr;
+using MutableColumns = std::vector<MutableColumnPtr>;
+****************************************************************************/
+
+#pragma once
+
+#include "vec/columns/column.h"
+#include "vec/columns/column_impl.h"
+#include "vec/columns/column_vector.h"
+#include "vec/common/arena.h"
+#include "vec/common/assert_cast.h"
+#include "vec/common/typeid_cast.h"
+#include "vec/core/field.h"
+#include "vec/core/types.h"
+
+namespace doris::vectorized {
+
+/** Column, that is just group of few another columns.
+ *
+ * For constant Tuples, see ColumnConst.
+ * Mixed constant/non-constant columns is prohibited in tuple
+ * for implementation simplicity.
+ */
+class ColumnStruct final : public COWHelper<IColumn, ColumnStruct> {
+private:
+ friend class COWHelper<IColumn, ColumnStruct>;
+
+ using TupleColumns = std::vector<WrappedPtr>;
+ TupleColumns columns;
+
+ template <bool positive>
+ struct Less;
+
+ ColumnStruct(Columns&& columns);
+ ColumnStruct(TupleColumns&& tuple_columns);
+ explicit ColumnStruct(MutableColumns&& mutable_columns);
+ ColumnStruct(const ColumnStruct&) = default;
+
+public:
+ /** Create immutable column using immutable arguments. This arguments may be shared with other columns.
+ * Use IColumn::mutate in order to make mutable column and mutate shared nested columns.
+ */
+ using Base = COWHelper<IColumn, ColumnStruct>;
+ static Ptr create(Columns& columns);
+ static Ptr create(MutableColumns& columns);
+ static Ptr create(TupleColumns& columns);
+ static Ptr create(Columns&& arg) { return create(arg); }
+
+ template <typename... Args>
+ static MutablePtr create(Args&&... args) {
+ return Base::create(std::forward<Args>(args)...);
+ }
+
+ std::string get_name() const override;
+ const char* get_family_name() const override { return "Struct"; }
+ TypeIndex get_data_type() const { return TypeIndex::Struct; }
+
+ MutableColumnPtr clone_empty() const override;
+ MutableColumnPtr clone_resized(size_t size) const override;
+
+ size_t size() const override { return columns.at(0)->size(); }
+
+ Field operator[](size_t n) const override;
+ void get(size_t n, Field& res) const override;
+
+ bool is_default_at(size_t n) const override;
+ StringRef get_data_at(size_t n) const override;
+ void insert_data(const char* pos, size_t length) override;
+ void insert(const Field& x) override;
+ void insert_from(const IColumn& src_, size_t n) override;
+ void insert_default() override;
+ void pop_back(size_t n) override;
+ StringRef serialize_value_into_arena(size_t n, Arena& arena, char const*& begin) const override;
+ const char* deserialize_and_insert_from_arena(const char* pos) override;
+ void update_hash_with_value(size_t n, SipHash& hash) const override;
+
+ // const char * skip_serialized_in_arena(const char * pos) const override;
+ // void update_weak_hash32(WeakHash32 & hash) const override;
+ // void update_hash_fast(SipHash & hash) const override;
+
+ void insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) override {
+ LOG(FATAL) << "insert_indices_from not implemented";
+ }
+
+ void get_permutation(bool reverse, size_t limit, int nan_direction_hint,
+ Permutation& res) const override {
+ LOG(FATAL) << "get_permutation not implemented";
+ }
+ void append_data_by_selector(MutableColumnPtr& res, const Selector& selector) const override {
+ return append_data_by_selector_impl<ColumnStruct>(res, selector);
+ }
+ void replace_column_data(const IColumn&, size_t row, size_t self_row = 0) override {
+ LOG(FATAL) << "replace_column_data not implemented";
+ }
+ void replace_column_data_default(size_t self_row = 0) override {
+ LOG(FATAL) << "replace_column_data_default not implemented";
+ }
+
+ void insert_range_from(const IColumn& src, size_t start, size_t length) override;
+ ColumnPtr filter(const Filter& filt, ssize_t result_size_hint) const override;
+ ColumnPtr permute(const Permutation& perm, size_t limit) const override;
+ ColumnPtr replicate(const Offsets& offsets) const override;
+ MutableColumns scatter(ColumnIndex num_columns, const Selector& selector) const override;
+
+ // ColumnPtr index(const IColumn & indexes, size_t limit) const override;
+ // void expand(const Filter & mask, bool inverted) override;
+ // void gather(ColumnGathererStream & gatherer_stream) override;
+ // bool has_equal_values() const override;
+
+ // void compare_column(const IColumn& rhs, size_t rhs_row_num, PaddedPODArray<UInt64>* row_indexes,
+ // PaddedPODArray<Int8>& compare_results, int direction,
+ // int nan_direction_hint) const override;
+ // int compare_at_with_collation(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint,
+ // const Collator& collator) const override;
+
+ int compare_at(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) const override;
+ void get_extremes(Field& min, Field& max) const override;
+
+ // void get_permutation(IColumn::PermutationSortDirection direction,
+ // IColumn::PermutationSortStability stability, size_t limit,
+ // int nan_direction_hint, IColumn::Permutation& res) const override;
+ // void update_permutation(IColumn::PermutationSortDirection direction,
+ // IColumn::PermutationSortStability stability, size_t limit,
+ // int nan_direction_hint, IColumn::Permutation& res,
+ // EqualRanges& equal_ranges) const override;
+ // void get_permutation_with_collation(const Collator& collator,
+ // IColumn::PermutationSortDirection direction,
+ // IColumn::PermutationSortStability stability, size_t limit,
+ // int nan_direction_hint,
+ // IColumn::Permutation& res) const override;
+ // void update_permutation_with_collation(const Collator& collator,
+ // IColumn::PermutationSortDirection direction,
+ // IColumn::PermutationSortStability stability,
+ // size_t limit, int nan_direction_hint,
+ // IColumn::Permutation& res,
+ // EqualRanges& equal_ranges) const override;
+
+ void reserve(size_t n) override;
+ size_t byte_size() const override;
+
+ // size_t byte_size_at(size_t n) const override;
+ // void ensure_ownership() override;
+
+ size_t allocated_bytes() const override;
+ void protect() override;
+ void for_each_subcolumn(ColumnCallback callback) override;
+ bool structure_equals(const IColumn& rhs) const override;
+
+ // void for_each_subcolumn_recursively(ColumnCallback callback) override;
+ // bool is_collation_supported() const override;
+ // ColumnPtr compress() const override;
+ // double get_ratio_of_default_rows(double sample_ratio) const override;
+ // void get_indices_of_nondefault_rows(Offsets & indices, size_t from, size_t limit) const override;
+ // void finalize() override;
+ // bool is_finalized() const override;
+
+ size_t tuple_size() const { return columns.size(); }
+
+ const IColumn& get_column(size_t idx) const { return *columns[idx]; }
+ IColumn& get_column(size_t idx) { return *columns[idx]; }
+
+ const TupleColumns& get_columns() const { return columns; }
+ Columns get_columns_copy() const { return {columns.begin(), columns.end()}; }
+
+ const ColumnPtr& get_column_ptr(size_t idx) const { return columns[idx]; }
+ ColumnPtr& get_column_ptr(size_t idx) { return columns[idx]; }
+
+private:
+ int compare_at_impl(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) const;
+
+ // void get_permutation_impl(IColumn::PermutationSortDirection direction,
+ // IColumn::PermutationSortStability stability, size_t limit,
+ // int nan_direction_hint, Permutation& res,
+ // const Collator* collator) const;
+
+ // void update_permutation_impl(IColumn::PermutationSortDirection direction,
+ // IColumn::PermutationSortStability stability, size_t limit,
+ // int nan_direction_hint, IColumn::Permutation& res,
+ // EqualRanges& equal_ranges,
+ // const Collator* collator = nullptr) const;
+};
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/core/types.h b/be/src/vec/core/types.h
index 332d310d57..aea08e11be 100644
--- a/be/src/vec/core/types.h
+++ b/be/src/vec/core/types.h
@@ -80,6 +80,7 @@ enum class TypeIndex {
FixedLengthObject,
JSONB,
Decimal128I,
+ Struct,
};
struct Consted {
@@ -525,6 +526,8 @@ inline const char* getTypeName(TypeIndex idx) {
return "FixedLengthObject";
case TypeIndex::JSONB:
return "JSONB";
+ case TypeIndex::Struct:
+ return "Struct";
}
__builtin_unreachable();
diff --git a/be/src/vec/data_types/data_type_struct.cpp b/be/src/vec/data_types/data_type_struct.cpp
new file mode 100644
index 0000000000..91aff67a40
--- /dev/null
+++ b/be/src/vec/data_types/data_type_struct.cpp
@@ -0,0 +1,361 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/DataTypeTuple.cpp
+// and modified by Doris
+
+#include "vec/data_types/data_type_struct.h"
+
+namespace doris::vectorized {
+
+namespace ErrorCodes {
+extern const int BAD_ARGUMENTS;
+extern const int DUPLICATE_COLUMN;
+extern const int EMPTY_DATA_PASSED;
+extern const int NOT_FOUND_COLUMN_IN_BLOCK;
+extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
+extern const int ILLEGAL_INDEX;
+extern const int LOGICAL_ERROR;
+} // namespace ErrorCodes
+
+DataTypeStruct::DataTypeStruct(const DataTypes& elems_)
+ : elems(elems_), have_explicit_names(false) {
+ /// Automatically assigned names in form of '1', '2', ...
+ size_t size = elems.size();
+ names.resize(size);
+ for (size_t i = 0; i < size; ++i) {
+ names[i] = std::to_string(i + 1);
+ }
+}
+
+static std::optional<Exception> check_tuple_names(const Strings& names) {
+ std::unordered_set<String> names_set;
+ for (const auto& name : names) {
+ if (name.empty()) {
+ return Exception("Names of tuple elements cannot be empty", ErrorCodes::BAD_ARGUMENTS);
+ }
+
+ if (!names_set.insert(name).second) {
+ return Exception("Names of tuple elements must be unique",
+ ErrorCodes::DUPLICATE_COLUMN);
+ }
+ }
+
+ return {};
+}
+
+DataTypeStruct::DataTypeStruct(const DataTypes& elems_, const Strings& names_)
+ : elems(elems_), names(names_), have_explicit_names(true) {
+ size_t size = elems.size();
+ if (names.size() != size) {
+ throw Exception("Wrong number of names passed to constructor of DataTypeStruct",
+ ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
+ }
+
+ if (auto exception = check_tuple_names(names)) {
+ throw std::move(*exception);
+ }
+}
+
+std::string DataTypeStruct::do_get_name() const {
+ size_t size = elems.size();
+ std::stringstream s;
+
+ s << "Struct(";
+ for (size_t i = 0; i < size; ++i) {
+ if (i != 0) {
+ s << ", ";
+ }
+
+ // if (have_explicit_names) {
+ // s << back_quote_if_need(names[i]) << ' ';
+ // }
+
+ s << elems[i]->get_name();
+ }
+ s << ")";
+
+ return s.str();
+}
+
+static inline IColumn& extract_element_column(IColumn& column, size_t idx) {
+ return assert_cast<ColumnStruct&>(column).get_column(idx);
+}
+
+template <typename F>
+static void add_element_safe(const DataTypes& elems, IColumn& column, F&& impl) {
+ /// We use the assumption that tuples of zero size do not exist.
+ size_t old_size = column.size();
+
+ try {
+ impl();
+
+ // Check that all columns now have the same size.
+ size_t new_size = column.size();
+
+ // for (auto i : collections::range(0, elems.size())) {
+ for (auto i = 0; i < elems.size(); i++) {
+ const auto& element_column = extract_element_column(column, i);
+ if (element_column.size() != new_size) {
+ // This is not a logical error because it may work with
+ // user-supplied data.
+ throw Exception("Cannot read a tuple because not all elements are present",
+ ErrorCodes::SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH);
+ }
+ }
+ } catch (...) {
+ // for (const auto& i : collections::range(0, elems.size())) {
+ for (auto i = 0; i < elems.size(); i++) {
+ auto& element_column = extract_element_column(column, i);
+
+ if (element_column.size() > old_size) {
+ element_column.pop_back(1);
+ }
+ }
+
+ throw;
+ }
+}
+
+MutableColumnPtr DataTypeStruct::create_column() const {
+ size_t size = elems.size();
+ MutableColumns tuple_columns(size);
+ for (size_t i = 0; i < size; ++i) {
+ tuple_columns[i] = elems[i]->create_column();
+ }
+ return ColumnStruct::create(std::move(tuple_columns));
+}
+
+// MutableColumnPtr DataTypeStruct::create_column(const ISerialization& serialization) const {
+// /// If we read subcolumn of nested Tuple, it may be wrapped to SerializationNamed
+// /// several times to allow to reconstruct the substream path name.
+// /// Here we don't need substream path name, so we drop first several wrapper serializations.
+
+// const auto* current_serialization = &serialization;
+// while (const auto* serialization_named =
+// typeid_cast<const SerializationNamed*>(current_serialization))
+// current_serialization = serialization_named->get_nested().get();
+
+// const auto* serialization_tuple = typeid_cast<const SerializationTuple*>(current_serialization);
+// if (!serialization_tuple)
+// throw Exception(ErrorCodes::LOGICAL_ERROR,
+// "Unexpected serialization to create column of type Tuple");
+
+// const auto& element_serializations = serialization_tuple->getElementsSerializations();
+
+// size_t size = elems.size();
+// assert(element_serializations.size() == size);
+// MutableColumns tuple_columns(size);
+// for (size_t i = 0; i < size; ++i) {
+// tuple_columns[i] = elems[i]->create_column(*element_serializations[i]->get_nested());
+// }
+
+// return ColumnStruct::create(std::move(tuple_columns));
+// }
+
+// Field DataTypeStruct::get_default() const {
+// return Tuple(collections::map<Tuple>(
+// elems, [](const DataTypePtr& elem) { return elem->get_default(); }));
+// }
+
+void DataTypeStruct::insert_default_into(IColumn& column) const {
+ add_element_safe(elems, column, [&] {
+ // for (const auto& i : collections::range(0, elems.size()))
+ for (auto i = 0; i < elems.size(); i++) {
+ elems[i]->insert_default_into(extract_element_column(column, i));
+ }
+ });
+}
+
+bool DataTypeStruct::equals(const IDataType& rhs) const {
+ if (typeid(rhs) != typeid(*this)) {
+ return false;
+ }
+
+ const DataTypeStruct& rhs_tuple = static_cast<const DataTypeStruct&>(rhs);
+
+ size_t size = elems.size();
+ if (size != rhs_tuple.elems.size()) {
+ return false;
+ }
+
+ for (size_t i = 0; i < size; ++i) {
+ if (!elems[i]->equals(*rhs_tuple.elems[i]) || names[i] != rhs_tuple.names[i]) {
+ return false;
+ }
+ }
+
+ return true;
+}
+
+size_t DataTypeStruct::get_position_by_name(const String& name) const {
+ size_t size = elems.size();
+ for (size_t i = 0; i < size; ++i) {
+ if (names[i] == name) {
+ return i;
+ }
+ }
+ throw Exception("Struct doesn't have element with name '" + name + "'",
+ ErrorCodes::NOT_FOUND_COLUMN_IN_BLOCK);
+}
+
+std::optional<size_t> DataTypeStruct::try_get_position_by_name(const String& name) const {
+ size_t size = elems.size();
+ for (size_t i = 0; i < size; ++i) {
+ if (names[i] == name) {
+ return std::optional<size_t>(i);
+ }
+ }
+ return std::nullopt;
+}
+
+String DataTypeStruct::get_name_by_position(size_t i) const {
+ if (i == 0 || i > names.size()) {
+ fmt::memory_buffer error_msg;
+ fmt::format_to(error_msg, "Index of tuple element ({}) if out range ([1, {}])", i,
+ names.size());
+ throw Exception(fmt::to_string(error_msg), ErrorCodes::ILLEGAL_INDEX);
+ }
+
+ return names[i - 1];
+}
+
+bool DataTypeStruct::text_can_contain_only_valid_utf8() const {
+ return std::all_of(elems.begin(), elems.end(),
+ [](auto&& elem) { return elem->text_can_contain_only_valid_utf8(); });
+}
+
+bool DataTypeStruct::have_maximum_size_of_value() const {
+ return std::all_of(elems.begin(), elems.end(),
+ [](auto&& elem) { return elem->have_maximum_size_of_value(); });
+}
+
+bool DataTypeStruct::is_comparable() const {
+ return std::all_of(elems.begin(), elems.end(),
+ [](auto&& elem) { return elem->is_comparable(); });
+}
+
+size_t DataTypeStruct::get_maximum_size_of_value_in_memory() const {
+ size_t res = 0;
+ for (const auto& elem : elems) {
+ res += elem->get_maximum_size_of_value_in_memory();
+ }
+ return res;
+}
+
+size_t DataTypeStruct::get_size_of_value_in_memory() const {
+ size_t res = 0;
+ for (const auto& elem : elems) {
+ res += elem->get_size_of_value_in_memory();
+ }
+ return res;
+}
+
+// bool DataTypeStruct::has_dynamic_subcolumns() const {
+// return std::any_of(elems.begin(), elems.end(),
+// [](auto&& elem) { return elem->has_dynamic_subcolumns(); });
+// }
+
+// SerializationPtr DataTypeStruct::do_get_default_serialization() const {
+// SerializationTuple::ElementSerializations serializations(elems.size());
+
+// for (size_t i = 0; i < elems.size(); ++i) {
+// String elem_name = have_explicit_names ? names[i] : toString(i + 1);
+// auto serialization = elems[i]->get_default_serialization();
+// serializations[i] = std::make_shared<SerializationNamed>(serialization, elem_name);
+// }
+
+// return std::make_shared<SerializationTuple>(std::move(serializations), have_explicit_names);
+// }
+
+// SerializationPtr DataTypeStruct::get_serialization(const SerializationInfo& info) const {
+// SerializationTuple::ElementSerializations serializations(elems.size());
+// const auto& info_tuple = assert_cast<const SerializationInfoTuple&>(info);
+
+// for (size_t i = 0; i < elems.size(); ++i) {
+// String elem_name = have_explicit_names ? names[i] : toString(i + 1);
+// auto serialization = elems[i]->get_serialization(*info_tuple.get_element_info(i));
+// serializations[i] = std::make_shared<SerializationNamed>(serialization, elem_name);
+// }
+
+// return std::make_shared<SerializationTuple>(std::move(serializations), have_explicit_names);
+// }
+
+// MutableSerializationInfoPtr DataTypeStruct::create_serialization_info(
+// const SerializationInfo::Settings& settings) const {
+// MutableSerializationInfos infos;
+// infos.reserve(elems.size());
+// for (const auto& elem : elems) {
+// infos.push_back(elem->create_serializationInfo(settings));
+// }
+
+// return std::make_shared<SerializationInfoTuple>(std::move(infos), names, settings);
+// }
+
+// SerializationInfoPtr DataTypeStruct::get_serialization_info(const IColumn& column) const {
+// if (const auto* column_const = check_and_get_column<ColumnConst>(&column)) {
+// return get_serialization_info(column_const->get_data_column());
+// }
+
+// MutableSerializationInfos infos;
+// infos.reserve(elems.size());
+
+// const auto& column_tuple = assert_cast<const ColumnStruct&>(column);
+// assert(elems.size() == column_tuple.get_columns().size());
+
+// for (size_t i = 0; i < elems.size(); ++i) {
+// auto element_info = elems[i]->get_serialization_info(column_tuple.getColumn(i));
+// infos.push_back(const_pointer_cast<SerializationInfo>(element_info));
+// }
+
+// return std::make_shared<SerializationInfoTuple>(std::move(infos), names,
+// SerializationInfo::Settings {});
+// }
+
+// static DataTypePtr create(const ASTPtr& arguments) {
+// if (!arguments || arguments->children.empty())
+// throw Exception("Struct cannot be empty", ErrorCodes::EMPTY_DATA_PASSED);
+
+// DataTypes nested_types;
+// nested_types.reserve(arguments->children.size());
+
+// Strings names;
+// names.reserve(arguments->children.size());
+
+// for (const ASTPtr& child : arguments->children) {
+// if (const auto* name_and_type_pair = child->as<ASTNameTypePair>()) {
+// nested_types.emplace_back(DataTypeFactory::instance().get(name_and_type_pair->type));
+// names.emplace_back(name_and_type_pair->name);
+// } else
+// nested_types.emplace_back(DataTypeFactory::instance().get(child));
+// }
+
+// if (names.empty())
+// return std::make_shared<DataTypeStruct>(nested_types);
+// else if (names.size() != nested_types.size())
+// throw Exception("Names are specified not for all elements of Struct type",
+// ErrorCodes::BAD_ARGUMENTS);
+// else
+// return std::make_shared<DataTypeStruct>(nested_types, names);
+// }
+
+// void registerDataTypeStruct(DataTypeFactory& factory) {
+// factory.registerDataType("Struct", create);
+// }
+
+} // namespace doris::vectorized
diff --git a/be/src/vec/data_types/data_type_struct.h b/be/src/vec/data_types/data_type_struct.h
new file mode 100644
index 0000000000..4201583ef4
--- /dev/null
+++ b/be/src/vec/data_types/data_type_struct.h
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/ClickHouse/ClickHouse/blob/master/src/DataTypes/DataTypeTuple.h
+// and modified by Doris
+
+#pragma once
+
+#include <exception>
+
+#include "gen_cpp/data.pb.h"
+#include "util/stack_util.h"
+#include "vec/columns/column_array.h"
+#include "vec/columns/column_nullable.h"
+#include "vec/columns/column_struct.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_nullable.h"
+
+namespace doris::vectorized {
+
+/** Struct data type.
+ * Used as an intermediate result when evaluating expressions.
+ * Also can be used as a column - the result of the query execution.
+ *
+ * Struct elements can have names.
+ * If an element is unnamed, it will have automatically assigned name like '1', '2', '3' corresponding to its position.
+ * Manually assigned names must not begin with digit. Names must be unique.
+ *
+ * All tuples with same size and types of elements are equivalent for expressions, regardless to names of elements.
+ */
+class DataTypeStruct final : public IDataType {
+private:
+ // using DataTypePtr = std::shared_ptr<const IDataType>;
+ // using DataTypes = std::vector<DataTypePtr>;
+ // using Strings = std::vector<std::string>;
+
+ DataTypes elems;
+ Strings names;
+ bool have_explicit_names;
+
+public:
+ // static constexpr bool is_parametric = true;
+
+ explicit DataTypeStruct(const DataTypes& elems);
+ DataTypeStruct(const DataTypes& elems, const Strings& names);
+
+ TypeIndex get_type_id() const override { return TypeIndex::Struct; }
+ std::string do_get_name() const override;
+ const char* get_family_name() const override { return "Struct"; }
+
+ bool can_be_inside_nullable() const override { return false; }
+ bool supports_sparse_serialization() const { return true; }
+
+ MutableColumnPtr create_column() const override;
+ // MutableColumnPtr create_column(const ISerialization& serialization) const override;
+
+ Field get_default() const override;
+ void insert_default_into(IColumn& column) const override;
+
+ bool equals(const IDataType& rhs) const override;
+
+ bool get_is_parametric() const override { return true; }
+ bool have_subtypes() const override { return !elems.empty(); }
+ bool is_comparable() const override;
+ bool text_can_contain_only_valid_utf8() const override;
+ bool have_maximum_size_of_value() const override;
+ bool has_dynamic_subcolumns() const;
+ size_t get_maximum_size_of_value_in_memory() const override;
+ size_t get_size_of_value_in_memory() const override;
+
+ const DataTypePtr& get_element(size_t i) const { return elems[i]; }
+ const DataTypes& get_elements() const { return elems; }
+ const Strings& get_element_names() const { return names; }
+
+ size_t get_position_by_name(const String& name) const;
+ std::optional<size_t> try_get_position_by_name(const String& name) const;
+ String get_name_by_position(size_t i) const;
+
+ int64_t get_uncompressed_serialized_bytes(const IColumn& column,
+ int be_exec_version) const override {
+ LOG(FATAL) << "get_uncompressed_serialized_bytes not implemented";
+ }
+
+ char* serialize(const IColumn& column, char* buf, int be_exec_version) const override {
+ LOG(FATAL) << "serialize not implemented";
+ }
+
+ const char* deserialize(const char* buf, IColumn* column, int be_exec_version) const override {
+ LOG(FATAL) << "serialize not implemented";
+ }
+
+ // bool is_parametric() const { return true; }
+ // SerializationPtr do_get_default_serialization() const override;
+ // SerializationPtr get_serialization(const SerializationInfo& info) const override;
+ // MutableSerializationInfoPtr create_serialization_info(
+ // const SerializationInfo::Settings& settings) const override;
+ // SerializationInfoPtr get_serialization_info(const IColumn& column) const override;
+ // bool have_explicit_names() const { return have_explicit_names; }
+};
+
+} // namespace doris::vectorized
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 09/09: save (#15908)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c5be8ec48e233b1ba163b30bdc8356af1be661d3
Author: xy720 <22...@users.noreply.github.com>
AuthorDate: Fri Jan 13 17:46:49 2023 +0800
save (#15908)
---
be/src/vec/data_types/data_type_factory.cpp | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/data_types/data_type_factory.cpp b/be/src/vec/data_types/data_type_factory.cpp
index c4fbfbd61b..78c5fae6b6 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -38,7 +38,7 @@ DataTypePtr DataTypeFactory::create_data_type(const doris::Field& col_desc) {
names.reserve(field_size);
for (size_t i = 0; i < field_size; i++) {
dataTypes.push_back(create_data_type(*col_desc.get_sub_field(i)));
- names.push_back(col_desc.get_sub_field(i).name());
+ names.push_back(col_desc.get_sub_field(i)->name());
}
nested = std::make_shared<DataTypeStruct>(dataTypes, names);
} else {
@@ -66,7 +66,7 @@ DataTypePtr DataTypeFactory::create_data_type(const TabletColumn& col_desc, bool
names.reserve(col_size);
for (size_t i = 0; i < col_size; i++) {
dataTypes.push_back(create_data_type(col_desc.get_sub_column(i)));
- names.push_back(col_desc.get_sub_field(i).name());
+ names.push_back(col_desc.get_sub_column(i).name());
}
nested = std::make_shared<DataTypeStruct>(dataTypes, names);
} else {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 08/09: save (#15906)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 9b43fe8da1d06153ab4a0d7b3e42535aa0d50ae8
Author: xy720 <22...@users.noreply.github.com>
AuthorDate: Fri Jan 13 17:29:42 2023 +0800
save (#15906)
---
be/src/vec/data_types/data_type_factory.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/be/src/vec/data_types/data_type_factory.cpp b/be/src/vec/data_types/data_type_factory.cpp
index 553f117a48..c4fbfbd61b 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -38,7 +38,7 @@ DataTypePtr DataTypeFactory::create_data_type(const doris::Field& col_desc) {
names.reserve(field_size);
for (size_t i = 0; i < field_size; i++) {
dataTypes.push_back(create_data_type(*col_desc.get_sub_field(i)));
- names.push_back(col_desc.name());
+ names.push_back(col_desc.get_sub_field(i).name());
}
nested = std::make_shared<DataTypeStruct>(dataTypes, names);
} else {
@@ -66,7 +66,7 @@ DataTypePtr DataTypeFactory::create_data_type(const TabletColumn& col_desc, bool
names.reserve(col_size);
for (size_t i = 0; i < col_size; i++) {
dataTypes.push_back(create_data_type(col_desc.get_sub_column(i)));
- names.push_back(col_desc.name());
+ names.push_back(col_desc.get_sub_field(i).name());
}
nested = std::make_shared<DataTypeStruct>(dataTypes, names);
} else {
@@ -347,7 +347,7 @@ DataTypePtr DataTypeFactory::create_data_type(const PColumnMeta& pcolumn) {
names.reserve(col_size);
for (size_t i = 0; i < col_size; i++) {
dataTypes.push_back(create_data_type(pcolumn.children(i)));
- names.push_back(pcolumn.name());
+ names.push_back(pcolumn.children(i).name());
}
nested = std::make_shared<DataTypeStruct>(dataTypes, names);
break;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 05/09: [WIP](struct-type) support struct-type in vectorize engine (#15665)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 550e5b23f06b41ba9b3b61b82dc316c27112c03e
Author: xy720 <22...@users.noreply.github.com>
AuthorDate: Thu Jan 12 14:40:08 2023 +0800
[WIP](struct-type) support struct-type in vectorize engine (#15665)
---
be/src/olap/aggregate_func.cpp | 2 +
be/src/olap/aggregate_func.h | 15 ++
be/src/olap/field.h | 57 ++++-
be/src/olap/olap_define.h | 3 +
be/src/olap/rowset/segment_v2/column_reader.cpp | 112 ++++++++++
be/src/olap/rowset/segment_v2/column_reader.h | 38 ++++
be/src/olap/rowset/segment_v2/column_writer.cpp | 174 +++++++++++++++
be/src/olap/rowset/segment_v2/column_writer.h | 51 +++++
be/src/olap/rowset/segment_v2/segment_writer.cpp | 9 +
be/src/olap/tablet_meta.cpp | 8 +-
be/src/olap/tablet_schema.cpp | 20 +-
be/src/olap/types.cpp | 52 ++++-
be/src/olap/types.h | 245 ++++++++++++++++++++-
be/src/runtime/CMakeLists.txt | 1 +
be/src/runtime/primitive_type.cpp | 15 ++
be/src/runtime/primitive_type.h | 1 +
be/src/runtime/struct_value.cpp | 27 +++
be/src/runtime/struct_value.h | 67 ++++++
be/src/runtime/types.cpp | 48 +++-
be/src/runtime/types.h | 2 +-
be/src/udf/udf.h | 1 +
be/src/vec/CMakeLists.txt | 1 +
be/src/vec/columns/column_struct.cpp | 34 +--
be/src/vec/columns/column_struct.h | 15 +-
be/src/vec/data_types/data_type_factory.cpp | 41 +++-
be/src/vec/data_types/data_type_factory.hpp | 4 +
be/src/vec/data_types/data_type_struct.cpp | 33 ++-
be/src/vec/data_types/data_type_struct.h | 12 +-
be/src/vec/exprs/vcast_expr.cpp | 1 +
be/src/vec/exprs/vexpr.cpp | 5 +
be/src/vec/exprs/vstruct_literal.cpp | 37 ++++
be/src/vec/exprs/vstruct_literal.h | 34 +++
be/src/vec/functions/function_cast.h | 1 +
be/src/vec/olap/olap_data_convertor.cpp | 62 ++++++
be/src/vec/olap/olap_data_convertor.h | 21 ++
be/src/vec/sink/vmysql_result_writer.cpp | 100 +++++++--
be/src/vec/sink/vmysql_result_writer.h | 2 +-
be/src/vec/sink/vtablet_sink.cpp | 3 +-
fe/fe-core/src/main/cup/sql_parser.cup | 33 ++-
.../java/org/apache/doris/analysis/ColumnDef.java | 4 +
.../org/apache/doris/analysis/CreateTableStmt.java | 12 +-
.../main/java/org/apache/doris/analysis/Expr.java | 7 +-
.../org/apache/doris/analysis/StructLiteral.java | 150 +++++++++++++
.../java/org/apache/doris/analysis/Subquery.java | 2 +-
.../java/org/apache/doris/analysis/TypeDef.java | 47 ++--
.../java/org/apache/doris/catalog/ArrayType.java | 10 +
.../main/java/org/apache/doris/catalog/Column.java | 66 +++---
.../java/org/apache/doris/catalog/MapType.java | 5 +
.../org/apache/doris/catalog/PrimitiveType.java | 5 +-
.../java/org/apache/doris/catalog/StructField.java | 55 ++++-
.../java/org/apache/doris/catalog/StructType.java | 94 +++++++-
.../main/java/org/apache/doris/catalog/Type.java | 24 +-
fe/fe-core/src/main/jflex/sql_scanner.flex | 4 +
gensrc/proto/types.proto | 1 +
gensrc/thrift/Exprs.thrift | 1 +
gensrc/thrift/Types.thrift | 1 +
56 files changed, 1702 insertions(+), 173 deletions(-)
diff --git a/be/src/olap/aggregate_func.cpp b/be/src/olap/aggregate_func.cpp
index d943e94015..e8b45b52ab 100644
--- a/be/src/olap/aggregate_func.cpp
+++ b/be/src/olap/aggregate_func.cpp
@@ -144,6 +144,8 @@ AggregateFuncResolver::AggregateFuncResolver() {
OLAP_FIELD_TYPE_DECIMAL128I>();
add_aggregate_mapping<OLAP_FIELD_AGGREGATION_NONE, OLAP_FIELD_TYPE_ARRAY,
OLAP_FIELD_TYPE_ARRAY>();
+ // struct types
+ add_aggregate_mapping<OLAP_FIELD_AGGREGATION_NONE, OLAP_FIELD_TYPE_STRUCT>();
// Min Aggregate Function
add_aggregate_mapping<OLAP_FIELD_AGGREGATION_MIN, OLAP_FIELD_TYPE_TINYINT>();
diff --git a/be/src/olap/aggregate_func.h b/be/src/olap/aggregate_func.h
index 5a397b658f..b72be9a7cd 100644
--- a/be/src/olap/aggregate_func.h
+++ b/be/src/olap/aggregate_func.h
@@ -107,6 +107,21 @@ struct BaseAggregateFuncs {
static void finalize(RowCursorCell* src, MemPool* mem_pool) {}
};
+// Define an empty trait for struct type because
+// we now only support struct type in dup key table.
+template <FieldType sub_type>
+struct BaseAggregateFuncs<OLAP_FIELD_TYPE_STRUCT, sub_type> {
+ // Default init do nothing, use direct_copy in struct field instead.
+ static void init(RowCursorCell* dst, const char* src, bool src_null, MemPool* mem_pool,
+ ObjectPool* agg_pool) {}
+
+ // Default update do nothing.
+ static void update(RowCursorCell* dst, const RowCursorCell& src, MemPool* mem_pool) {}
+
+ // Default finalize do nothing.
+ static void finalize(RowCursorCell* src, MemPool* mem_pool) {}
+};
+
template <FieldType sub_type>
struct BaseAggregateFuncs<OLAP_FIELD_TYPE_ARRAY, sub_type> {
static void init(RowCursorCell* dst, const char* src, bool src_null, MemPool* mem_pool,
diff --git a/be/src/olap/field.h b/be/src/olap/field.h
index 391f7f45be..97a516f32c 100644
--- a/be/src/olap/field.h
+++ b/be/src/olap/field.h
@@ -49,7 +49,9 @@ public:
_index_size(column.index_length()),
_is_nullable(column.is_nullable()),
_unique_id(column.unique_id()) {
- if (column.type() == OLAP_FIELD_TYPE_ARRAY) {
+ if (column.type() == OLAP_FIELD_TYPE_STRUCT) {
+ _agg_info = get_aggregate_info(column.aggregation(), column.type());
+ } else if (column.type() == OLAP_FIELD_TYPE_ARRAY) {
_agg_info = get_aggregate_info(column.aggregation(), column.type(),
column.get_sub_column(0).type());
} else {
@@ -311,6 +313,9 @@ protected:
const AggregateInfo* _agg_info;
// unit : byte
// except for strings, other types have fixed lengths
+ // Note that, the struct type itself has fixed length, but due to
+ // its number of subfields is a variable, so the actual length of
+ // a struct field is not fixed.
uint32_t _length;
// Since the length of the STRING type cannot be determined,
// only dynamic memory can be used. Mempool cannot realize realloc.
@@ -450,6 +455,38 @@ uint32_t Field::hash_code(const CellType& cell, uint32_t seed) const {
return _type_info->hash_code(cell.cell_ptr(), seed);
}
+class StructField : public Field {
+public:
+ explicit StructField(const TabletColumn& column) : Field(column) {}
+
+ void consume(RowCursorCell* dst, const char* src, bool src_null, MemPool* mem_pool,
+ ObjectPool* agg_pool) const override {
+ dst->set_is_null(src_null);
+ if (src_null) {
+ return;
+ }
+ _type_info->deep_copy(dst->mutable_cell_ptr(), src, mem_pool);
+ }
+
+ char* allocate_memory(char* cell_ptr, char* variable_ptr) const override {
+ auto struct_v = (StructValue*)cell_ptr;
+ struct_v->set_values(reinterpret_cast<void**>(variable_ptr));
+ variable_ptr += _length;
+ for (size_t i = 0; i < get_sub_field_count(); i++) {
+ variable_ptr += get_sub_field(i)->get_variable_len();
+ }
+ return variable_ptr;
+ }
+
+ size_t get_variable_len() const override {
+ size_t variable_len = _length;
+ for (size_t i = 0; i < get_sub_field_count(); i++) {
+ variable_len += get_sub_field(i)->get_variable_len();
+ }
+ return variable_len;
+ }
+};
+
class ArrayField : public Field {
public:
explicit ArrayField(const TabletColumn& column) : Field(column) {}
@@ -746,6 +783,15 @@ public:
return new VarcharField(column);
case OLAP_FIELD_TYPE_STRING:
return new StringField(column);
+ case OLAP_FIELD_TYPE_STRUCT: {
+ auto* local = new StructField(column);
+ for (uint32_t i = 0; i < column.get_subtype_count(); i++) {
+ std::unique_ptr<Field> sub_field(
+ FieldFactory::create(column.get_sub_column(i)));
+ local->add_sub_field(std::move(sub_field));
+ }
+ return local;
+ }
case OLAP_FIELD_TYPE_ARRAY: {
std::unique_ptr<Field> item_field(FieldFactory::create(column.get_sub_column(0)));
auto* local = new ArrayField(column);
@@ -786,6 +832,15 @@ public:
return new VarcharField(column);
case OLAP_FIELD_TYPE_STRING:
return new StringField(column);
+ case OLAP_FIELD_TYPE_STRUCT: {
+ auto* local = new StructField(column);
+ for (uint32_t i = 0; i < column.get_subtype_count(); i++) {
+ std::unique_ptr<Field> sub_field(
+ FieldFactory::create(column.get_sub_column(i)));
+ local->add_sub_field(std::move(sub_field));
+ }
+ return local;
+ }
case OLAP_FIELD_TYPE_ARRAY: {
std::unique_ptr<Field> item_field(FieldFactory::create(column.get_sub_column(0)));
auto* local = new ArrayField(column);
diff --git a/be/src/olap/olap_define.h b/be/src/olap/olap_define.h
index a9c92c69e2..fadbea6834 100644
--- a/be/src/olap/olap_define.h
+++ b/be/src/olap/olap_define.h
@@ -60,6 +60,9 @@ static const uint32_t OLAP_STRING_MAX_LENGTH = 2147483647;
// the max length supported for jsonb type 2G
static const uint32_t OLAP_JSONB_MAX_LENGTH = 2147483647;
+// the max length supported for struct, but excluding the length of its subtypes.
+static const uint16_t OLAP_STRUCT_MAX_LENGTH = 65535;
+
// the max length supported for array
static const uint16_t OLAP_ARRAY_MAX_LENGTH = 65535;
diff --git a/be/src/olap/rowset/segment_v2/column_reader.cpp b/be/src/olap/rowset/segment_v2/column_reader.cpp
index 6f49aa23cf..968caae9c5 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.cpp
+++ b/be/src/olap/rowset/segment_v2/column_reader.cpp
@@ -31,6 +31,7 @@
#include "util/rle_encoding.h" // for RleDecoder
#include "vec/columns/column.h"
#include "vec/columns/column_array.h"
+#include "vec/columns/column_struct.h"
#include "vec/core/types.h"
#include "vec/runtime/vdatetime_value.h" //for VecDateTime
@@ -49,6 +50,23 @@ Status ColumnReader::create(const ColumnReaderOptions& opts, const ColumnMetaPB&
} else {
auto type = (FieldType)meta.type();
switch (type) {
+ case FieldType::OLAP_FIELD_TYPE_STRUCT: {
+ // not support empty struct
+ DCHECK(meta.children_columns_size() >= 1);
+ // create struct column reader
+ std::unique_ptr<ColumnReader> struct_reader(
+ new ColumnReader(opts, meta, num_rows, file_reader));
+ struct_reader->_sub_readers.reserve(meta.children_columns_size());
+ for (size_t i = 0; i < meta.children_columns_size(); i++) {
+ std::unique_ptr<ColumnReader> sub_reader;
+ RETURN_IF_ERROR(ColumnReader::create(opts, meta.children_columns(i),
+ meta.children_columns(i).num_rows(),
+ file_reader, &sub_reader));
+ struct_reader->_sub_readers.push_back(std::move(sub_reader));
+ }
+ *reader = std::move(struct_reader);
+ return Status::OK();
+ }
case FieldType::OLAP_FIELD_TYPE_ARRAY: {
DCHECK(meta.children_columns_size() == 2 || meta.children_columns_size() == 3);
@@ -433,6 +451,24 @@ Status ColumnReader::new_iterator(ColumnIterator** iterator) {
} else {
auto type = (FieldType)_meta.type();
switch (type) {
+ case FieldType::OLAP_FIELD_TYPE_STRUCT: {
+ std::vector<ColumnIterator*> sub_column_iterators;
+ size_t child_size = is_nullable() ? _sub_readers.size() - 1 : _sub_readers.size();
+ sub_column_iterators.reserve(child_size);
+
+ ColumnIterator* sub_column_iterator;
+ for (size_t i = 0; i < child_size; i++) {
+ RETURN_IF_ERROR(_sub_readers[i]->new_iterator(&sub_column_iterator));
+ sub_column_iterators.push_back(sub_column_iterator);
+ }
+
+ ColumnIterator* null_iterator = nullptr;
+ if (is_nullable()) {
+ RETURN_IF_ERROR(_sub_readers[child_size]->new_iterator(&null_iterator));
+ }
+ *iterator = new StructFileColumnIterator(this, null_iterator, sub_column_iterators);
+ return Status::OK();
+ }
case FieldType::OLAP_FIELD_TYPE_ARRAY: {
ColumnIterator* item_iterator = nullptr;
RETURN_IF_ERROR(_sub_readers[0]->new_iterator(&item_iterator));
@@ -458,6 +494,82 @@ Status ColumnReader::new_iterator(ColumnIterator** iterator) {
////////////////////////////////////////////////////////////////////////////////
+StructFileColumnIterator::StructFileColumnIterator(
+ ColumnReader* reader, ColumnIterator* null_iterator,
+ std::vector<ColumnIterator*>& sub_column_iterators)
+ : _struct_reader(reader) {
+ _sub_column_iterators.resize(sub_column_iterators.size());
+ for (size_t i = 0; i < sub_column_iterators.size(); i++) {
+ _sub_column_iterators[i].reset(sub_column_iterators[i]);
+ }
+ if (_struct_reader->is_nullable()) {
+ _null_iterator.reset(null_iterator);
+ }
+}
+
+Status StructFileColumnIterator::init(const ColumnIteratorOptions& opts) {
+ for (auto& column_iterator : _sub_column_iterators) {
+ RETURN_IF_ERROR(column_iterator->init(opts));
+ }
+ if (_struct_reader->is_nullable()) {
+ RETURN_IF_ERROR(_null_iterator->init(opts));
+ }
+ return Status::OK();
+}
+
+Status StructFileColumnIterator::next_batch(size_t* n, ColumnBlockView* dst, bool* has_null) {
+ return Status::NotSupported("not supported");
+}
+
+Status StructFileColumnIterator::next_batch(size_t* n, vectorized::MutableColumnPtr& dst,
+ bool* has_null) {
+ const auto* column_struct = vectorized::check_and_get_column<vectorized::ColumnStruct>(
+ dst->is_nullable() ? static_cast<vectorized::ColumnNullable&>(*dst).get_nested_column()
+ : *dst);
+ for (size_t i = 0; i < column_struct->tuple_size(); i++) {
+ size_t num_read = *n;
+ auto sub_column_ptr = column_struct->get_column(i).assume_mutable();
+ bool column_has_null = false;
+ RETURN_IF_ERROR(
+ _sub_column_iterators[i]->next_batch(&num_read, sub_column_ptr, &column_has_null));
+ DCHECK(num_read == *n);
+ }
+
+ if (dst->is_nullable()) {
+ size_t num_read = *n;
+ auto null_map_ptr =
+ static_cast<vectorized::ColumnNullable&>(*dst).get_null_map_column_ptr();
+ bool null_signs_has_null = false;
+ RETURN_IF_ERROR(_null_iterator->next_batch(&num_read, null_map_ptr, &null_signs_has_null));
+ DCHECK(num_read == *n);
+ }
+
+ return Status::OK();
+}
+
+Status StructFileColumnIterator::seek_to_ordinal(ordinal_t ord) {
+ for (auto& column_iterator : _sub_column_iterators) {
+ RETURN_IF_ERROR(column_iterator->seek_to_ordinal(ord));
+ }
+ if (_struct_reader->is_nullable()) {
+ RETURN_IF_ERROR(_null_iterator->seek_to_ordinal(ord));
+ }
+ return Status::OK();
+}
+
+Status StructFileColumnIterator::read_by_rowids(const rowid_t* rowids, const size_t count,
+ vectorized::MutableColumnPtr& dst) {
+ for (size_t i = 0; i < count; ++i) {
+ RETURN_IF_ERROR(seek_to_ordinal(rowids[i]));
+ size_t num_read = 1;
+ RETURN_IF_ERROR(next_batch(&num_read, dst, nullptr));
+ DCHECK(num_read == 1);
+ }
+ return Status::OK();
+}
+
+////////////////////////////////////////////////////////////////////////////////
+
ArrayFileColumnIterator::ArrayFileColumnIterator(ColumnReader* reader,
FileColumnIterator* offset_reader,
ColumnIterator* item_iterator,
diff --git a/be/src/olap/rowset/segment_v2/column_reader.h b/be/src/olap/rowset/segment_v2/column_reader.h
index c5dd729d82..b10e46ea98 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.h
+++ b/be/src/olap/rowset/segment_v2/column_reader.h
@@ -393,6 +393,44 @@ public:
ordinal_t get_current_ordinal() const override { return 0; }
};
+class StructFileColumnIterator final : public ColumnIterator {
+public:
+ explicit StructFileColumnIterator(ColumnReader* reader, ColumnIterator* null_iterator,
+ std::vector<ColumnIterator*>& sub_column_iterators);
+
+ ~StructFileColumnIterator() override = default;
+
+ Status init(const ColumnIteratorOptions& opts) override;
+
+ Status next_batch(size_t* n, ColumnBlockView* dst, bool* has_null) override;
+
+ Status next_batch(size_t* n, vectorized::MutableColumnPtr& dst, bool* has_null) override;
+
+ Status read_by_rowids(const rowid_t* rowids, const size_t count,
+ vectorized::MutableColumnPtr& dst) override;
+
+ Status seek_to_first() override {
+ for (auto& column_iterator : _sub_column_iterators) {
+ RETURN_IF_ERROR(column_iterator->seek_to_first());
+ }
+ if (_struct_reader->is_nullable()) {
+ RETURN_IF_ERROR(_null_iterator->seek_to_first());
+ }
+ return Status::OK();
+ }
+
+ Status seek_to_ordinal(ordinal_t ord) override;
+
+ ordinal_t get_current_ordinal() const override {
+ return _sub_column_iterators[0]->get_current_ordinal();
+ }
+
+private:
+ ColumnReader* _struct_reader;
+ std::unique_ptr<ColumnIterator> _null_iterator;
+ std::vector<std::unique_ptr<ColumnIterator>> _sub_column_iterators;
+};
+
class ArrayFileColumnIterator final : public ColumnIterator {
public:
explicit ArrayFileColumnIterator(ColumnReader* reader, FileColumnIterator* offset_reader,
diff --git a/be/src/olap/rowset/segment_v2/column_writer.cpp b/be/src/olap/rowset/segment_v2/column_writer.cpp
index df820b141e..fdd42f9444 100644
--- a/be/src/olap/rowset/segment_v2/column_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/column_writer.cpp
@@ -88,6 +88,78 @@ Status ColumnWriter::create(const ColumnWriterOptions& opts, const TabletColumn*
return Status::OK();
} else {
switch (column->type()) {
+ case FieldType::OLAP_FIELD_TYPE_STRUCT: {
+ // not support empty struct
+ DCHECK(column->get_subtype_count() >= 1);
+ std::vector<std::unique_ptr<ColumnWriter>> sub_column_writers;
+ sub_column_writers.reserve(column->get_subtype_count());
+ for (uint32_t i = 0; i < column->get_subtype_count(); i++) {
+ const TabletColumn& sub_column = column->get_sub_column(i);
+
+ // create sub writer
+ ColumnWriterOptions column_options;
+ column_options.meta = opts.meta->mutable_children_columns(i);
+ column_options.need_zone_map = false;
+ column_options.need_bloom_filter = sub_column.is_bf_column();
+ column_options.need_bitmap_index = sub_column.has_bitmap_index();
+ column_options.inverted_index = nullptr;
+ if (sub_column.type() == FieldType::OLAP_FIELD_TYPE_STRUCT) {
+ if (column_options.need_bloom_filter) {
+ return Status::NotSupported("Do not support bloom filter for struct type");
+ }
+ if (column_options.need_bitmap_index) {
+ return Status::NotSupported("Do not support bitmap index for struct type");
+ }
+ }
+ if (sub_column.type() == FieldType::OLAP_FIELD_TYPE_ARRAY) {
+ if (column_options.need_bloom_filter) {
+ return Status::NotSupported("Do not support bloom filter for array type");
+ }
+ if (column_options.need_bitmap_index) {
+ return Status::NotSupported("Do not support bitmap index for array type");
+ }
+ }
+ std::unique_ptr<ColumnWriter> sub_column_writer;
+ RETURN_IF_ERROR(ColumnWriter::create(column_options, &sub_column, file_writer,
+ &sub_column_writer));
+ sub_column_writers.push_back(std::move(sub_column_writer));
+ }
+
+ // if nullable, create null writer
+ ScalarColumnWriter* null_writer = nullptr;
+ if (opts.meta->is_nullable()) {
+ FieldType null_type = FieldType::OLAP_FIELD_TYPE_TINYINT;
+ ColumnWriterOptions null_options;
+ null_options.meta = opts.meta->add_children_columns();
+ null_options.meta->set_column_id(column->get_subtype_count() + 1);
+ null_options.meta->set_unique_id(column->get_subtype_count() + 1);
+ null_options.meta->set_type(null_type);
+ null_options.meta->set_is_nullable(false);
+ null_options.meta->set_length(
+ get_scalar_type_info<OLAP_FIELD_TYPE_TINYINT>()->size());
+ null_options.meta->set_encoding(DEFAULT_ENCODING);
+ null_options.meta->set_compression(opts.meta->compression());
+
+ null_options.need_zone_map = false;
+ null_options.need_bloom_filter = false;
+ null_options.need_bitmap_index = false;
+
+ TabletColumn null_column = TabletColumn(
+ OLAP_FIELD_AGGREGATION_NONE, null_type, null_options.meta->is_nullable(),
+ null_options.meta->unique_id(), null_options.meta->length());
+ null_column.set_name("nullable");
+ null_column.set_index_length(-1); // no short key index
+ std::unique_ptr<Field> null_field(FieldFactory::create(null_column));
+ null_writer =
+ new ScalarColumnWriter(null_options, std::move(null_field), file_writer);
+ }
+
+ std::unique_ptr<ColumnWriter> writer_local =
+ std::unique_ptr<ColumnWriter>(new StructColumnWriter(
+ opts, std::move(field), null_writer, sub_column_writers));
+ *writer = std::move(writer_local);
+ return Status::OK();
+ }
case FieldType::OLAP_FIELD_TYPE_ARRAY: {
DCHECK(column->get_subtype_count() == 1);
const TabletColumn& item_column = column->get_sub_column(0);
@@ -542,6 +614,108 @@ Status ScalarColumnWriter::finish_current_page() {
////////////////////////////////////////////////////////////////////////////////
+StructColumnWriter::StructColumnWriter(
+ const ColumnWriterOptions& opts, std::unique_ptr<Field> field,
+ ScalarColumnWriter* null_writer,
+ std::vector<std::unique_ptr<ColumnWriter>>& sub_column_writers)
+ : ColumnWriter(std::move(field), opts.meta->is_nullable()), _opts(opts) {
+ for (auto& sub_column_writer : sub_column_writers) {
+ _sub_column_writers.push_back(std::move(sub_column_writer));
+ }
+ _num_sub_column_writers = _sub_column_writers.size();
+ DCHECK(_num_sub_column_writers >= 1);
+ if (is_nullable()) {
+ _null_writer.reset(null_writer);
+ }
+}
+
+Status StructColumnWriter::init() {
+ for (auto& column_writer : _sub_column_writers) {
+ RETURN_IF_ERROR(column_writer->init());
+ }
+ if (is_nullable()) {
+ RETURN_IF_ERROR(_null_writer->init());
+ }
+ return Status::OK();
+}
+
+Status StructColumnWriter::write_inverted_index() {
+ if (_opts.inverted_index) {
+ for (auto& column_writer : _sub_column_writers) {
+ RETURN_IF_ERROR(column_writer->write_inverted_index());
+ }
+ }
+ return Status::OK();
+}
+
+Status StructColumnWriter::append_nullable(const uint8_t* null_map, const uint8_t** ptr,
+ size_t num_rows) {
+ RETURN_IF_ERROR(append_data(ptr, num_rows));
+ RETURN_IF_ERROR(_null_writer->append_data(&null_map, num_rows));
+ return Status::OK();
+}
+
+Status StructColumnWriter::append_data(const uint8_t** ptr, size_t num_rows) {
+ auto data_cursor = reinterpret_cast<const void**>(ptr);
+ auto null_map_cursor = data_cursor + _num_sub_column_writers;
+ for (auto& column_writer : _sub_column_writers) {
+ RETURN_IF_ERROR(column_writer->append(reinterpret_cast<const uint8_t*>(*null_map_cursor),
+ *data_cursor, num_rows));
+ data_cursor++;
+ null_map_cursor++;
+ }
+ return Status::OK();
+}
+
+uint64_t StructColumnWriter::estimate_buffer_size() {
+ uint64_t size = 0;
+ for (auto& column_writer : _sub_column_writers) {
+ size += column_writer->estimate_buffer_size();
+ }
+ size += is_nullable() ? _null_writer->estimate_buffer_size() : 0;
+ return size;
+}
+
+Status StructColumnWriter::finish() {
+ for (auto& column_writer : _sub_column_writers) {
+ RETURN_IF_ERROR(column_writer->finish());
+ }
+ if (is_nullable()) {
+ RETURN_IF_ERROR(_null_writer->finish());
+ }
+ return Status::OK();
+}
+
+Status StructColumnWriter::write_data() {
+ for (auto& column_writer : _sub_column_writers) {
+ RETURN_IF_ERROR(column_writer->write_data());
+ }
+ if (is_nullable()) {
+ RETURN_IF_ERROR(_null_writer->write_data());
+ }
+ return Status::OK();
+}
+
+Status StructColumnWriter::write_ordinal_index() {
+ for (auto& column_writer : _sub_column_writers) {
+ RETURN_IF_ERROR(column_writer->write_ordinal_index());
+ }
+ if (is_nullable()) {
+ RETURN_IF_ERROR(_null_writer->write_ordinal_index());
+ }
+ return Status::OK();
+}
+
+Status StructColumnWriter::append_nulls(size_t num_rows) {
+ return Status::NotSupported("struct writer not support append nulls");
+}
+
+Status StructColumnWriter::finish_current_page() {
+ return Status::NotSupported("struct writer has no data, can not finish_current_page");
+}
+
+////////////////////////////////////////////////////////////////////////////////
+
ArrayColumnWriter::ArrayColumnWriter(const ColumnWriterOptions& opts, std::unique_ptr<Field> field,
ScalarColumnWriter* offset_writer,
ScalarColumnWriter* null_writer,
diff --git a/be/src/olap/rowset/segment_v2/column_writer.h b/be/src/olap/rowset/segment_v2/column_writer.h
index 5ea7ae654c..007a69a6d9 100644
--- a/be/src/olap/rowset/segment_v2/column_writer.h
+++ b/be/src/olap/rowset/segment_v2/column_writer.h
@@ -265,6 +265,57 @@ private:
FlushPageCallback* _new_page_callback = nullptr;
};
+class StructColumnWriter final : public ColumnWriter {
+public:
+ explicit StructColumnWriter(const ColumnWriterOptions& opts, std::unique_ptr<Field> field,
+ ScalarColumnWriter* null_writer,
+ std::vector<std::unique_ptr<ColumnWriter>>& sub_column_writers);
+ ~StructColumnWriter() override = default;
+
+ Status init() override;
+
+ Status append_nullable(const uint8_t* null_map, const uint8_t** data, size_t num_rows);
+ Status append_data(const uint8_t** ptr, size_t num_rows) override;
+
+ uint64_t estimate_buffer_size() override;
+
+ Status finish() override;
+ Status write_data() override;
+ Status write_ordinal_index() override;
+ Status append_nulls(size_t num_rows) override;
+
+ Status finish_current_page() override;
+
+ Status write_zone_map() override {
+ if (_opts.need_zone_map) {
+ return Status::NotSupported("struct not support zone map");
+ }
+ return Status::OK();
+ }
+
+ Status write_bitmap_index() override {
+ if (_opts.need_bitmap_index) {
+ return Status::NotSupported("struct not support bitmap index");
+ }
+ return Status::OK();
+ }
+ Status write_inverted_index() override;
+ Status write_bloom_filter_index() override {
+ if (_opts.need_bloom_filter) {
+ return Status::NotSupported("struct not support bloom filter index");
+ }
+ return Status::OK();
+ }
+
+ ordinal_t get_next_rowid() const override { return _sub_column_writers[0]->get_next_rowid(); }
+
+private:
+ size_t _num_sub_column_writers;
+ std::unique_ptr<ScalarColumnWriter> _null_writer;
+ std::vector<std::unique_ptr<ColumnWriter>> _sub_column_writers;
+ ColumnWriterOptions _opts;
+};
+
class ArrayColumnWriter final : public ColumnWriter, public FlushPageCallback {
public:
explicit ArrayColumnWriter(const ColumnWriterOptions& opts, std::unique_ptr<Field> field,
diff --git a/be/src/olap/rowset/segment_v2/segment_writer.cpp b/be/src/olap/rowset/segment_v2/segment_writer.cpp
index df7f26c34f..28ef07d40d 100644
--- a/be/src/olap/rowset/segment_v2/segment_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_writer.cpp
@@ -135,6 +135,15 @@ Status SegmentWriter::init(const std::vector<uint32_t>& col_ids, bool has_key) {
break;
}
}
+ if (column.type() == FieldType::OLAP_FIELD_TYPE_STRUCT) {
+ opts.need_zone_map = false;
+ if (opts.need_bloom_filter) {
+ return Status::NotSupported("Do not support bloom filter for struct type");
+ }
+ if (opts.need_bitmap_index) {
+ return Status::NotSupported("Do not support bitmap index for struct type");
+ }
+ }
if (column.type() == FieldType::OLAP_FIELD_TYPE_ARRAY) {
opts.need_zone_map = false;
if (opts.need_bloom_filter) {
diff --git a/be/src/olap/tablet_meta.cpp b/be/src/olap/tablet_meta.cpp
index e3d4c6f3ff..24b5460549 100644
--- a/be/src/olap/tablet_meta.cpp
+++ b/be/src/olap/tablet_meta.cpp
@@ -295,8 +295,12 @@ void TabletMeta::init_column_from_tcolumn(uint32_t unique_id, const TColumn& tco
if (tcolumn.__isset.is_bloom_filter_column) {
column->set_is_bf_column(tcolumn.is_bloom_filter_column);
}
-
- if (tcolumn.column_type.type == TPrimitiveType::ARRAY) {
+ if (tcolumn.column_type.type == TPrimitiveType::STRUCT) {
+ for (size_t i = 0; i < tcolumn.children_column.size(); i++) {
+ ColumnPB* children_column = column->add_children_columns();
+ init_column_from_tcolumn(i, tcolumn.children_column[i], children_column);
+ }
+ } else if (tcolumn.column_type.type == TPrimitiveType::ARRAY) {
ColumnPB* children_column = column->add_children_columns();
init_column_from_tcolumn(0, tcolumn.children_column[0], children_column);
}
diff --git a/be/src/olap/tablet_schema.cpp b/be/src/olap/tablet_schema.cpp
index 55f030cbb9..9fe92eb6b4 100644
--- a/be/src/olap/tablet_schema.cpp
+++ b/be/src/olap/tablet_schema.cpp
@@ -310,6 +310,10 @@ uint32_t TabletColumn::get_field_length_by_type(TPrimitiveType::type type, uint3
return string_length + sizeof(OLAP_STRING_MAX_LENGTH);
case TPrimitiveType::JSONB:
return string_length + sizeof(OLAP_JSONB_MAX_LENGTH);
+ case TPrimitiveType::STRUCT:
+ // Note that(xy): this is the length of struct type itself,
+ // the length of its subtypes are not included.
+ return OLAP_STRUCT_MAX_LENGTH;
case TPrimitiveType::ARRAY:
return OLAP_ARRAY_MAX_LENGTH;
case TPrimitiveType::DECIMAL32:
@@ -402,8 +406,13 @@ void TabletColumn::init_from_pb(const ColumnPB& column) {
if (column.has_visible()) {
_visible = column.visible();
}
-
- if (_type == FieldType::OLAP_FIELD_TYPE_ARRAY) {
+ if (_type == FieldType::OLAP_FIELD_TYPE_STRUCT) {
+ for (size_t i = 0; i < column.children_columns_size(); i++) {
+ TabletColumn child_column;
+ child_column.init_from_pb(column.children_columns(i));
+ add_sub_column(child_column);
+ }
+ } else if (_type == FieldType::OLAP_FIELD_TYPE_ARRAY) {
DCHECK(column.children_columns_size() == 1) << "ARRAY type has more than 1 children types.";
TabletColumn child_column;
child_column.init_from_pb(column.children_columns(0));
@@ -435,7 +444,12 @@ void TabletColumn::to_schema_pb(ColumnPB* column) const {
}
column->set_visible(_visible);
- if (_type == OLAP_FIELD_TYPE_ARRAY) {
+ if (_type == OLAP_FIELD_TYPE_STRUCT) {
+ for (size_t i = 0; i < _sub_columns.size(); i++) {
+ ColumnPB* child = column->add_children_columns();
+ _sub_columns[i].to_schema_pb(child);
+ }
+ } else if (_type == OLAP_FIELD_TYPE_ARRAY) {
DCHECK(_sub_columns.size() == 1) << "ARRAY type has more than 1 children types.";
ColumnPB* child = column->add_children_columns();
_sub_columns[0].to_schema_pb(child);
diff --git a/be/src/olap/types.cpp b/be/src/olap/types.cpp
index aa2226a84a..916f7768af 100644
--- a/be/src/olap/types.cpp
+++ b/be/src/olap/types.cpp
@@ -166,10 +166,33 @@ const TypeInfo* get_array_type_info(FieldType leaf_type, int32_t iterations) {
return array_type_Info_arr[leaf_type][iterations];
}
+// Produce a struct type info
+// TODO(xy): Need refactor to this produce method
+const TypeInfo* get_struct_type_info(std::vector<FieldType> field_types) {
+ std::vector<TypeInfoPtr> type_infos;
+ type_infos.reserve(field_types.size());
+ for (FieldType& type : field_types) {
+ if (is_scalar_type(type)) {
+ type_infos.push_back(create_static_type_info_ptr(get_scalar_type_info(type)));
+ } else {
+ // TODO(xy): Not supported nested complex type now
+ }
+ }
+ return new StructTypeInfo(type_infos);
+}
+
// TODO: Support the type info of the nested array with more than 9 depths.
+// TODO(xy): Support the type info of the nested struct
TypeInfoPtr get_type_info(segment_v2::ColumnMetaPB* column_meta_pb) {
FieldType type = (FieldType)column_meta_pb->type();
- if (UNLIKELY(type == OLAP_FIELD_TYPE_ARRAY)) {
+ if (UNLIKELY(type == OLAP_FIELD_TYPE_STRUCT)) {
+ std::vector<FieldType> field_types;
+ for (uint32_t i = 0; i < column_meta_pb->children_columns_size(); i++) {
+ const auto* child_column = &column_meta_pb->children_columns(i);
+ field_types.push_back((FieldType)child_column->type());
+ }
+ return create_dynamic_type_info_ptr(get_struct_type_info(field_types));
+ } else if (UNLIKELY(type == OLAP_FIELD_TYPE_ARRAY)) {
int32_t iterations = 0;
const auto* child_column = &column_meta_pb->children_columns(0);
while (child_column->type() == OLAP_FIELD_TYPE_ARRAY) {
@@ -202,7 +225,14 @@ TypeInfoPtr create_type_info_ptr(const TypeInfo* type_info, bool should_reclaim_
// TODO: Support the type info of the nested array with more than 9 depths.
TypeInfoPtr get_type_info(const TabletColumn* col) {
auto type = col->type();
- if (UNLIKELY(type == OLAP_FIELD_TYPE_ARRAY)) {
+ if (UNLIKELY(type == OLAP_FIELD_TYPE_STRUCT)) {
+ std::vector<FieldType> field_types;
+ for (uint32_t i = 0; i < col->get_subtype_count(); i++) {
+ const auto* child_column = &col->get_sub_column(i);
+ field_types.push_back(child_column->type());
+ }
+ return create_dynamic_type_info_ptr(get_struct_type_info(field_types));
+ } else if (UNLIKELY(type == OLAP_FIELD_TYPE_ARRAY)) {
int32_t iterations = 0;
const auto* child_column = &col->get_sub_column(0);
while (child_column->type() == OLAP_FIELD_TYPE_ARRAY) {
@@ -219,9 +249,21 @@ TypeInfoPtr clone_type_info(const TypeInfo* type_info) {
if (is_scalar_type(type_info->type())) {
return create_static_type_info_ptr(type_info);
} else {
- const auto array_type_info = dynamic_cast<const ArrayTypeInfo*>(type_info);
- return create_dynamic_type_info_ptr(
- new ArrayTypeInfo(clone_type_info(array_type_info->item_type_info())));
+ auto type = type_info->type();
+ if (type == OLAP_FIELD_TYPE_STRUCT) {
+ const auto struct_type_info = dynamic_cast<const StructTypeInfo*>(type_info);
+ std::vector<TypeInfoPtr> clone_type_infos;
+ const std::vector<TypeInfoPtr>* sub_type_infos = struct_type_info->type_infos();
+ clone_type_infos.reserve(sub_type_infos->size());
+ for (size_t i = 0; i < sub_type_infos->size(); i++) {
+ clone_type_infos.push_back(clone_type_info((*sub_type_infos)[i].get()));
+ }
+ return create_dynamic_type_info_ptr(new StructTypeInfo(clone_type_infos));
+ } else {
+ const auto array_type_info = dynamic_cast<const ArrayTypeInfo*>(type_info);
+ return create_dynamic_type_info_ptr(
+ new ArrayTypeInfo(clone_type_info(array_type_info->item_type_info())));
+ }
}
}
diff --git a/be/src/olap/types.h b/be/src/olap/types.h
index c580f2ea53..a8f21b66ae 100644
--- a/be/src/olap/types.h
+++ b/be/src/olap/types.h
@@ -32,6 +32,7 @@
#include "runtime/collection_value.h"
#include "runtime/jsonb_value.h"
#include "runtime/mem_pool.h"
+#include "runtime/struct_value.h"
#include "util/jsonb_document.h"
#include "util/jsonb_utils.h"
#include "util/mem_util.hpp"
@@ -431,6 +432,245 @@ private:
const size_t _item_size;
};
+class StructTypeInfo : public TypeInfo {
+public:
+ explicit StructTypeInfo(std::vector<TypeInfoPtr>& type_infos) {
+ for (TypeInfoPtr& type_info : type_infos) {
+ _type_infos.push_back(std::move(type_info));
+ }
+ }
+ ~StructTypeInfo() override = default;
+
+ bool equal(const void* left, const void* right) const override {
+ auto l_value = reinterpret_cast<const StructValue*>(left);
+ auto r_value = reinterpret_cast<const StructValue*>(right);
+ if (l_value->size() != r_value->size()) {
+ return false;
+ }
+ uint32_t size = l_value->size();
+
+ if (!l_value->has_null() && !r_value->has_null()) {
+ for (size_t i = 0; i < size; ++i) {
+ if (!_type_infos[i]->equal(l_value->child_value(i), r_value->child_value(i))) {
+ return false;
+ }
+ }
+ } else {
+ for (size_t i = 0; i < size; ++i) {
+ if (l_value->is_null_at(i)) {
+ if (r_value->is_null_at(i)) { // both are null
+ continue;
+ } else { // left is null & right is not null
+ return false;
+ }
+ } else if (r_value->is_null_at(i)) { // left is not null & right is null
+ return false;
+ }
+ if (!_type_infos[i]->equal(l_value->child_value(i), r_value->child_value(i))) {
+ return false;
+ }
+ }
+ }
+ return true;
+ }
+
+ int cmp(const void* left, const void* right) const override {
+ auto l_value = reinterpret_cast<const StructValue*>(left);
+ auto r_value = reinterpret_cast<const StructValue*>(right);
+ uint32_t l_size = l_value->size();
+ uint32_t r_size = r_value->size();
+ size_t cur = 0;
+
+ if (!l_value->has_null() && !r_value->has_null()) {
+ while (cur < l_size && cur < r_size) {
+ int result =
+ _type_infos[cur]->cmp(l_value->child_value(cur), r_value->child_value(cur));
+ if (result != 0) {
+ return result;
+ }
+ ++cur;
+ }
+ } else {
+ while (cur < l_size && cur < r_size) {
+ if (l_value->is_null_at(cur)) {
+ if (!r_value->is_null_at(cur)) { // left is null & right is not null
+ return -1;
+ }
+ } else if (r_value->is_null_at(cur)) { // left is not null & right is null
+ return 1;
+ } else { // both are not null
+ int result = _type_infos[cur]->cmp(l_value->child_value(cur),
+ r_value->child_value(cur));
+ if (result != 0) {
+ return result;
+ }
+ }
+ ++cur;
+ }
+ }
+
+ if (l_size < r_size) {
+ return -1;
+ } else if (l_size > r_size) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+
+ void shallow_copy(void* dest, const void* src) const override {
+ auto dest_value = reinterpret_cast<StructValue*>(dest);
+ auto src_value = reinterpret_cast<const StructValue*>(src);
+ dest_value->shallow_copy(src_value);
+ }
+
+ void deep_copy(void* dest, const void* src, MemPool* mem_pool) const override {
+ auto dest_value = reinterpret_cast<StructValue*>(dest);
+ auto src_value = reinterpret_cast<const StructValue*>(src);
+
+ if (src_value->size() == 0) {
+ new (dest_value) StructValue(src_value->size());
+ return;
+ }
+
+ dest_value->set_size(src_value->size());
+ dest_value->set_has_null(src_value->has_null());
+
+ size_t allocate_size = src_value->size() * sizeof(*src_value->values());
+ // allocate memory for children value
+ for (size_t i = 0; i < src_value->size(); ++i) {
+ if (src_value->is_null_at(i)) continue;
+ allocate_size += _type_infos[i]->size();
+ }
+
+ dest_value->set_values((void**)mem_pool->allocate(allocate_size));
+ auto ptr = reinterpret_cast<uint8_t*>(dest_value->mutable_values());
+ ptr += dest_value->size() * sizeof(*dest_value->values());
+
+ for (size_t i = 0; i < src_value->size(); ++i) {
+ dest_value->set_child_value(nullptr, i);
+ if (src_value->is_null_at(i)) continue;
+ dest_value->set_child_value(ptr, i);
+ ptr += _type_infos[i]->size();
+ }
+
+ // copy children value
+ for (size_t i = 0; i < src_value->size(); ++i) {
+ if (src_value->is_null_at(i)) continue;
+ _type_infos[i]->deep_copy(dest_value->mutable_child_value(i), src_value->child_value(i),
+ mem_pool);
+ }
+ }
+
+ void copy_object(void* dest, const void* src, MemPool* mem_pool) const override {
+ deep_copy(dest, src, mem_pool);
+ }
+
+ void direct_copy(void* dest, const void* src) const override {
+ auto dest_value = static_cast<StructValue*>(dest);
+ auto base = reinterpret_cast<uint8_t*>(dest_value->mutable_values());
+ direct_copy(&base, dest, src);
+ }
+
+ void direct_copy(uint8_t** base, void* dest, const void* src) const {
+ auto dest_value = static_cast<StructValue*>(dest);
+ auto src_value = static_cast<const StructValue*>(src);
+
+ dest_value->set_size(src_value->size());
+ dest_value->set_has_null(src_value->has_null());
+ *base += src_value->size() * sizeof(*src_value->values());
+
+ for (size_t i = 0; i < src_value->size(); ++i) {
+ dest_value->set_child_value(nullptr, i);
+ if (src_value->is_null_at(i)) continue;
+ dest_value->set_child_value(*base, i);
+ *base += _type_infos[i]->size();
+ }
+
+ for (size_t i = 0; i < src_value->size(); ++i) {
+ if (dest_value->is_null_at(i)) {
+ continue;
+ }
+ auto dest_address = dest_value->mutable_child_value(i);
+ auto src_address = src_value->child_value(i);
+ if (_type_infos[i]->type() == OLAP_FIELD_TYPE_STRUCT) {
+ dynamic_cast<const StructTypeInfo*>(_type_infos[i].get())
+ ->direct_copy(base, dest_address, src_address);
+ } else if (_type_infos[i]->type() == OLAP_FIELD_TYPE_ARRAY) {
+ dynamic_cast<const ArrayTypeInfo*>(_type_infos[i].get())
+ ->direct_copy(base, dest_address, src_address);
+ } else {
+ if (is_olap_string_type(_type_infos[i]->type())) {
+ auto dest_slice = reinterpret_cast<Slice*>(dest_address);
+ auto src_slice = reinterpret_cast<const Slice*>(src_address);
+ dest_slice->data = reinterpret_cast<char*>(*base);
+ dest_slice->size = src_slice->size;
+ *base += src_slice->size;
+ }
+ _type_infos[i]->direct_copy(dest_address, src_address);
+ }
+ }
+ }
+
+ void direct_copy_may_cut(void* dest, const void* src) const override { direct_copy(dest, src); }
+
+ Status convert_from(void* dest, const void* src, const TypeInfo* src_type, MemPool* mem_pool,
+ size_t variable_len = 0) const override {
+ return Status::Error<ErrorCode::NOT_IMPLEMENTED_ERROR>();
+ }
+
+ Status from_string(void* buf, const std::string& scan_key, const int precision = 0,
+ const int scale = 0) const override {
+ return Status::Error<ErrorCode::NOT_IMPLEMENTED_ERROR>();
+ }
+
+ std::string to_string(const void* src) const override {
+ auto src_value = reinterpret_cast<const StructValue*>(src);
+ std::string result = "{";
+
+ for (size_t i = 0; i < src_value->size(); ++i) {
+ std::string field_value = _type_infos[i]->to_string(src_value->child_value(i));
+ result += field_value;
+ if (i < src_value->size() - 1) {
+ result += ", ";
+ }
+ }
+ result += "}";
+ return result;
+ }
+
+ void set_to_max(void* buf) const override {
+ DCHECK(false) << "set_to_max of list is not implemented.";
+ }
+
+ void set_to_min(void* buf) const override {
+ DCHECK(false) << "set_to_min of list is not implemented.";
+ }
+
+ uint32_t hash_code(const void* data, uint32_t seed) const override {
+ auto struct_value = reinterpret_cast<const StructValue*>(data);
+ auto size = struct_value->size();
+ uint32_t result = HashUtil::hash(&size, sizeof(size), seed);
+ for (size_t i = 0; i < size; ++i) {
+ if (struct_value->is_null_at(i)) {
+ result = seed * result;
+ } else {
+ result = seed * result + _type_infos[i]->hash_code(struct_value->values()[i], seed);
+ }
+ }
+ return result;
+ }
+
+ const size_t size() const override { return sizeof(StructValue); }
+
+ FieldType type() const override { return OLAP_FIELD_TYPE_STRUCT; }
+
+ inline const std::vector<TypeInfoPtr>* type_infos() const { return &_type_infos; }
+
+private:
+ std::vector<TypeInfoPtr> _type_infos;
+};
+
bool is_scalar_type(FieldType field_type);
const TypeInfo* get_scalar_type_info(FieldType field_type);
@@ -566,12 +806,15 @@ template <>
struct CppTypeTraits<OLAP_FIELD_TYPE_OBJECT> {
using CppType = Slice;
};
-
template <>
struct CppTypeTraits<OLAP_FIELD_TYPE_QUANTILE_STATE> {
using CppType = Slice;
};
template <>
+struct CppTypeTraits<OLAP_FIELD_TYPE_STRUCT> {
+ using CppType = StructValue;
+};
+template <>
struct CppTypeTraits<OLAP_FIELD_TYPE_ARRAY> {
using CppType = CollectionValue;
};
diff --git a/be/src/runtime/CMakeLists.txt b/be/src/runtime/CMakeLists.txt
index dfc375bd24..92ffc2071d 100644
--- a/be/src/runtime/CMakeLists.txt
+++ b/be/src/runtime/CMakeLists.txt
@@ -48,6 +48,7 @@ set(RUNTIME_FILES
threadlocal.cc
decimalv2_value.cpp
large_int_value.cpp
+ struct_value.cpp
collection_value.cpp
tuple.cpp
tuple_row.cpp
diff --git a/be/src/runtime/primitive_type.cpp b/be/src/runtime/primitive_type.cpp
index 779676a45d..27bc9f5b6e 100644
--- a/be/src/runtime/primitive_type.cpp
+++ b/be/src/runtime/primitive_type.cpp
@@ -22,6 +22,7 @@
#include "runtime/define_primitive_type.h"
#include "runtime/jsonb_value.h"
#include "runtime/string_value.h"
+#include "runtime/struct_value.h"
namespace doris {
@@ -53,6 +54,8 @@ PrimitiveType convert_type_to_primitive(FunctionContext::Type type) {
return PrimitiveType::TYPE_BOOLEAN;
case FunctionContext::Type::TYPE_ARRAY:
return PrimitiveType::TYPE_ARRAY;
+ case FunctionContext::Type::TYPE_STRUCT:
+ return PrimitiveType::TYPE_STRUCT;
case FunctionContext::Type::TYPE_OBJECT:
return PrimitiveType::TYPE_OBJECT;
case FunctionContext::Type::TYPE_HLL:
@@ -95,6 +98,7 @@ int get_byte_size(PrimitiveType type) {
case TYPE_HLL:
case TYPE_QUANTILE_STATE:
case TYPE_ARRAY:
+ case TYPE_STRUCT:
case TYPE_MAP:
return 0;
@@ -262,6 +266,9 @@ PrimitiveType thrift_to_type(TPrimitiveType::type ttype) {
case TPrimitiveType::ARRAY:
return TYPE_ARRAY;
+ case TPrimitiveType::STRUCT:
+ return TYPE_STRUCT;
+
default:
return INVALID_TYPE;
}
@@ -356,6 +363,9 @@ TPrimitiveType::type to_thrift(PrimitiveType ptype) {
case TYPE_ARRAY:
return TPrimitiveType::ARRAY;
+ case TYPE_STRUCT:
+ return TPrimitiveType::STRUCT;
+
default:
return TPrimitiveType::INVALID_TYPE;
}
@@ -450,6 +460,9 @@ std::string type_to_string(PrimitiveType t) {
case TYPE_ARRAY:
return "ARRAY";
+ case TYPE_STRUCT:
+ return "STRUCT";
+
default:
return "";
};
@@ -589,6 +602,8 @@ int get_slot_size(PrimitiveType type) {
return sizeof(JsonBinaryValue);
case TYPE_ARRAY:
return sizeof(CollectionValue);
+ case TYPE_STRUCT:
+ return sizeof(StructValue);
case TYPE_NULL:
case TYPE_BOOLEAN:
diff --git a/be/src/runtime/primitive_type.h b/be/src/runtime/primitive_type.h
index 5af7639290..f84f9a3348 100644
--- a/be/src/runtime/primitive_type.h
+++ b/be/src/runtime/primitive_type.h
@@ -54,6 +54,7 @@ constexpr bool is_enumeration_type(PrimitiveType type) {
case TYPE_DECIMAL128I:
case TYPE_BOOLEAN:
case TYPE_ARRAY:
+ case TYPE_STRUCT:
case TYPE_HLL:
return false;
case TYPE_TINYINT:
diff --git a/be/src/runtime/struct_value.cpp b/be/src/runtime/struct_value.cpp
new file mode 100644
index 0000000000..ff0b9cc62a
--- /dev/null
+++ b/be/src/runtime/struct_value.cpp
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/struct_value.h"
+
+namespace doris {
+
+void StructValue::shallow_copy(const StructValue* other) {
+ _size = other->_size;
+ _values = other->_values;
+ _has_null = other->_has_null;
+}
+} // namespace doris
\ No newline at end of file
diff --git a/be/src/runtime/struct_value.h b/be/src/runtime/struct_value.h
new file mode 100644
index 0000000000..ec243d729c
--- /dev/null
+++ b/be/src/runtime/struct_value.h
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <type_traits>
+
+#include "runtime/primitive_type.h"
+
+namespace doris {
+
+class StructValue {
+public:
+ StructValue() = default;
+
+ explicit StructValue(uint32_t size) : _values(nullptr), _size(size), _has_null(false) {}
+ StructValue(void** values, uint32_t size) : _values(values), _size(size), _has_null(false) {}
+ StructValue(void** values, uint32_t size, bool has_null)
+ : _values(values), _size(size), _has_null(has_null) {}
+
+ //void to_struct_val(StructVal* val) const;
+ //static StructValue from_struct_val(const StructVal& val);
+
+ uint32_t size() const { return _size; }
+ void set_size(uint32_t size) { _size = size; }
+ bool has_null() const { return _has_null; }
+ void set_has_null(bool has_null) { _has_null = has_null; }
+ bool is_null_at(uint32_t index) const {
+ return this->_has_null && this->_values[index] == nullptr;
+ }
+
+ void shallow_copy(const StructValue* other);
+
+ // size_t get_byte_size(const TypeDescriptor& type) const;
+
+ const void** values() const { return const_cast<const void**>(_values); }
+ void** mutable_values() { return _values; }
+ void set_values(void** values) { _values = values; }
+ const void* child_value(uint32_t index) const { return _values[index]; }
+ void* mutable_child_value(uint32_t index) { return _values[index]; }
+ void set_child_value(void* value, uint32_t index) { _values[index] = value; }
+
+private:
+ // pointer to the start of the vector of children pointers. These pointers are
+ // point to children values where a null pointer means that this child is NULL.
+ void** _values;
+ // the number of values in this struct value.
+ uint32_t _size;
+ // child has no null value if has_null is false.
+ // child may has null value if has_null is true.
+ bool _has_null;
+};
+} // namespace doris
\ No newline at end of file
diff --git a/be/src/runtime/types.cpp b/be/src/runtime/types.cpp
index 37f4933569..f26b1dcbce 100644
--- a/be/src/runtime/types.cpp
+++ b/be/src/runtime/types.cpp
@@ -55,14 +55,31 @@ TypeDescriptor::TypeDescriptor(const std::vector<TTypeNode>& types, int* idx)
case TTypeNodeType::ARRAY: {
DCHECK(!node.__isset.scalar_type);
DCHECK_LT(*idx, types.size() - 1);
+ // contains_null should be always set in ArrayType
+ DCHECK(node.__isset.contains_null);
type = TYPE_ARRAY;
- if (node.__isset.contains_null) {
- contains_null = node.contains_null;
- }
+ contains_nulls.reserve(1);
+ contains_nulls.push_back(node.contains_null);
++(*idx);
children.push_back(TypeDescriptor(types, idx));
break;
}
+ case TTypeNodeType::STRUCT: {
+ DCHECK(!node.__isset.scalar_type);
+ DCHECK_LT(*idx, types.size() - 1);
+ DCHECK(!node.__isset.contains_null);
+ DCHECK(node.__isset.struct_fields);
+ DCHECK_GE(node.struct_fields.size(), 1);
+ type = TYPE_STRUCT;
+ contains_nulls.reserve(node.struct_fields.size());
+ for (size_t i = 0; i < node.struct_fields.size(); i++) {
+ ++(*idx);
+ children.push_back(TypeDescriptor(types, idx));
+ field_names.push_back(node.struct_fields[i].name);
+ contains_nulls.push_back(node.struct_fields[i].contains_null);
+ }
+ break;
+ }
// case TTypeNodeType::STRUCT:
// type = TYPE_STRUCT;
// for (int i = 0; i < node.struct_fields.size(); ++i) {
@@ -98,15 +115,17 @@ void TypeDescriptor::to_thrift(TTypeDesc* thrift_type) const {
if (is_complex_type()) {
if (type == TYPE_ARRAY) {
node.type = TTypeNodeType::ARRAY;
+ node.contains_null = contains_nulls[0];
} else if (type == TYPE_MAP) {
node.type = TTypeNodeType::MAP;
} else {
DCHECK_EQ(type, TYPE_STRUCT);
node.type = TTypeNodeType::STRUCT;
node.__set_struct_fields(std::vector<TStructField>());
- for (auto& field_name : field_names) {
+ for (size_t i = 0; i < field_names.size(); i++) {
node.struct_fields.push_back(TStructField());
- node.struct_fields.back().name = field_name;
+ node.struct_fields.back().name = field_names[i];
+ node.struct_fields.back().contains_null = contains_nulls[i];
}
}
for (const TypeDescriptor& child : children) {
@@ -147,10 +166,12 @@ void TypeDescriptor::to_protobuf(PTypeDesc* ptype) const {
scalar_type->set_scale(scale);
} else if (type == TYPE_ARRAY) {
node->set_type(TTypeNodeType::ARRAY);
+ node->set_contains_null(contains_nulls[0]);
for (const TypeDescriptor& child : children) {
child.to_protobuf(ptype);
}
}
+ // TODO(xy): support struct
}
TypeDescriptor::TypeDescriptor(const google::protobuf::RepeatedPtrField<PTypeNode>& types, int* idx)
@@ -184,13 +205,15 @@ TypeDescriptor::TypeDescriptor(const google::protobuf::RepeatedPtrField<PTypeNod
}
case TTypeNodeType::ARRAY: {
type = TYPE_ARRAY;
+ contains_nulls.push_back(true);
if (node.has_contains_null()) {
- contains_null = node.contains_null();
+ contains_nulls[0] = node.contains_null();
}
++(*idx);
children.push_back(TypeDescriptor(types, idx));
break;
}
+ // TODO(xy): support struct
default:
DCHECK(false) << node.type();
}
@@ -218,6 +241,19 @@ std::string TypeDescriptor::debug_string() const {
ss << "ARRAY<" << children[0].debug_string() << ">";
return ss.str();
}
+ case TYPE_STRUCT: {
+ ss << "STRUCT<";
+ for (size_t i = 0; i < children.size(); i++) {
+ ss << field_names[i];
+ ss << ":";
+ ss << children[i].debug_string();
+ if (i != children.size() - 1) {
+ ss << ",";
+ }
+ }
+ ss << ">";
+ return ss.str();
+ }
default:
return type_to_string(type);
}
diff --git a/be/src/runtime/types.h b/be/src/runtime/types.h
index aca6336f6b..dec60926dd 100644
--- a/be/src/runtime/types.h
+++ b/be/src/runtime/types.h
@@ -64,7 +64,7 @@ struct TypeDescriptor {
std::vector<std::string> field_names;
// Used for complex types only.
- bool contains_null = true;
+ std::vector<bool> contains_nulls;
TypeDescriptor() : type(INVALID_TYPE), len(-1), precision(-1), scale(-1) {}
diff --git a/be/src/udf/udf.h b/be/src/udf/udf.h
index 49758f40d1..ec60fae072 100644
--- a/be/src/udf/udf.h
+++ b/be/src/udf/udf.h
@@ -89,6 +89,7 @@ public:
TYPE_DECIMALV2,
TYPE_OBJECT,
TYPE_ARRAY,
+ TYPE_STRUCT,
TYPE_QUANTILE_STATE,
TYPE_DATEV2,
TYPE_DATETIMEV2,
diff --git a/be/src/vec/CMakeLists.txt b/be/src/vec/CMakeLists.txt
index be5b60c6dd..7a569ded2e 100644
--- a/be/src/vec/CMakeLists.txt
+++ b/be/src/vec/CMakeLists.txt
@@ -136,6 +136,7 @@ set(VEC_FILES
exprs/vexpr_context.cpp
exprs/vliteral.cpp
exprs/varray_literal.cpp
+ exprs/vstruct_literal.cpp
exprs/vin_predicate.cpp
exprs/vbloom_predicate.cpp
exprs/vbitmap_predicate.cpp
diff --git a/be/src/vec/columns/column_struct.cpp b/be/src/vec/columns/column_struct.cpp
index 05d5df0fd4..c7e5c23d2a 100644
--- a/be/src/vec/columns/column_struct.cpp
+++ b/be/src/vec/columns/column_struct.cpp
@@ -48,8 +48,7 @@ ColumnStruct::ColumnStruct(MutableColumns&& mutable_columns) {
columns.reserve(mutable_columns.size());
for (auto& column : mutable_columns) {
if (is_column_const(*column)) {
- throw Exception {"ColumnStruct cannot have ColumnConst as its element",
- ErrorCodes::ILLEGAL_COLUMN};
+ LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
}
columns.push_back(std::move(column));
}
@@ -59,8 +58,7 @@ ColumnStruct::ColumnStruct(Columns&& columns) {
columns.reserve(columns.size());
for (auto& column : columns) {
if (is_column_const(*column)) {
- throw Exception {"ColumnStruct cannot have ColumnConst as its element",
- ErrorCodes::ILLEGAL_COLUMN};
+ LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
}
columns.push_back(std::move(column));
}
@@ -70,8 +68,7 @@ ColumnStruct::ColumnStruct(TupleColumns&& tuple_columns) {
columns.reserve(tuple_columns.size());
for (auto& column : tuple_columns) {
if (is_column_const(*column)) {
- throw Exception {"ColumnStruct cannot have ColumnConst as its element",
- ErrorCodes::ILLEGAL_COLUMN};
+ LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
}
columns.push_back(std::move(column));
}
@@ -79,9 +76,9 @@ ColumnStruct::ColumnStruct(TupleColumns&& tuple_columns) {
ColumnStruct::Ptr ColumnStruct::create(Columns& columns) {
for (const auto& column : columns) {
- if (is_column_const(*column))
- throw Exception {"ColumnStruct cannot have ColumnConst as its element",
- ErrorCodes::ILLEGAL_COLUMN};
+ if (is_column_const(*column)) {
+ LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
+ }
}
auto column_struct = ColumnStruct::create(columns);
return column_struct;
@@ -90,8 +87,7 @@ ColumnStruct::Ptr ColumnStruct::create(Columns& columns) {
ColumnStruct::Ptr ColumnStruct::create(TupleColumns& tuple_columns) {
for (const auto& column : tuple_columns) {
if (is_column_const(*column)) {
- throw Exception {"ColumnStruct cannot have ColumnConst as its element",
- ErrorCodes::ILLEGAL_COLUMN};
+ LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
}
}
auto column_struct = ColumnStruct::create(tuple_columns);
@@ -144,22 +140,11 @@ bool ColumnStruct::is_default_at(size_t n) const {
return true;
}
-StringRef ColumnStruct::get_data_at(size_t) const {
- throw Exception("Method get_data_at is not supported for " + get_name(),
- ErrorCodes::NOT_IMPLEMENTED);
-}
-
-void ColumnStruct::insert_data(const char*, size_t) {
- throw Exception("Method insert_data is not supported for " + get_name(),
- ErrorCodes::NOT_IMPLEMENTED);
-}
-
void ColumnStruct::insert(const Field& x) {
const auto& tuple = x.get<const Tuple&>();
const size_t tuple_size = columns.size();
if (tuple.size() != tuple_size) {
- throw Exception("Cannot insert value of different size into tuple",
- ErrorCodes::CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE);
+ LOG(FATAL) << "Cannot insert value of different size into tuple.";
}
for (size_t i = 0; i < tuple_size; ++i) {
@@ -172,8 +157,7 @@ void ColumnStruct::insert_from(const IColumn& src_, size_t n) {
const size_t tuple_size = columns.size();
if (src.columns.size() != tuple_size) {
- throw Exception("Cannot insert value of different size into tuple",
- ErrorCodes::CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE);
+ LOG(FATAL) << "Cannot insert value of different size into tuple.";
}
for (size_t i = 0; i < tuple_size; ++i) {
diff --git a/be/src/vec/columns/column_struct.h b/be/src/vec/columns/column_struct.h
index 895a2796b7..75dade874d 100644
--- a/be/src/vec/columns/column_struct.h
+++ b/be/src/vec/columns/column_struct.h
@@ -102,7 +102,7 @@ public:
std::string get_name() const override;
const char* get_family_name() const override { return "Struct"; }
TypeIndex get_data_type() const { return TypeIndex::Struct; }
-
+ bool can_be_inside_nullable() const override { return true; }
MutableColumnPtr clone_empty() const override;
MutableColumnPtr clone_resized(size_t size) const override;
@@ -112,8 +112,12 @@ public:
void get(size_t n, Field& res) const override;
bool is_default_at(size_t n) const override;
- StringRef get_data_at(size_t n) const override;
- void insert_data(const char* pos, size_t length) override;
+ [[noreturn]] StringRef get_data_at(size_t n) const override {
+ LOG(FATAL) << "Method get_data_at is not supported for " + get_name();
+ }
+ [[noreturn]] void insert_data(const char* pos, size_t length) override {
+ LOG(FATAL) << "Method insert_data is not supported for " + get_name();
+ }
void insert(const Field& x) override;
void insert_from(const IColumn& src_, size_t n) override;
void insert_default() override;
@@ -162,7 +166,10 @@ public:
// int compare_at_with_collation(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint,
// const Collator& collator) const override;
- int compare_at(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) const override;
+ [[noreturn]] int compare_at(size_t n, size_t m, const IColumn& rhs_,
+ int nan_direction_hint) const override {
+ LOG(FATAL) << "compare_at not implemented";
+ }
void get_extremes(Field& min, Field& max) const override;
// void get_permutation(IColumn::PermutationSortDirection direction,
diff --git a/be/src/vec/data_types/data_type_factory.cpp b/be/src/vec/data_types/data_type_factory.cpp
index e622d979d5..7eec1eb2fd 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -29,6 +29,18 @@ DataTypePtr DataTypeFactory::create_data_type(const doris::Field& col_desc) {
if (col_desc.type() == OLAP_FIELD_TYPE_ARRAY) {
DCHECK(col_desc.get_sub_field_count() == 1);
nested = std::make_shared<DataTypeArray>(create_data_type(*col_desc.get_sub_field(0)));
+ } else if (col_desc.type() == OLAP_FIELD_TYPE_STRUCT) {
+ DCHECK(col_desc.get_sub_field_count() >= 1);
+ size_t field_size = col_desc.get_sub_field_count();
+ DataTypes dataTypes;
+ Strings names;
+ dataTypes.reserve(field_size);
+ names.reserve(field_size);
+ for (size_t i = 0; i < field_size; i++) {
+ dataTypes.push_back(create_data_type(*col_desc.get_sub_field(i)));
+ names.push_back(col_desc.name());
+ }
+ nested = std::make_shared<DataTypeStruct>(dataTypes, names);
} else {
nested = _create_primitive_data_type(col_desc.type(), col_desc.get_precision(),
col_desc.get_scale());
@@ -45,6 +57,18 @@ DataTypePtr DataTypeFactory::create_data_type(const TabletColumn& col_desc, bool
if (col_desc.type() == OLAP_FIELD_TYPE_ARRAY) {
DCHECK(col_desc.get_subtype_count() == 1);
nested = std::make_shared<DataTypeArray>(create_data_type(col_desc.get_sub_column(0)));
+ } else if (col_desc.type() == OLAP_FIELD_TYPE_STRUCT) {
+ DCHECK(col_desc.get_subtype_count() >= 1);
+ size_t col_size = col_desc.get_subtype_count();
+ DataTypes dataTypes;
+ Strings names;
+ dataTypes.reserve(col_size);
+ names.reserve(col_size);
+ for (size_t i = 0; i < col_size; i++) {
+ dataTypes.push_back(create_data_type(col_desc.get_sub_column(i)));
+ names.push_back(col_desc.name());
+ }
+ nested = std::make_shared<DataTypeStruct>(dataTypes, names);
} else {
nested =
_create_primitive_data_type(col_desc.type(), col_desc.precision(), col_desc.frac());
@@ -129,8 +153,23 @@ DataTypePtr DataTypeFactory::create_data_type(const TypeDescriptor& col_desc, bo
case TYPE_ARRAY:
DCHECK(col_desc.children.size() == 1);
nested = std::make_shared<vectorized::DataTypeArray>(
- create_data_type(col_desc.children[0], col_desc.contains_null));
+ create_data_type(col_desc.children[0], col_desc.contains_nulls[0]));
+ break;
+ case TYPE_STRUCT: {
+ DCHECK(col_desc.children.size() >= 1);
+ size_t child_size = col_desc.children.size();
+ DCHECK_EQ(col_desc.field_names.size(), child_size);
+ DataTypes dataTypes;
+ Strings names;
+ dataTypes.reserve(child_size);
+ names.reserve(child_size);
+ for (size_t i = 0; i < child_size; i++) {
+ dataTypes.push_back(create_data_type(col_desc.children[i], col_desc.contains_nulls[i]));
+ names.push_back(col_desc.field_names[i]);
+ }
+ nested = std::make_shared<DataTypeStruct>(dataTypes, names);
break;
+ }
case INVALID_TYPE:
default:
DCHECK(false) << "invalid PrimitiveType:" << (int)col_desc.type;
diff --git a/be/src/vec/data_types/data_type_factory.hpp b/be/src/vec/data_types/data_type_factory.hpp
index ed270b40ea..d46dd3435d 100644
--- a/be/src/vec/data_types/data_type_factory.hpp
+++ b/be/src/vec/data_types/data_type_factory.hpp
@@ -41,6 +41,7 @@
#include "vec/data_types/data_type_nullable.h"
#include "vec/data_types/data_type_number.h"
#include "vec/data_types/data_type_string.h"
+#include "vec/data_types/data_type_struct.h"
namespace doris::vectorized {
@@ -95,7 +96,10 @@ public:
});
return instance;
}
+
+ // TODO(xy): support creator to create dynamic struct type
DataTypePtr get(const std::string& name) { return _data_type_map[name]; }
+ // TODO(xy): support creator to create dynamic struct type
const std::string& get(const DataTypePtr& data_type) const {
auto type_ptr = data_type->is_nullable()
? ((DataTypeNullable*)(data_type.get()))->get_nested_type()
diff --git a/be/src/vec/data_types/data_type_struct.cpp b/be/src/vec/data_types/data_type_struct.cpp
index 91aff67a40..8fd2179bfa 100644
--- a/be/src/vec/data_types/data_type_struct.cpp
+++ b/be/src/vec/data_types/data_type_struct.cpp
@@ -43,16 +43,15 @@ DataTypeStruct::DataTypeStruct(const DataTypes& elems_)
}
}
-static std::optional<Exception> check_tuple_names(const Strings& names) {
+static Status check_tuple_names(const Strings& names) {
std::unordered_set<String> names_set;
for (const auto& name : names) {
if (name.empty()) {
- return Exception("Names of tuple elements cannot be empty", ErrorCodes::BAD_ARGUMENTS);
+ return Status::InvalidArgument("Names of tuple elements cannot be empty");
}
if (!names_set.insert(name).second) {
- return Exception("Names of tuple elements must be unique",
- ErrorCodes::DUPLICATE_COLUMN);
+ return Status::InvalidArgument("Names of tuple elements must be unique");
}
}
@@ -63,13 +62,12 @@ DataTypeStruct::DataTypeStruct(const DataTypes& elems_, const Strings& names_)
: elems(elems_), names(names_), have_explicit_names(true) {
size_t size = elems.size();
if (names.size() != size) {
- throw Exception("Wrong number of names passed to constructor of DataTypeStruct",
- ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH);
+ LOG(FATAL) << "Wrong number of names passed to constructor of DataTypeStruct";
}
- if (auto exception = check_tuple_names(names)) {
- throw std::move(*exception);
- }
+ Status st = check_tuple_names(names);
+ //if (!st.ok()) {
+ //}
}
std::string DataTypeStruct::do_get_name() const {
@@ -114,8 +112,7 @@ static void add_element_safe(const DataTypes& elems, IColumn& column, F&& impl)
if (element_column.size() != new_size) {
// This is not a logical error because it may work with
// user-supplied data.
- throw Exception("Cannot read a tuple because not all elements are present",
- ErrorCodes::SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH);
+ LOG(FATAL) << "Cannot read a tuple because not all elements are present";
}
}
} catch (...) {
@@ -168,10 +165,11 @@ MutableColumnPtr DataTypeStruct::create_column() const {
// return ColumnStruct::create(std::move(tuple_columns));
// }
-// Field DataTypeStruct::get_default() const {
-// return Tuple(collections::map<Tuple>(
-// elems, [](const DataTypePtr& elem) { return elem->get_default(); }));
-// }
+Field DataTypeStruct::get_default() const {
+ return Tuple();
+ //return Tuple(collections::map<Tuple>(
+ // elems, [](const DataTypePtr& elem) { return elem->get_default(); }));
+}
void DataTypeStruct::insert_default_into(IColumn& column) const {
add_element_safe(elems, column, [&] {
@@ -210,8 +208,7 @@ size_t DataTypeStruct::get_position_by_name(const String& name) const {
return i;
}
}
- throw Exception("Struct doesn't have element with name '" + name + "'",
- ErrorCodes::NOT_FOUND_COLUMN_IN_BLOCK);
+ LOG(FATAL) << "Struct doesn't have element with name '" + name + "'";
}
std::optional<size_t> DataTypeStruct::try_get_position_by_name(const String& name) const {
@@ -229,7 +226,7 @@ String DataTypeStruct::get_name_by_position(size_t i) const {
fmt::memory_buffer error_msg;
fmt::format_to(error_msg, "Index of tuple element ({}) if out range ([1, {}])", i,
names.size());
- throw Exception(fmt::to_string(error_msg), ErrorCodes::ILLEGAL_INDEX);
+ LOG(FATAL) << fmt::to_string(error_msg);
}
return names[i - 1];
diff --git a/be/src/vec/data_types/data_type_struct.h b/be/src/vec/data_types/data_type_struct.h
index 4201583ef4..9405544b76 100644
--- a/be/src/vec/data_types/data_type_struct.h
+++ b/be/src/vec/data_types/data_type_struct.h
@@ -62,7 +62,7 @@ public:
std::string do_get_name() const override;
const char* get_family_name() const override { return "Struct"; }
- bool can_be_inside_nullable() const override { return false; }
+ bool can_be_inside_nullable() const override { return true; }
bool supports_sparse_serialization() const { return true; }
MutableColumnPtr create_column() const override;
@@ -90,16 +90,18 @@ public:
std::optional<size_t> try_get_position_by_name(const String& name) const;
String get_name_by_position(size_t i) const;
- int64_t get_uncompressed_serialized_bytes(const IColumn& column,
- int be_exec_version) const override {
+ [[noreturn]] int64_t get_uncompressed_serialized_bytes(const IColumn& column,
+ int be_exec_version) const override {
LOG(FATAL) << "get_uncompressed_serialized_bytes not implemented";
}
- char* serialize(const IColumn& column, char* buf, int be_exec_version) const override {
+ [[noreturn]] char* serialize(const IColumn& column, char* buf,
+ int be_exec_version) const override {
LOG(FATAL) << "serialize not implemented";
}
- const char* deserialize(const char* buf, IColumn* column, int be_exec_version) const override {
+ [[noreturn]] const char* deserialize(const char* buf, IColumn* column,
+ int be_exec_version) const override {
LOG(FATAL) << "serialize not implemented";
}
diff --git a/be/src/vec/exprs/vcast_expr.cpp b/be/src/vec/exprs/vcast_expr.cpp
index 68f3b05cc1..6f90ede85b 100644
--- a/be/src/vec/exprs/vcast_expr.cpp
+++ b/be/src/vec/exprs/vcast_expr.cpp
@@ -38,6 +38,7 @@ doris::Status VCastExpr::prepare(doris::RuntimeState* state, const doris::RowDes
// create a const string column
_target_data_type = _data_type;
+ // TODO(xy): support return struct type name
_target_data_type_name = DataTypeFactory::instance().get(_target_data_type);
_cast_param_data_type = std::make_shared<DataTypeString>();
_cast_param = _cast_param_data_type->create_column_const(1, _target_data_type_name);
diff --git a/be/src/vec/exprs/vexpr.cpp b/be/src/vec/exprs/vexpr.cpp
index 9ad3ccc7e8..923aee07d6 100644
--- a/be/src/vec/exprs/vexpr.cpp
+++ b/be/src/vec/exprs/vexpr.cpp
@@ -35,6 +35,7 @@
#include "vec/exprs/vliteral.h"
#include "vec/exprs/vruntimefilter_wrapper.h"
#include "vec/exprs/vslot_ref.h"
+#include "vec/exprs/vstruct_literal.h"
#include "vec/exprs/vtuple_is_null_predicate.h"
namespace doris::vectorized {
@@ -124,6 +125,10 @@ Status VExpr::create_expr(doris::ObjectPool* pool, const doris::TExprNode& texpr
*expr = pool->add(new VArrayLiteral(texpr_node));
return Status::OK();
}
+ case TExprNodeType::STRUCT_LITERAL: {
+ *expr = pool->add(new VStructLiteral(texpr_node));
+ return Status::OK();
+ }
case doris::TExprNodeType::SLOT_REF: {
*expr = pool->add(new VSlotRef(texpr_node));
break;
diff --git a/be/src/vec/exprs/vstruct_literal.cpp b/be/src/vec/exprs/vstruct_literal.cpp
new file mode 100644
index 0000000000..9ae647b481
--- /dev/null
+++ b/be/src/vec/exprs/vstruct_literal.cpp
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/exprs/vstruct_literal.h"
+
+namespace doris::vectorized {
+
+Status VStructLiteral::prepare(RuntimeState* state, const RowDescriptor& row_desc,
+ VExprContext* context) {
+ RETURN_IF_ERROR_OR_PREPARED(VExpr::prepare(state, row_desc, context));
+ Field struct_field = Tuple();
+ for (const auto child : _children) {
+ Field item;
+ ColumnPtrWrapper* const_col_wrapper = nullptr;
+ RETURN_IF_ERROR(child->get_const_col(context, &const_col_wrapper));
+ const_col_wrapper->column_ptr->get(0, item);
+ struct_field.get<Tuple>().push_back(item);
+ }
+ _column_ptr = _data_type->create_column_const(1, struct_field);
+ return Status::OK();
+}
+
+} // namespace doris::vectorized
\ No newline at end of file
diff --git a/be/src/vec/exprs/vstruct_literal.h b/be/src/vec/exprs/vstruct_literal.h
new file mode 100644
index 0000000000..4cc4fe78d6
--- /dev/null
+++ b/be/src/vec/exprs/vstruct_literal.h
@@ -0,0 +1,34 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "vec/exprs/vliteral.h"
+
+namespace doris {
+
+namespace vectorized {
+class VStructLiteral : public VLiteral {
+public:
+ VStructLiteral(const TExprNode& node) : VLiteral(node, false) {}
+ ~VStructLiteral() override = default;
+ Status prepare(RuntimeState* state, const RowDescriptor& row_desc,
+ VExprContext* context) override;
+};
+} // namespace vectorized
+
+} // namespace doris
diff --git a/be/src/vec/functions/function_cast.h b/be/src/vec/functions/function_cast.h
index e3baaecdd2..d300c09a76 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -1754,6 +1754,7 @@ protected:
LOG(FATAL) << fmt::format(
"Second argument to {} must be a constant string describing type", get_name());
}
+ // TODO(xy): support return struct type for factory
auto type = DataTypeFactory::instance().get(type_col->get_value<String>());
DCHECK(type != nullptr);
diff --git a/be/src/vec/olap/olap_data_convertor.cpp b/be/src/vec/olap/olap_data_convertor.cpp
index e9b4c8fd24..0b3ad4ae03 100644
--- a/be/src/vec/olap/olap_data_convertor.cpp
+++ b/be/src/vec/olap/olap_data_convertor.cpp
@@ -20,8 +20,10 @@
#include "olap/tablet_schema.h"
#include "vec/columns/column_array.h"
#include "vec/columns/column_complex.h"
+#include "vec/columns/column_struct.h"
#include "vec/columns/column_vector.h"
#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_struct.h"
namespace doris::vectorized {
@@ -114,6 +116,14 @@ OlapBlockDataConvertor::create_olap_column_data_convertor(const TabletColumn& co
case FieldType::OLAP_FIELD_TYPE_DOUBLE: {
return std::make_unique<OlapColumnDataConvertorSimple<vectorized::Float64>>();
}
+ case FieldType::OLAP_FIELD_TYPE_STRUCT: {
+ std::vector<OlapColumnDataConvertorBaseUPtr> sub_convertors;
+ for (uint32_t i = 0; i < column.get_subtype_count(); i++) {
+ const TabletColumn& sub_column = column.get_sub_column(i);
+ sub_convertors.emplace_back(create_olap_column_data_convertor(sub_column));
+ }
+ return std::make_unique<OlapColumnDataConvertorStruct>(sub_convertors);
+ }
case FieldType::OLAP_FIELD_TYPE_ARRAY: {
const auto& sub_column = column.get_sub_column(0);
return std::make_unique<OlapColumnDataConvertorArray>(
@@ -639,6 +649,58 @@ Status OlapBlockDataConvertor::OlapColumnDataConvertorDecimal::convert_to_olap()
return Status::OK();
}
+void OlapBlockDataConvertor::OlapColumnDataConvertorStruct::set_source_column(
+ const ColumnWithTypeAndName& typed_column, size_t row_pos, size_t num_rows) {
+ OlapBlockDataConvertor::OlapColumnDataConvertorBase::set_source_column(typed_column, row_pos,
+ num_rows);
+}
+
+const void* OlapBlockDataConvertor::OlapColumnDataConvertorStruct::get_data() const {
+ return _results[0];
+}
+
+const void* OlapBlockDataConvertor::OlapColumnDataConvertorStruct::get_data_at(
+ size_t offset) const {
+ // Todo(xy): struct not supported
+ return nullptr;
+}
+
+Status OlapBlockDataConvertor::OlapColumnDataConvertorStruct::convert_to_olap() {
+ assert(_typed_column.column);
+ const vectorized::ColumnStruct* column_struct = nullptr;
+ const vectorized::DataTypeStruct* data_type_struct = nullptr;
+ if (_nullmap) {
+ auto nullable_column =
+ assert_cast<const vectorized::ColumnNullable*>(_typed_column.column.get());
+ column_struct = assert_cast<const vectorized::ColumnStruct*>(
+ nullable_column->get_nested_column_ptr().get());
+ data_type_struct = assert_cast<const DataTypeStruct*>(
+ (assert_cast<const DataTypeNullable*>(_typed_column.type.get())->get_nested_type())
+ .get());
+ } else {
+ column_struct = assert_cast<const vectorized::ColumnStruct*>(_typed_column.column.get());
+ data_type_struct = assert_cast<const DataTypeStruct*>(_typed_column.type.get());
+ }
+ assert(column_struct);
+ assert(data_type_struct);
+
+ size_t data_size = column_struct->tuple_size();
+ size_t data_cursor = 0;
+ size_t null_map_cursor = data_cursor + data_size;
+ for (size_t i = 0; i < data_size; i++) {
+ ColumnPtr sub_column = column_struct->get_column_ptr(i);
+ DataTypePtr sub_type = data_type_struct->get_element(i);
+ ColumnWithTypeAndName sub_typed_column = {sub_column, sub_type, ""};
+ _sub_convertors[i]->set_source_column(sub_typed_column, _row_pos, _num_rows);
+ _sub_convertors[i]->convert_to_olap();
+ _results[data_cursor] = _sub_convertors[i]->get_data();
+ _results[null_map_cursor] = _sub_convertors[i]->get_nullmap();
+ data_cursor++;
+ null_map_cursor++;
+ }
+ return Status::OK();
+}
+
Status OlapBlockDataConvertor::OlapColumnDataConvertorArray::convert_to_olap() {
const ColumnArray* column_array = nullptr;
const DataTypeArray* data_type_array = nullptr;
diff --git a/be/src/vec/olap/olap_data_convertor.h b/be/src/vec/olap/olap_data_convertor.h
index 6898b44a9d..aafbe8d3e3 100644
--- a/be/src/vec/olap/olap_data_convertor.h
+++ b/be/src/vec/olap/olap_data_convertor.h
@@ -359,6 +359,27 @@ private:
}
};
+ class OlapColumnDataConvertorStruct : public OlapColumnDataConvertorBase {
+ public:
+ OlapColumnDataConvertorStruct(
+ std::vector<OlapColumnDataConvertorBaseUPtr>& sub_convertors) {
+ for (auto& sub_convertor : sub_convertors) {
+ _sub_convertors.push_back(std::move(sub_convertor));
+ }
+ size_t allocate_size = _sub_convertors.size() * 2;
+ _results.resize(allocate_size);
+ }
+ void set_source_column(const ColumnWithTypeAndName& typed_column, size_t row_pos,
+ size_t num_rows) override;
+ const void* get_data() const override;
+ const void* get_data_at(size_t offset) const override;
+ Status convert_to_olap() override;
+
+ private:
+ std::vector<OlapColumnDataConvertorBaseUPtr> _sub_convertors;
+ std::vector<const void*> _results;
+ };
+
class OlapColumnDataConvertorArray
: public OlapColumnDataConvertorPaddedPODArray<CollectionValue> {
public:
diff --git a/be/src/vec/sink/vmysql_result_writer.cpp b/be/src/vec/sink/vmysql_result_writer.cpp
index 4c7f5028d7..2accb4dd03 100644
--- a/be/src/vec/sink/vmysql_result_writer.cpp
+++ b/be/src/vec/sink/vmysql_result_writer.cpp
@@ -29,6 +29,7 @@
#include "vec/common/assert_cast.h"
#include "vec/data_types/data_type_array.h"
#include "vec/data_types/data_type_decimal.h"
+#include "vec/data_types/data_type_struct.h"
#include "vec/exprs/vexpr.h"
#include "vec/exprs/vexpr_context.h"
#include "vec/runtime/vdatetime_value.h"
@@ -61,8 +62,8 @@ void VMysqlResultWriter::_init_profile() {
template <PrimitiveType type, bool is_nullable>
Status VMysqlResultWriter::_add_one_column(const ColumnPtr& column_ptr,
- std::unique_ptr<TFetchDataResult>& result,
- const DataTypePtr& nested_type_ptr, int scale) {
+ std::unique_ptr<TFetchDataResult>& result, int scale,
+ const DataTypes& sub_types) {
SCOPED_TIMER(_convert_tuple_timer);
const auto row_size = column_ptr->size();
@@ -145,6 +146,7 @@ Status VMysqlResultWriter::_add_one_column(const ColumnPtr& column_ptr,
result->result_batch.rows[i].append(_buffer.buf(), _buffer.length());
}
} else if constexpr (type == TYPE_ARRAY) {
+ DCHECK_EQ(sub_types.size(), 1);
auto& column_array = assert_cast<const ColumnArray&>(*column);
auto& offsets = column_array.get_offsets();
for (ssize_t i = 0; i < row_size; ++i) {
@@ -172,12 +174,12 @@ Status VMysqlResultWriter::_add_one_column(const ColumnPtr& column_ptr,
if (data->is_null_at(j)) {
buf_ret = _buffer.push_string("NULL", strlen("NULL"));
} else {
- if (WhichDataType(remove_nullable(nested_type_ptr)).is_string()) {
+ if (WhichDataType(remove_nullable(sub_types[0])).is_string()) {
buf_ret = _buffer.push_string("'", 1);
- buf_ret = _add_one_cell(data, j, nested_type_ptr, _buffer);
+ buf_ret = _add_one_cell(data, j, sub_types[0], _buffer);
buf_ret = _buffer.push_string("'", 1);
} else {
- buf_ret = _add_one_cell(data, j, nested_type_ptr, _buffer);
+ buf_ret = _add_one_cell(data, j, sub_types[0], _buffer);
}
}
begin = false;
@@ -186,8 +188,51 @@ Status VMysqlResultWriter::_add_one_column(const ColumnPtr& column_ptr,
_buffer.close_dynamic_mode();
result->result_batch.rows[i].append(_buffer.buf(), _buffer.length());
}
+ } else if constexpr (type == TYPE_STRUCT) {
+ DCHECK_GE(sub_types.size(), 1);
+ auto& column_struct = assert_cast<const ColumnStruct&>(*column);
+ for (ssize_t i = 0; i < row_size; ++i) {
+ if (0 != buf_ret) {
+ return Status::InternalError("pack mysql buffer failed.");
+ }
+ _buffer.reset();
+
+ if constexpr (is_nullable) {
+ if (column_ptr->is_null_at(i)) {
+ buf_ret = _buffer.push_null();
+ result->result_batch.rows[i].append(_buffer.buf(), _buffer.length());
+ continue;
+ }
+ }
+
+ _buffer.open_dynamic_mode();
+ buf_ret = _buffer.push_string("{", 1);
+ bool begin = true;
+ for (size_t j = 0; j < sub_types.size(); ++j) {
+ if (!begin) {
+ buf_ret = _buffer.push_string(", ", 2);
+ }
+ const auto& data = column_struct.get_column_ptr(j);
+ if (data->is_null_at(j)) {
+ buf_ret = _buffer.push_string("NULL", strlen("NULL"));
+ } else {
+ if (WhichDataType(remove_nullable(sub_types[j])).is_string()) {
+ buf_ret = _buffer.push_string("'", 1);
+ buf_ret = _add_one_cell(data, j, sub_types[j], _buffer);
+ buf_ret = _buffer.push_string("'", 1);
+ } else {
+ buf_ret = _add_one_cell(data, j, sub_types[j], _buffer);
+ }
+ }
+ begin = false;
+ }
+ buf_ret = _buffer.push_string("}", 1);
+ _buffer.close_dynamic_mode();
+ result->result_batch.rows[i].append(_buffer.buf(), _buffer.length());
+ }
} else if constexpr (type == TYPE_DECIMAL32 || type == TYPE_DECIMAL64 ||
type == TYPE_DECIMAL128I) {
+ DCHECK_EQ(sub_types.size(), 1);
for (int i = 0; i < row_size; ++i) {
if (0 != buf_ret) {
return Status::InternalError("pack mysql buffer failed.");
@@ -201,7 +246,7 @@ Status VMysqlResultWriter::_add_one_column(const ColumnPtr& column_ptr,
continue;
}
}
- std::string decimal_str = nested_type_ptr->to_string(*column, i);
+ std::string decimal_str = sub_types[0]->to_string(*column, i);
buf_ret = _buffer.push_string(decimal_str.c_str(), decimal_str.length());
result->result_batch.rows[i].append(_buffer.buf(), _buffer.length());
}
@@ -397,6 +442,7 @@ int VMysqlResultWriter::_add_one_cell(const ColumnPtr& column_ptr, size_t row_id
auto decimal_str = assert_cast<const DataTypeDecimal<Decimal128I>*>(nested_type.get())
->to_string(*column, row_idx);
return buffer.push_string(decimal_str.c_str(), decimal_str.length());
+ // TODO(xy): support nested struct
} else if (which.is_array()) {
auto& column_array = assert_cast<const ColumnArray&>(*column);
auto& offsets = column_array.get_offsets();
@@ -554,10 +600,10 @@ Status VMysqlResultWriter::append_block(Block& input_block) {
auto& nested_type =
assert_cast<const DataTypeNullable&>(*type_ptr).get_nested_type();
status = _add_one_column<PrimitiveType::TYPE_DECIMALV2, true>(column_ptr, result,
- nested_type, scale);
+ scale, {nested_type});
} else {
status = _add_one_column<PrimitiveType::TYPE_DECIMALV2, false>(column_ptr, result,
- type_ptr, scale);
+ scale, {type_ptr});
}
break;
}
@@ -566,10 +612,10 @@ Status VMysqlResultWriter::append_block(Block& input_block) {
auto& nested_type =
assert_cast<const DataTypeNullable&>(*type_ptr).get_nested_type();
status = _add_one_column<PrimitiveType::TYPE_DECIMAL32, true>(column_ptr, result,
- nested_type, scale);
+ scale, {nested_type});
} else {
status = _add_one_column<PrimitiveType::TYPE_DECIMAL32, false>(column_ptr, result,
- type_ptr, scale);
+ scale, {type_ptr});
}
break;
}
@@ -578,10 +624,10 @@ Status VMysqlResultWriter::append_block(Block& input_block) {
auto& nested_type =
assert_cast<const DataTypeNullable&>(*type_ptr).get_nested_type();
status = _add_one_column<PrimitiveType::TYPE_DECIMAL64, true>(column_ptr, result,
- nested_type, scale);
+ scale, {nested_type});
} else {
status = _add_one_column<PrimitiveType::TYPE_DECIMAL64, false>(column_ptr, result,
- type_ptr, scale);
+ scale, {type_ptr});
}
break;
}
@@ -589,11 +635,11 @@ Status VMysqlResultWriter::append_block(Block& input_block) {
if (type_ptr->is_nullable()) {
auto& nested_type =
assert_cast<const DataTypeNullable&>(*type_ptr).get_nested_type();
- status = _add_one_column<PrimitiveType::TYPE_DECIMAL128I, true>(column_ptr, result,
- nested_type, scale);
+ status = _add_one_column<PrimitiveType::TYPE_DECIMAL128I, true>(
+ column_ptr, result, scale, {nested_type});
} else {
status = _add_one_column<PrimitiveType::TYPE_DECIMAL128I, false>(column_ptr, result,
- type_ptr, scale);
+ scale, {type_ptr});
}
break;
}
@@ -625,10 +671,10 @@ Status VMysqlResultWriter::append_block(Block& input_block) {
case TYPE_DATETIMEV2: {
if (type_ptr->is_nullable()) {
status = _add_one_column<PrimitiveType::TYPE_DATETIMEV2, true>(column_ptr, result,
- nullptr, scale);
+ scale);
} else {
status = _add_one_column<PrimitiveType::TYPE_DATETIMEV2, false>(column_ptr, result,
- nullptr, scale);
+ scale);
}
break;
}
@@ -646,12 +692,26 @@ Status VMysqlResultWriter::append_block(Block& input_block) {
auto& nested_type =
assert_cast<const DataTypeNullable&>(*type_ptr).get_nested_type();
auto& sub_type = assert_cast<const DataTypeArray&>(*nested_type).get_nested_type();
- status = _add_one_column<PrimitiveType::TYPE_ARRAY, true>(column_ptr, result,
- sub_type);
+ status = _add_one_column<PrimitiveType::TYPE_ARRAY, true>(column_ptr, result, scale,
+ {sub_type});
} else {
auto& sub_type = assert_cast<const DataTypeArray&>(*type_ptr).get_nested_type();
status = _add_one_column<PrimitiveType::TYPE_ARRAY, false>(column_ptr, result,
- sub_type);
+ scale, {sub_type});
+ }
+ break;
+ }
+ case TYPE_STRUCT: {
+ if (type_ptr->is_nullable()) {
+ auto& nested_type =
+ assert_cast<const DataTypeNullable&>(*type_ptr).get_nested_type();
+ auto& sub_types = assert_cast<const DataTypeStruct&>(*nested_type).get_elements();
+ status = _add_one_column<PrimitiveType::TYPE_STRUCT, true>(column_ptr, result,
+ scale, sub_types);
+ } else {
+ auto& sub_types = assert_cast<const DataTypeStruct&>(*type_ptr).get_elements();
+ status = _add_one_column<PrimitiveType::TYPE_STRUCT, false>(column_ptr, result,
+ scale, sub_types);
}
break;
}
diff --git a/be/src/vec/sink/vmysql_result_writer.h b/be/src/vec/sink/vmysql_result_writer.h
index 3f79f0e2d6..db61dcbf41 100644
--- a/be/src/vec/sink/vmysql_result_writer.h
+++ b/be/src/vec/sink/vmysql_result_writer.h
@@ -49,7 +49,7 @@ private:
template <PrimitiveType type, bool is_nullable>
Status _add_one_column(const ColumnPtr& column_ptr, std::unique_ptr<TFetchDataResult>& result,
- const DataTypePtr& nested_type_ptr = nullptr, int scale = -1);
+ int scale = -1, const DataTypes& sub_types = DataTypes());
int _add_one_cell(const ColumnPtr& column_ptr, size_t row_idx, const DataTypePtr& type,
MysqlRowBuffer& buffer);
diff --git a/be/src/vec/sink/vtablet_sink.cpp b/be/src/vec/sink/vtablet_sink.cpp
index 32bb070d22..2f4cc66cc9 100644
--- a/be/src/vec/sink/vtablet_sink.cpp
+++ b/be/src/vec/sink/vtablet_sink.cpp
@@ -1403,11 +1403,12 @@ Status VOlapTableSink::_validate_column(RuntimeState* state, const TypeDescripto
}
fmt::format_to(error_prefix, "ARRAY type failed: ");
RETURN_IF_ERROR(_validate_column(
- state, nested_type, nested_type.contains_null, column_array->get_data_ptr(),
+ state, nested_type, nested_type.contains_nulls[0], column_array->get_data_ptr(),
slot_index, filter_bitmap, stop_processing, error_prefix, &permutation));
}
break;
}
+ // TODO(xy): add struct type validate
default:
break;
}
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup b/fe/fe-core/src/main/cup/sql_parser.cup
index c81e8b6423..35be33e918 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -618,7 +618,7 @@ terminal String
KW_TYPECAST,
KW_HISTOGRAM;
-terminal COMMA, COLON, DOT, DOTDOTDOT, AT, STAR, LPAREN, RPAREN, SEMICOLON, LBRACKET, RBRACKET, DIVIDE, MOD, ADD, SUBTRACT;
+terminal COMMA, COLON, DOT, DOTDOTDOT, AT, STAR, LPAREN, RPAREN, SEMICOLON, LBRACKET, RBRACKET, LBRACE, RBRACE, DIVIDE, MOD, ADD, SUBTRACT;
terminal BITAND, BITOR, BITXOR, BITNOT;
terminal EQUAL, NOT, LESSTHAN, GREATERTHAN, SET_VAR;
terminal COMMENTED_PLAN_HINT_START, COMMENTED_PLAN_HINT_END;
@@ -729,6 +729,7 @@ nonterminal Expr function_call_expr, array_expr;
nonterminal ArrayLiteral array_literal;
nonterminal StructField struct_field;
nonterminal ArrayList<StructField> struct_field_list;
+nonterminal StructLiteral struct_literal;
nonterminal AnalyticWindow opt_window_clause;
nonterminal AnalyticWindow.Type window_type;
nonterminal AnalyticWindow.Boundary window_boundary;
@@ -928,6 +929,7 @@ precedence left KW_PARTITION;
precedence left KW_PARTITIONS;
precedence right KW_TEMPORARY;
precedence right LBRACKET;
+precedence right LBRACE;
precedence left KW_ENGINE;
// unused
@@ -5835,16 +5837,23 @@ struct_field ::=
;
struct_field_list ::=
- struct_field:field
- {:
- RESULT = Lists.newArrayList(field);
- :}
- | struct_field_list:fields COMMA struct_field:field
- {:
- fields.add(field);
- RESULT = fields;
- :}
- ;
+ struct_field:field
+ {:
+ RESULT = Lists.newArrayList(field);
+ :}
+ | struct_field_list:fields COMMA struct_field:field
+ {:
+ fields.add(field);
+ RESULT = fields;
+ :}
+ ;
+
+struct_literal ::=
+ LBRACE expr_list:list RBRACE
+ {:
+ RESULT = new StructLiteral(list.toArray(new LiteralExpr[0]));
+ :}
+ ;
exists_predicate ::=
KW_EXISTS subquery:s
@@ -5868,6 +5877,8 @@ non_pred_expr ::=
{: RESULT = a; :}
| array_literal:a
{: RESULT = a; :}
+ | struct_literal:s
+ {: RESULT = s; :}
| function_call_expr:e
{: RESULT = e; :}
| KW_DATE STRING_LITERAL:l
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
index 66670db2f2..3f57f92af8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
@@ -292,6 +292,10 @@ public class ColumnDef {
}
if (type.getPrimitiveType() == PrimitiveType.STRUCT) {
+ if (isKey()) {
+ throw new AnalysisException("Struct can only be used in the non-key column of"
+ + " the duplicate table at present.");
+ }
if (defaultValue.isSet && defaultValue != DefaultValue.NULL_DEFAULT_VALUE) {
throw new AnalysisException("Struct type column default value just support null");
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index 37d73d7bb0..72fbfcadd0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -336,7 +336,7 @@ public class CreateTableStmt extends DdlStmt {
// So the float and double could not be the first column in OLAP table.
if (keysColumnNames.isEmpty()) {
throw new AnalysisException("The olap table first column could not be float, double, string"
- + " or array, please use decimal or varchar instead.");
+ + " or array, struct, map, please use decimal or varchar instead.");
}
keysDesc = new KeysDesc(KeysType.DUP_KEYS, keysColumnNames);
}
@@ -390,14 +390,14 @@ public class CreateTableStmt extends DdlStmt {
for (ColumnDef columnDef : columnDefs) {
columnDef.analyze(engineName.equals("olap"));
- if (columnDef.getType().isArrayType() && engineName.equals("olap")) {
+ if (columnDef.getType().isComplexType()) {
if (columnDef.getAggregateType() != null && columnDef.getAggregateType() != AggregateType.NONE) {
- throw new AnalysisException("Array column can't support aggregation "
- + columnDef.getAggregateType());
+ throw new AnalysisException(columnDef.getType().getPrimitiveType()
+ + " column can't support aggregation " + columnDef.getAggregateType());
}
if (columnDef.isKey()) {
- throw new AnalysisException("Array can only be used in the non-key column of"
- + " the duplicate table at present.");
+ throw new AnalysisException(columnDef.getType().getPrimitiveType()
+ + " can only be used in the non-key column of the duplicate table at present.");
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java
index 35bd80e9a3..31b2382f0a 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Expr.java
@@ -1790,7 +1790,8 @@ public abstract class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
ARRAY_LITERAL(13),
CAST_EXPR(14),
JSON_LITERAL(15),
- ARITHMETIC_EXPR(16);
+ ARITHMETIC_EXPR(16),
+ STRUCT_LITERAL(17);
private static Map<Integer, ExprSerCode> codeMap = Maps.newHashMap();
@@ -1842,6 +1843,8 @@ public abstract class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
output.writeInt(ExprSerCode.FUNCTION_CALL.getCode());
} else if (expr instanceof ArrayLiteral) {
output.writeInt(ExprSerCode.ARRAY_LITERAL.getCode());
+ } else if (expr instanceof StructLiteral) {
+ output.writeInt(ExprSerCode.STRUCT_LITERAL.getCode());
} else if (expr instanceof CastExpr) {
output.writeInt(ExprSerCode.CAST_EXPR.getCode());
} else if (expr instanceof ArithmeticExpr) {
@@ -1891,6 +1894,8 @@ public abstract class Expr extends TreeNode<Expr> implements ParseNode, Cloneabl
return FunctionCallExpr.read(in);
case ARRAY_LITERAL:
return ArrayLiteral.read(in);
+ case STRUCT_LITERAL:
+ return StructLiteral.read(in);
case CAST_EXPR:
return CastExpr.read(in);
case ARITHMETIC_EXPR:
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/StructLiteral.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/StructLiteral.java
new file mode 100644
index 0000000000..a9209f1c5a
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StructLiteral.java
@@ -0,0 +1,150 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.StructField;
+import org.apache.doris.catalog.StructType;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.thrift.TExprNode;
+import org.apache.doris.thrift.TExprNodeType;
+
+import org.apache.commons.lang.StringUtils;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+public class StructLiteral extends LiteralExpr {
+ // only for persist
+ public StructLiteral() {
+ type = new StructType();
+ children = new ArrayList<>();
+ }
+
+ public StructLiteral(LiteralExpr... exprs) throws AnalysisException {
+ type = new StructType();
+ children = new ArrayList<>();
+ for (LiteralExpr expr : exprs) {
+ if (!type.supportSubType(expr.getType())) {
+ throw new AnalysisException("Invalid element type in STRUCT.");
+ }
+ ((StructType) type).addField(new StructField(expr.getType()));
+ children.add(expr);
+ }
+ }
+
+ protected StructLiteral(StructLiteral other) {
+ super(other);
+ }
+
+ @Override
+ protected String toSqlImpl() {
+ List<String> list = new ArrayList<>(children.size());
+ children.forEach(v -> list.add(v.toDigestImpl()));
+ return "STRUCT(" + StringUtils.join(list, ", ") + ")";
+ }
+
+ @Override
+ public String toDigestImpl() {
+ List<String> list = new ArrayList<>(children.size());
+ children.forEach(v -> list.add(v.toDigestImpl()));
+ return "STRUCT(" + StringUtils.join(list, ", ") + ")";
+ }
+
+ @Override
+ public String getStringValue() {
+ List<String> list = new ArrayList<>(children.size());
+ children.forEach(v -> list.add(v.getStringValue()));
+ return "{" + StringUtils.join(list, ", ") + "}";
+ }
+
+ @Override
+ public String getStringValueForArray() {
+ return null;
+ }
+
+ @Override
+ protected void toThrift(TExprNode msg) {
+ msg.node_type = TExprNodeType.STRUCT_LITERAL;
+ }
+
+ @Override
+ public void write(DataOutput out) throws IOException {
+ super.write(out);
+ out.writeInt(children.size());
+ for (Expr e : children) {
+ Expr.writeTo(e, out);
+ }
+ }
+
+ @Override
+ public void readFields(DataInput in) throws IOException {
+ super.readFields(in);
+ int size = in.readInt();
+ children = new ArrayList<>(size);
+ for (int i = 0; i < size; i++) {
+ children.add(Expr.readIn(in));
+ }
+ }
+
+ public static StructLiteral read(DataInput in) throws IOException {
+ StructLiteral literal = new StructLiteral();
+ literal.readFields(in);
+ return literal;
+ }
+
+ @Override
+ public Expr clone() {
+ return new StructLiteral(this);
+ }
+
+ @Override
+ public boolean isMinValue() {
+ return false;
+ }
+
+ @Override
+ public int compareLiteral(LiteralExpr expr) {
+ return 0;
+ }
+
+ @Override
+ public Expr uncheckedCastTo(Type targetType) throws AnalysisException {
+ if (!targetType.isStructType()) {
+ return super.uncheckedCastTo(targetType);
+ }
+ ArrayList<StructField> fields = ((StructType) targetType).getFields();
+ StructLiteral literal = new StructLiteral(this);
+ for (int i = 0; i < children.size(); ++ i) {
+ Expr child = children.get(i);
+ literal.children.set(i, child.uncheckedCastTo((fields.get(i).getType())));
+ }
+ literal.setType(targetType);
+ return literal;
+ }
+
+ @Override
+ public void checkValueValid() throws AnalysisException {
+ for (Expr e : children) {
+ e.checkValueValid();
+ }
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Subquery.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/Subquery.java
index 4246edb5e4..c318e067a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Subquery.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Subquery.java
@@ -172,7 +172,7 @@ public class Subquery extends Expr {
fieldName = "_" + Integer.toString(i);
}
Preconditions.checkNotNull(fieldName);
- structFields.add(new StructField(fieldName, expr.getType(), null));
+ structFields.add(new StructField(fieldName, expr.getType()));
}
Preconditions.checkState(structFields.size() != 0);
return new StructType(structFields);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
index df00ef62b5..5690c115c1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
@@ -30,6 +30,8 @@ import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Set;
/**
* Represents an anonymous type definition, e.g., used in DDL and CASTs.
@@ -94,45 +96,46 @@ public class TypeDef implements ParseNode {
if (type.isArrayType()) {
Type itemType = ((ArrayType) type).getItemType();
if (itemType instanceof ScalarType) {
- analyzeNestedType((ScalarType) itemType);
+ analyzeNestedType(type, (ScalarType) itemType);
}
}
if (type.isMapType()) {
ScalarType keyType = (ScalarType) ((MapType) type).getKeyType();
ScalarType valueType = (ScalarType) ((MapType) type).getKeyType();
- analyzeNestedType(keyType);
- analyzeNestedType(valueType);
+ analyzeNestedType(type, keyType);
+ analyzeNestedType(type, valueType);
}
if (type.isStructType()) {
ArrayList<StructField> fields = ((StructType) type).getFields();
- for (int i = 0; i < fields.size(); i++) {
- ScalarType filedType = (ScalarType) fields.get(i).getType();
- analyzeNestedType(filedType);
+ Set<String> fieldNames = new HashSet<>();
+ for (StructField field : fields) {
+ Type fieldType = field.getType();
+ if (fieldType instanceof ScalarType) {
+ analyzeNestedType(type, (ScalarType) fieldType);
+ if (!fieldNames.add(field.getName())) {
+ throw new AnalysisException("Duplicate field name "
+ + field.getName() + " in struct " + type.toSql());
+ }
+ }
}
}
}
}
- private void analyzeNestedType(ScalarType type) throws AnalysisException {
- if (type.isNull()) {
- throw new AnalysisException("Unsupported data type: " + type.toSql());
- }
- // check whether the array sub-type is supported
- Boolean isSupportType = false;
- for (Type subType : Type.getArraySubTypes()) {
- if (type.getPrimitiveType() == subType.getPrimitiveType()) {
- isSupportType = true;
- break;
- }
+ private void analyzeNestedType(Type parent, ScalarType child) throws AnalysisException {
+ if (child.isNull()) {
+ throw new AnalysisException("Unsupported data type: " + child.toSql());
}
- if (!isSupportType) {
- throw new AnalysisException("Array unsupported sub-type: " + type.toSql());
+ // check whether the sub-type is supported
+ if (!parent.supportSubType(child)) {
+ throw new AnalysisException(
+ parent.getPrimitiveType() + "unsupported sub-type: " + child.toSql());
}
- if (type.getPrimitiveType().isStringType() && !type.isLengthSet()) {
- type.setLength(1);
+ if (child.getPrimitiveType().isStringType() && !child.isLengthSet()) {
+ child.setLength(1);
}
- analyze(type);
+ analyze(child);
}
private void analyzeScalarType(ScalarType scalarType)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
index 45d01019d1..8aa6acb31a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/ArrayType.java
@@ -159,6 +159,16 @@ public class ArrayType extends Type {
return !itemType.isNull();
}
+ @Override
+ public boolean supportSubType(Type subType) {
+ for (Type supportedType : getArraySubTypes()) {
+ if (subType.getPrimitiveType() == supportedType.getPrimitiveType()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
@Override
public String toString() {
return toSql(0).toUpperCase();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
index ae96365332..9efe901907 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
@@ -111,7 +111,7 @@ public class Column implements Writable, GsonPostProcessable {
this.stats = new ColumnStats();
this.visible = true;
this.defineExpr = null;
- this.children = new ArrayList<>(Type.MAX_NESTING_DEPTH);
+ this.children = new ArrayList<>();
this.uniqueId = -1;
}
@@ -160,7 +160,7 @@ public class Column implements Writable, GsonPostProcessable {
this.comment = comment;
this.stats = new ColumnStats();
this.visible = visible;
- this.children = new ArrayList<>(Type.MAX_NESTING_DEPTH);
+ this.children = new ArrayList<>();
createChildrenColumn(this.type, this);
this.uniqueId = colUniqueId;
}
@@ -187,6 +187,13 @@ public class Column implements Writable, GsonPostProcessable {
Column c = new Column(COLUMN_ARRAY_CHILDREN, ((ArrayType) type).getItemType());
c.setIsAllowNull(((ArrayType) type).getContainsNull());
column.addChildrenColumn(c);
+ } else if (type.isStructType()) {
+ ArrayList<StructField> fields = ((StructType) type).getFields();
+ for (StructField field : fields) {
+ Column c = new Column(field.getName(), field.getType());
+ c.setIsAllowNull(field.getContainsNull());
+ column.addChildrenColumn(c);
+ }
}
}
@@ -401,34 +408,41 @@ public class Column implements Writable, GsonPostProcessable {
return tColumn;
}
+ private void setChildrenTColumn(Column children, TColumn tColumn) {
+ TColumn childrenTColumn = new TColumn();
+ childrenTColumn.setColumnName(children.name);
+
+ TColumnType childrenTColumnType = new TColumnType();
+ childrenTColumnType.setType(children.getDataType().toThrift());
+ childrenTColumnType.setLen(children.getStrLen());
+ childrenTColumnType.setPrecision(children.getPrecision());
+ childrenTColumnType.setScale(children.getScale());
+ childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
+
+ childrenTColumn.setColumnType(childrenTColumnType);
+ childrenTColumn.setIsAllowNull(children.isAllowNull());
+ // TODO: If we don't set the aggregate type for children, the type will be
+ // considered as TAggregationType::SUM after deserializing in BE.
+ // For now, we make children inherit the aggregate type from their parent.
+ if (tColumn.getAggregationType() != null) {
+ childrenTColumn.setAggregationType(tColumn.getAggregationType());
+ }
+
+ tColumn.children_column.add(childrenTColumn);
+ toChildrenThrift(children, childrenTColumn);
+ }
+
private void toChildrenThrift(Column column, TColumn tColumn) {
if (column.type.isArrayType()) {
Column children = column.getChildren().get(0);
-
- TColumn childrenTColumn = new TColumn();
- childrenTColumn.setColumnName(children.name);
-
- TColumnType childrenTColumnType = new TColumnType();
- childrenTColumnType.setType(children.getDataType().toThrift());
- childrenTColumnType.setType(children.getDataType().toThrift());
- childrenTColumnType.setLen(children.getStrLen());
- childrenTColumnType.setPrecision(children.getPrecision());
- childrenTColumnType.setScale(children.getScale());
-
- childrenTColumnType.setIndexLen(children.getOlapColumnIndexSize());
- childrenTColumn.setColumnType(childrenTColumnType);
- childrenTColumn.setIsAllowNull(children.isAllowNull());
- // TODO: If we don't set the aggregate type for children, the type will be
- // considered as TAggregationType::SUM after deserializing in BE.
- // For now, we make children inherit the aggregate type from their parent.
- if (tColumn.getAggregationType() != null) {
- childrenTColumn.setAggregationType(tColumn.getAggregationType());
- }
-
tColumn.setChildrenColumn(new ArrayList<>());
- tColumn.children_column.add(childrenTColumn);
-
- toChildrenThrift(children, childrenTColumn);
+ setChildrenTColumn(children, tColumn);
+ } else if (column.type.isStructType()) {
+ List<Column> childrenColumns = column.getChildren();
+ tColumn.setChildrenColumn(new ArrayList<>());
+ for (Column children : childrenColumns) {
+ setChildrenTColumn(children, tColumn);
+ }
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MapType.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MapType.java
index 1a746a2374..adf74f5c6a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MapType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MapType.java
@@ -88,6 +88,11 @@ public class MapType extends Type {
return String.format("%sMAP<%s,%s>", leftPadding, keyType.toSql(), structStr);
}
+ @Override
+ public boolean supportSubType(Type subType) {
+ return true;
+ }
+
@Override
public void toThrift(TTypeDesc container) {
TTypeNode node = new TTypeNode();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
index e91face521..9e279c8d2d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
@@ -68,7 +68,10 @@ public enum PrimitiveType {
// sizeof(CollectionValue)
ARRAY("ARRAY", 32, TPrimitiveType.ARRAY),
MAP("MAP", 24, TPrimitiveType.MAP),
- STRUCT("STRUCT", 24, TPrimitiveType.STRUCT),
+ // sizeof(StructValue)
+ // 8-byte pointer and 4-byte size and 1 bytes has_null (13 bytes total)
+ // Aligning to 16 bytes total.
+ STRUCT("STRUCT", 16, TPrimitiveType.STRUCT),
STRING("STRING", 16, TPrimitiveType.STRING),
// Unsupported scalar types.
BINARY("BINARY", -1, TPrimitiveType.BINARY),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/StructField.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/StructField.java
index 6e87b1be40..a084bfe0e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/StructField.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/StructField.java
@@ -22,25 +22,43 @@ import org.apache.doris.thrift.TTypeDesc;
import org.apache.doris.thrift.TTypeNode;
import com.google.common.base.Strings;
+import com.google.gson.annotations.SerializedName;
/**
* TODO: Support comments for struct fields. The Metastore does not properly store
* comments of struct fields. We set comment to null to avoid compatibility issues.
*/
public class StructField {
+ @SerializedName(value = "name")
protected final String name;
+
+ @SerializedName(value = "type")
protected final Type type;
+
+ @SerializedName(value = "comment")
protected final String comment;
+
+ @SerializedName(value = "position")
protected int position; // in struct
- public StructField(String name, Type type, String comment) {
- this.name = name;
+ @SerializedName(value = "containsNull")
+ private final boolean containsNull; // Now always true (nullable field)
+
+ private static final String DEFAULT_FIELD_NAME = "col";
+
+ public StructField(String name, Type type, String comment, boolean containsNull) {
+ this.name = name.toLowerCase();
this.type = type;
this.comment = comment;
+ this.containsNull = containsNull;
}
public StructField(String name, Type type) {
- this(name, type, null);
+ this(name, type, null, true);
+ }
+
+ public StructField(Type type) {
+ this(DEFAULT_FIELD_NAME, type, null, true);
}
public String getComment() {
@@ -63,11 +81,20 @@ public class StructField {
this.position = position;
}
+ public boolean getContainsNull() {
+ return containsNull;
+ }
+
public String toSql(int depth) {
- String typeSql = (depth < Type.MAX_NESTING_DEPTH) ? type.toSql(depth) : "...";
+ String typeSql;
+ if (depth < Type.MAX_NESTING_DEPTH) {
+ typeSql = !containsNull ? "not_null(" + type.toSql(depth) + ")" : type.toSql(depth);
+ } else {
+ typeSql = "...";
+ }
StringBuilder sb = new StringBuilder(name);
if (type != null) {
- sb.append(":" + typeSql);
+ sb.append(":").append(typeSql);
}
if (comment != null) {
sb.append(String.format(" COMMENT '%s'", comment));
@@ -87,7 +114,7 @@ public class StructField {
// even if we then strip the top-level padding.
String typeStr = type.prettyPrint(lpad);
typeStr = typeStr.substring(lpad);
- sb.append(":" + typeStr);
+ sb.append(":").append(typeStr);
}
if (comment != null) {
sb.append(String.format(" COMMENT '%s'", comment));
@@ -95,12 +122,25 @@ public class StructField {
return sb.toString();
}
+ public static boolean canCastTo(StructField field, StructField targetField) {
+ // TODO(xy): support cast field
+ return false;
+ }
+
+ public boolean matchesField(StructField f) {
+ if (equals(f)) {
+ return true;
+ }
+ return type.matchesType(f.getType()) && containsNull == f.getContainsNull();
+ }
+
public void toThrift(TTypeDesc container, TTypeNode node) {
TStructField field = new TStructField();
field.setName(name);
if (comment != null) {
field.setComment(comment);
}
+ field.setContainsNull(containsNull);
node.struct_fields.add(field);
type.toThrift(container);
}
@@ -111,6 +151,7 @@ public class StructField {
return false;
}
StructField otherStructField = (StructField) other;
- return otherStructField.name.equals(name) && otherStructField.type.equals(type);
+ return otherStructField.name.equals(name) && otherStructField.type.equals(type)
+ && otherStructField.containsNull == containsNull;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/StructType.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/StructType.java
index e3ffa3a521..881d451743 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/StructType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/StructType.java
@@ -17,6 +17,7 @@
package org.apache.doris.catalog;
+import org.apache.doris.thrift.TColumnType;
import org.apache.doris.thrift.TStructField;
import org.apache.doris.thrift.TTypeDesc;
import org.apache.doris.thrift.TTypeNode;
@@ -27,6 +28,7 @@ import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
+import com.google.gson.annotations.SerializedName;
import java.util.ArrayList;
import java.util.HashMap;
@@ -35,7 +37,14 @@ import java.util.HashMap;
* Describes a STRUCT type. STRUCT types have a list of named struct fields.
*/
public class StructType extends Type {
+
+ @SerializedName(value = "fieldMap")
private final HashMap<String, StructField> fieldMap = Maps.newHashMap();
+
+ //@SerializedName(value = "positionToField")
+ //private final HashMap<Integer, StructField> positionToField = Maps.newHashMap();
+
+ @SerializedName(value = "fields")
private final ArrayList<StructField> fields;
public StructType(ArrayList<StructField> fields) {
@@ -44,11 +53,12 @@ public class StructType extends Type {
for (int i = 0; i < this.fields.size(); ++i) {
this.fields.get(i).setPosition(i);
fieldMap.put(this.fields.get(i).getName().toLowerCase(), this.fields.get(i));
+ //positionToField.put(this.fields.get(i).getPosition(), this.fields.get(i));
}
}
public StructType() {
- fields = Lists.newArrayList();
+ this.fields = Lists.newArrayList();
}
@Override
@@ -74,10 +84,36 @@ public class StructType extends Type {
leftPadding, Joiner.on(",\n").join(fieldsSql), leftPadding);
}
+ public static boolean canCastTo(StructType type, StructType targetType) {
+ // TODO(xy) : support cast struct type
+ return false;
+ }
+
+ @Override
+ public boolean isSupported() {
+ for (StructField f : fields) {
+ if (!f.getType().isSupported()) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ @Override
+ public boolean supportSubType(Type subType) {
+ for (Type supportedType : Type.getStructSubTypes()) {
+ if (subType.getPrimitiveType() == supportedType.getPrimitiveType()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
public void addField(StructField field) {
field.setPosition(fields.size());
fields.add(field);
fieldMap.put(field.getName().toLowerCase(), field);
+ //positionToField.put(field.getPosition(), field);
}
public ArrayList<StructField> getFields() {
@@ -88,9 +124,41 @@ public class StructType extends Type {
return fieldMap.get(fieldName.toLowerCase());
}
+ //public StructField getField(int position) {
+ // return positionToField.get(position);
+ //}
+
public void clearFields() {
fields.clear();
fieldMap.clear();
+ //positionToField.clear();
+ }
+
+ @Override
+ public PrimitiveType getPrimitiveType() {
+ return PrimitiveType.STRUCT;
+ }
+
+ @Override
+ public boolean matchesType(Type t) {
+ if (equals(t)) {
+ return true;
+ }
+
+ if (t.isStructType()) {
+ return false;
+ }
+
+ if (fields.size() != ((StructType) t).getFields().size()) {
+ return false;
+ }
+
+ for (int i = 0; i < fields.size(); i++) {
+ if (!fields.get(i).matchesField(((StructType) t).getFields().get(i))) {
+ return false;
+ }
+ }
+ return true;
}
@Override
@@ -107,7 +175,7 @@ public class StructType extends Type {
TTypeNode node = new TTypeNode();
container.types.add(node);
Preconditions.checkNotNull(fields);
- Preconditions.checkNotNull(!fields.isEmpty());
+ Preconditions.checkState(!fields.isEmpty());
node.setType(TTypeNodeType.STRUCT);
node.setStructFields(new ArrayList<TStructField>());
for (StructField field : fields) {
@@ -119,4 +187,26 @@ public class StructType extends Type {
public String toString() {
return toSql(0);
}
+
+ @Override
+ public TColumnType toColumnTypeThrift() {
+ TColumnType thrift = new TColumnType();
+ thrift.type = PrimitiveType.STRUCT.toThrift();
+ return thrift;
+ }
+
+ @Override
+ public boolean isFixedLengthType() {
+ return false;
+ }
+
+ @Override
+ public boolean supportsTablePartitioning() {
+ return false;
+ }
+
+ @Override
+ public int getSlotSize() {
+ return PrimitiveType.STRUCT.getSlotSize();
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
index d80aa4fc22..45d88897a9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
@@ -112,6 +112,7 @@ public abstract class Type {
private static final ArrayList<ScalarType> numericDateTimeTypes;
private static final ArrayList<ScalarType> supportedTypes;
private static final ArrayList<Type> arraySubTypes;
+ private static final ArrayList<Type> structSubTypes;
private static final ArrayList<ScalarType> trivialTypes;
static {
@@ -173,6 +174,10 @@ public abstract class Type {
arraySubTypes.add(CHAR);
arraySubTypes.add(VARCHAR);
arraySubTypes.add(STRING);
+
+ structSubTypes = Lists.newArrayList();
+ structSubTypes.add(INT);
+ structSubTypes.add(STRING);
}
public static ArrayList<ScalarType> getIntegerTypes() {
@@ -199,6 +204,17 @@ public abstract class Type {
return arraySubTypes;
}
+ public static ArrayList<Type> getStructSubTypes() {
+ return structSubTypes;
+ }
+
+ /**
+ * Return true if this is complex type and support subType
+ */
+ public boolean supportSubType(Type subType) {
+ return false;
+ }
+
/**
* The output of this is stored directly in the hive metastore as the column type.
* The string must match exactly.
@@ -381,7 +397,7 @@ public abstract class Type {
}
public boolean isCollectionType() {
- return isMapType() || isArrayType() || isMultiRowType();
+ return isMapType() || isArrayType() || isMultiRowType() || isStructType();
}
public boolean isMapType() {
@@ -506,6 +522,10 @@ public abstract class Type {
&& !sourceType.isNull()) {
// TODO: current not support cast any non-array type(except for null) to nested array type.
return false;
+ } else if (targetType.isStructType() && sourceType.isStringType()) {
+ return true;
+ } else if (sourceType.isStructType() && targetType.isStructType()) {
+ return StructType.canCastTo((StructType) sourceType, (StructType) targetType);
}
return sourceType.isNull() || sourceType.getPrimitiveType().isCharFamily();
}
@@ -828,7 +848,7 @@ public abstract class Type {
}
Pair<Type, Integer> res = fromThrift(col, tmpNodeIdx);
tmpNodeIdx = res.second.intValue();
- structFields.add(new StructField(name, res.first, comment));
+ structFields.add(new StructField(name, res.first, comment, true));
}
type = new StructType(structFields);
break;
diff --git a/fe/fe-core/src/main/jflex/sql_scanner.flex b/fe/fe-core/src/main/jflex/sql_scanner.flex
index b66a91004a..c56743bd22 100644
--- a/fe/fe-core/src/main/jflex/sql_scanner.flex
+++ b/fe/fe-core/src/main/jflex/sql_scanner.flex
@@ -499,6 +499,8 @@ import org.apache.doris.qe.SqlModeHelper;
tokenIdMap.put(new Integer(SqlParserSymbols.RPAREN), ")");
tokenIdMap.put(new Integer(SqlParserSymbols.LBRACKET), "[");
tokenIdMap.put(new Integer(SqlParserSymbols.RBRACKET), "]");
+ tokenIdMap.put(new Integer(SqlParserSymbols.LBRACE), "{");
+ tokenIdMap.put(new Integer(SqlParserSymbols.RBRACE), "}");
tokenIdMap.put(new Integer(SqlParserSymbols.COLON), ":");
tokenIdMap.put(new Integer(SqlParserSymbols.SEMICOLON), ";");
tokenIdMap.put(new Integer(SqlParserSymbols.FLOATINGPOINT_LITERAL),
@@ -647,6 +649,8 @@ EndOfLineComment = "--" !({HintContent}|{ContainsLineTerminator}) {LineTerminato
";" { return newToken(SqlParserSymbols.SEMICOLON, null); }
"[" { return newToken(SqlParserSymbols.LBRACKET, null); }
"]" { return newToken(SqlParserSymbols.RBRACKET, null); }
+"{" { return newToken(SqlParserSymbols.LBRACE, null); }
+"}" { return newToken(SqlParserSymbols.RBRACE, null); }
"/" { return newToken(SqlParserSymbols.DIVIDE, null); }
"%" { return newToken(SqlParserSymbols.MOD, null); }
"+" { return newToken(SqlParserSymbols.ADD, null); }
diff --git a/gensrc/proto/types.proto b/gensrc/proto/types.proto
index 5686a6ac9d..dea020d0b4 100644
--- a/gensrc/proto/types.proto
+++ b/gensrc/proto/types.proto
@@ -39,6 +39,7 @@ message PScalarType {
message PStructField {
required string name = 1;
optional string comment = 2;
+ optional bool contains_null = 3;
};
message PTypeNode {
diff --git a/gensrc/thrift/Exprs.thrift b/gensrc/thrift/Exprs.thrift
index 230deb51ff..d25dce63a1 100644
--- a/gensrc/thrift/Exprs.thrift
+++ b/gensrc/thrift/Exprs.thrift
@@ -44,6 +44,7 @@ enum TExprNodeType {
INFO_FUNC,
FUNCTION_CALL,
ARRAY_LITERAL,
+ STRUCT_LITERAL,
// TODO: old style compute functions. this will be deprecated
COMPUTE_FUNCTION_CALL,
diff --git a/gensrc/thrift/Types.thrift b/gensrc/thrift/Types.thrift
index 589b95f77c..466d0d3b18 100644
--- a/gensrc/thrift/Types.thrift
+++ b/gensrc/thrift/Types.thrift
@@ -125,6 +125,7 @@ struct TScalarType {
struct TStructField {
1: required string name
2: optional string comment
+ 3: optional bool contains_null
}
struct TTypeNode {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/09: save
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3d3d92f6c87c8c8701cce45dd05045ac0a52c0e7
Author: xy720 <xu...@baidu.com>
AuthorDate: Thu Nov 17 19:15:50 2022 +0800
save
---
test_file | 1 -
1 file changed, 1 deletion(-)
diff --git a/test_file b/test_file
deleted file mode 100644
index 63d3c2d75e..0000000000
--- a/test_file
+++ /dev/null
@@ -1 +0,0 @@
-just for test
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 07/09: [feature](struct-type) adjust the vexpr for struct type (#15669)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0d4da4f5f7797c53c9a2389519d5a81073baf82c
Author: carlvinhust2012 <hu...@126.com>
AuthorDate: Fri Jan 13 17:14:11 2023 +0800
[feature](struct-type) adjust the vexpr for struct type (#15669)
---
be/src/vec/columns/column_struct.cpp | 7 --
be/src/vec/columns/column_struct.h | 1 -
be/src/vec/data_types/data_type_factory.hpp | 8 ++
be/src/vec/data_types/data_type_struct.cpp | 99 +++++++++++++++++++---
be/src/vec/data_types/data_type_struct.h | 3 +
be/src/vec/functions/function_cast.h | 13 ++-
.../java/org/apache/doris/analysis/CastExpr.java | 8 ++
.../java/org/apache/doris/analysis/TypeDef.java | 2 +-
.../main/java/org/apache/doris/catalog/Column.java | 1 +
.../main/java/org/apache/doris/catalog/Type.java | 14 ++-
10 files changed, 134 insertions(+), 22 deletions(-)
diff --git a/be/src/vec/columns/column_struct.cpp b/be/src/vec/columns/column_struct.cpp
index d51bebb536..afef00c314 100644
--- a/be/src/vec/columns/column_struct.cpp
+++ b/be/src/vec/columns/column_struct.cpp
@@ -22,13 +22,6 @@
namespace doris::vectorized {
-namespace ErrorCodes {
-extern const int ILLEGAL_COLUMN;
-extern const int NOT_IMPLEMENTED;
-extern const int CANNOT_INSERT_VALUE_OF_DIFFERENT_SIZE_INTO_TUPLE;
-extern const int LOGICAL_ERROR;
-} // namespace ErrorCodes
-
std::string ColumnStruct::get_name() const {
std::stringstream res;
res << "Struct(";
diff --git a/be/src/vec/columns/column_struct.h b/be/src/vec/columns/column_struct.h
index a66d91f3e0..393b2c275f 100644
--- a/be/src/vec/columns/column_struct.h
+++ b/be/src/vec/columns/column_struct.h
@@ -103,7 +103,6 @@ public:
bool can_be_inside_nullable() const override { return true; }
MutableColumnPtr clone_empty() const override;
MutableColumnPtr clone_resized(size_t size) const override;
-
size_t size() const override { return columns.at(0)->size(); }
Field operator[](size_t n) const override;
diff --git a/be/src/vec/data_types/data_type_factory.hpp b/be/src/vec/data_types/data_type_factory.hpp
index d46dd3435d..9bc5e20d5c 100644
--- a/be/src/vec/data_types/data_type_factory.hpp
+++ b/be/src/vec/data_types/data_type_factory.hpp
@@ -112,6 +112,14 @@ public:
return entity.second;
}
}
+ if (type_ptr->get_type_id() == TypeIndex::Struct) {
+ DataTypeFactory::instance().register_data_type(type_ptr->get_name(), type_ptr);
+ for (const auto& entity : _invert_data_type_map) {
+ if (entity.first->equals(*type_ptr)) {
+ return entity.second;
+ }
+ }
+ }
return _empty_string;
}
diff --git a/be/src/vec/data_types/data_type_struct.cpp b/be/src/vec/data_types/data_type_struct.cpp
index 4364f5c34f..87b5c4f110 100644
--- a/be/src/vec/data_types/data_type_struct.cpp
+++ b/be/src/vec/data_types/data_type_struct.cpp
@@ -22,17 +22,6 @@
namespace doris::vectorized {
-namespace ErrorCodes {
-extern const int BAD_ARGUMENTS;
-extern const int DUPLICATE_COLUMN;
-extern const int EMPTY_DATA_PASSED;
-extern const int NOT_FOUND_COLUMN_IN_BLOCK;
-extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
-extern const int SIZES_OF_COLUMNS_IN_TUPLE_DOESNT_MATCH;
-extern const int ILLEGAL_INDEX;
-extern const int LOGICAL_ERROR;
-} // namespace ErrorCodes
-
DataTypeStruct::DataTypeStruct(const DataTypes& elems_)
: elems(elems_), have_explicit_names(false) {
/// Automatically assigned names in form of '1', '2', ...
@@ -91,6 +80,94 @@ std::string DataTypeStruct::do_get_name() const {
return s.str();
}
+Status DataTypeStruct::from_string(ReadBuffer& rb, IColumn* column) const {
+ DCHECK(!rb.eof());
+ auto* struct_column = assert_cast<ColumnStruct*>(column);
+
+ if (*rb.position() != '{') {
+ return Status::InvalidArgument("Struct does not start with '{' character, found '{}'",
+ *rb.position());
+ }
+ if (rb.count() < 2 || *(rb.end() - 1) != '}') {
+ return Status::InvalidArgument("Struct does not end with '}' character, found '{}'",
+ *(rb.end() - 1));
+ }
+
+ // here need handle the empty struct '{}'
+ if (rb.count() == 2) {
+ return Status::OK();
+ }
+
+ ++rb.position();
+ std::vector<ReadBuffer> field_rbs;
+ field_rbs.reserve(elems.size());
+
+ // here get the value "jack" and 20 from {"name":"jack","age":20}
+ while (!rb.eof()) {
+ size_t field_len = 0;
+ auto start = rb.position();
+ while (!rb.eof() && *start != ',' && *start != '}') {
+ field_len++;
+ start++;
+ }
+ if (field_len >= rb.count()) {
+ return Status::InvalidArgument("Invalid Length");
+ }
+ ReadBuffer field_rb(rb.position(), field_len);
+ size_t len = 0;
+ auto start_rb = field_rb.position();
+ while (!field_rb.eof() && *start_rb != ':') {
+ len++;
+ start_rb++;
+ }
+ ReadBuffer field(field_rb.position() + len + 1, field_rb.count() - len - 1);
+
+ if (field.count() < 2 || *field.position() != '"' || *field.end() != '"') {
+ field_rbs.push_back(field);
+ } else {
+ ReadBuffer field_has_quote(field.position() + 1, field.count() - 2);
+ field_rbs.push_back(field_has_quote);
+ }
+
+ rb.position() += field_len + 1;
+ }
+
+ for (size_t idx = 0; idx < elems.size(); idx++) {
+ elems[idx]->from_string(field_rbs[idx], &struct_column->get_column(idx));
+ }
+
+ return Status::OK();
+}
+
+std::string DataTypeStruct::to_string(const IColumn& column, size_t row_num) const {
+ auto ptr = column.convert_to_full_column_if_const();
+ auto& struct_column = assert_cast<const ColumnStruct&>(*ptr.get());
+
+ std::stringstream ss;
+ ss << "<";
+ for (size_t idx = 0; idx < elems.size(); idx++) {
+ if (idx != 0) {
+ ss << ", ";
+ }
+ ss << elems[idx]->to_string(struct_column.get_column(idx), row_num);
+ }
+ ss << ">";
+ return ss.str();
+}
+
+void DataTypeStruct::to_string(const IColumn& column, size_t row_num, BufferWritable& ostr) const {
+ auto ptr = column.convert_to_full_column_if_const();
+ auto& struct_column = assert_cast<const ColumnStruct&>(*ptr.get());
+ ostr.write("<", 1);
+ for (size_t idx = 0; idx < elems.size(); idx++) {
+ if (idx != 0) {
+ ostr.write(", ", 2);
+ }
+ elems[idx]->to_string(struct_column.get_column(idx), row_num, ostr);
+ }
+ ostr.write(">", 1);
+}
+
static inline IColumn& extract_element_column(IColumn& column, size_t idx) {
return assert_cast<ColumnStruct&>(column).get_column(idx);
}
diff --git a/be/src/vec/data_types/data_type_struct.h b/be/src/vec/data_types/data_type_struct.h
index 88255de38c..47cfaab1a3 100644
--- a/be/src/vec/data_types/data_type_struct.h
+++ b/be/src/vec/data_types/data_type_struct.h
@@ -96,6 +96,9 @@ public:
const char* deserialize(const char* buf, IColumn* column, int be_exec_version) const override;
void to_pb_column_meta(PColumnMeta* col_meta) const override;
+ Status from_string(ReadBuffer& rb, IColumn* column) const override;
+ std::string to_string(const IColumn& column, size_t row_num) const override;
+ void to_string(const IColumn& column, size_t row_num, BufferWritable& ostr) const override;
// bool is_parametric() const { return true; }
// SerializationPtr do_get_default_serialization() const override;
// SerializationPtr get_serialization(const SerializationInfo& info) const override;
diff --git a/be/src/vec/functions/function_cast.h b/be/src/vec/functions/function_cast.h
index d300c09a76..ec4608c115 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -1541,6 +1541,16 @@ private:
return &ConvertImplGenericToJsonb::execute;
}
}
+ // check struct value type and get to_type value
+ // TODO: need handle another type to cast struct
+ WrapperType create_struct_wrapper(const DataTypePtr& from_type,
+ const DataTypeStruct& to_type) const {
+ switch (from_type->get_type_id()) {
+ case TypeIndex::String:
+ default:
+ return &ConvertImplGenericFromString<ColumnString>::execute;
+ }
+ }
WrapperType prepare_unpack_dictionaries(FunctionContext* context, const DataTypePtr& from_type,
const DataTypePtr& to_type) const {
@@ -1712,6 +1722,8 @@ private:
case TypeIndex::Array:
return create_array_wrapper(context, from_type,
static_cast<const DataTypeArray&>(*to_type));
+ case TypeIndex::Struct:
+ return create_struct_wrapper(from_type, static_cast<const DataTypeStruct&>(*to_type));
default:
break;
}
@@ -1757,7 +1769,6 @@ protected:
// TODO(xy): support return struct type for factory
auto type = DataTypeFactory::instance().get(type_col->get_value<String>());
DCHECK(type != nullptr);
-
bool need_to_be_nullable = false;
// 1. from_type is nullable
need_to_be_nullable |= arguments[0].type->is_nullable();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
index ab5bc992f8..77975c1eb9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CastExpr.java
@@ -318,9 +318,17 @@ public class CastExpr extends Expr {
type, Function.NullableMode.ALWAYS_NULLABLE,
Lists.newArrayList(Type.VARCHAR), false,
"doris::CastFunctions::cast_to_array_val", null, null, true);
+ } else if (type.isStructType()) {
+ fn = ScalarFunction.createBuiltin(getFnName(Type.STRUCT),
+ type, Function.NullableMode.ALWAYS_NULLABLE,
+ Lists.newArrayList(Type.VARCHAR), false,
+ "doris::CastFunctions::cast_to_struct_val", null, null, true);
}
if (fn == null) {
+ if (type.isStructType() && childType.isStringType()) {
+ return;
+ }
if (childType.isNull() && Type.canCastTo(childType, type)) {
return;
} else {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
index 5690c115c1..c89c659207 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TypeDef.java
@@ -129,7 +129,7 @@ public class TypeDef implements ParseNode {
// check whether the sub-type is supported
if (!parent.supportSubType(child)) {
throw new AnalysisException(
- parent.getPrimitiveType() + "unsupported sub-type: " + child.toSql());
+ parent.getPrimitiveType() + " unsupported sub-type: " + child.toSql());
}
if (child.getPrimitiveType().isStringType() && !child.isLengthSet()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
index 9efe901907..7b0de9b7e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java
@@ -59,6 +59,7 @@ public class Column implements Writable, GsonPostProcessable {
public static final String DELETE_SIGN = "__DORIS_DELETE_SIGN__";
public static final String SEQUENCE_COL = "__DORIS_SEQUENCE_COL__";
private static final String COLUMN_ARRAY_CHILDREN = "item";
+ private static final String COLUMN_STRUCT_CHILDREN = "field";
public static final int COLUMN_UNIQUE_ID_INIT_VALUE = -1;
@SerializedName(value = "name")
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
index 45d88897a9..587266926c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java
@@ -176,8 +176,20 @@ public abstract class Type {
arraySubTypes.add(STRING);
structSubTypes = Lists.newArrayList();
- structSubTypes.add(INT);
+ structSubTypes.addAll(numericTypes);
+ structSubTypes.add(BOOLEAN);
+ structSubTypes.add(VARCHAR);
structSubTypes.add(STRING);
+ structSubTypes.add(CHAR);
+ structSubTypes.add(DATE);
+ structSubTypes.add(DATETIME);
+ structSubTypes.add(DATEV2);
+ structSubTypes.add(DATETIMEV2);
+ structSubTypes.add(TIME);
+ structSubTypes.add(TIMEV2);
+ structSubTypes.add(DECIMAL32);
+ structSubTypes.add(DECIMAL64);
+ structSubTypes.add(DECIMAL128);
}
public static ArrayList<ScalarType> getIntegerTypes() {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/09: [test] (test) test modify branch (#14375)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e2c95a9098b7e485bb4b83d0dba04a31de204099
Author: xy720 <22...@users.noreply.github.com>
AuthorDate: Thu Nov 17 19:27:40 2022 +0800
[test] (test) test modify branch (#14375)
---
README.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/README.md b/README.md
index 57b1b16580..e155ad58a8 100644
--- a/README.md
+++ b/README.md
@@ -236,3 +236,4 @@ some Doris features to be complied with Apache 2.0 License. For details, refer t
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 06/09: [fix](struct-type) struct create fix and add more struct codes (#15879)
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 22b88667539108b29bc5b2ded932697908a64a0d
Author: camby <10...@qq.com>
AuthorDate: Fri Jan 13 11:24:09 2023 +0800
[fix](struct-type) struct create fix and add more struct codes (#15879)
Fix the dead loop when create column struct
Co-authored-by: cambyzju <zh...@baidu.com>
---
be/src/runtime/types.cpp | 29 +++++++++++++++++--
be/src/vec/columns/column_struct.cpp | 38 ++++++++++---------------
be/src/vec/columns/column_struct.h | 18 +++++-------
be/src/vec/data_types/data_type.cpp | 2 ++
be/src/vec/data_types/data_type_factory.cpp | 14 ++++++++++
be/src/vec/data_types/data_type_struct.cpp | 43 +++++++++++++++++++++++++++++
be/src/vec/data_types/data_type_struct.h | 19 ++++---------
7 files changed, 111 insertions(+), 52 deletions(-)
diff --git a/be/src/runtime/types.cpp b/be/src/runtime/types.cpp
index f26b1dcbce..45b8f8e5f5 100644
--- a/be/src/runtime/types.cpp
+++ b/be/src/runtime/types.cpp
@@ -150,7 +150,7 @@ void TypeDescriptor::to_thrift(TTypeDesc* thrift_type) const {
}
void TypeDescriptor::to_protobuf(PTypeDesc* ptype) const {
- DCHECK(!is_complex_type() || type == TYPE_ARRAY)
+ DCHECK(!is_complex_type() || type == TYPE_ARRAY || type == TYPE_STRUCT)
<< "Don't support complex type now, type=" << type;
auto node = ptype->add_types();
node->set_type(TTypeNodeType::SCALAR);
@@ -170,8 +170,18 @@ void TypeDescriptor::to_protobuf(PTypeDesc* ptype) const {
for (const TypeDescriptor& child : children) {
child.to_protobuf(ptype);
}
+ } else if (type == TYPE_STRUCT) {
+ node->set_type(TTypeNodeType::STRUCT);
+ DCHECK_EQ(field_names.size(), contains_nulls.size());
+ for (size_t i = 0; i < field_names.size(); ++i) {
+ auto field = node->add_struct_fields();
+ field->set_name(field_names[i]);
+ field->set_contains_null(contains_nulls[i]);
+ }
+ for (const TypeDescriptor& child : children) {
+ child.to_protobuf(ptype);
+ }
}
- // TODO(xy): support struct
}
TypeDescriptor::TypeDescriptor(const google::protobuf::RepeatedPtrField<PTypeNode>& types, int* idx)
@@ -213,7 +223,20 @@ TypeDescriptor::TypeDescriptor(const google::protobuf::RepeatedPtrField<PTypeNod
children.push_back(TypeDescriptor(types, idx));
break;
}
- // TODO(xy): support struct
+ case TTypeNodeType::STRUCT: {
+ type = TYPE_STRUCT;
+ size_t children_size = node.struct_fields_size();
+ for (size_t i = 0; i < children_size; ++i) {
+ const auto& field = node.struct_fields(i);
+ field_names.push_back(field.name());
+ contains_nulls.push_back(field.contains_null());
+ }
+ for (size_t i = 0; i < children_size; ++i) {
+ ++(*idx);
+ children.push_back(TypeDescriptor(types, idx));
+ }
+ break;
+ }
default:
DCHECK(false) << node.type();
}
diff --git a/be/src/vec/columns/column_struct.cpp b/be/src/vec/columns/column_struct.cpp
index c7e5c23d2a..d51bebb536 100644
--- a/be/src/vec/columns/column_struct.cpp
+++ b/be/src/vec/columns/column_struct.cpp
@@ -54,43 +54,25 @@ ColumnStruct::ColumnStruct(MutableColumns&& mutable_columns) {
}
}
-ColumnStruct::ColumnStruct(Columns&& columns) {
- columns.reserve(columns.size());
- for (auto& column : columns) {
- if (is_column_const(*column)) {
- LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
- }
- columns.push_back(std::move(column));
- }
-}
-
-ColumnStruct::ColumnStruct(TupleColumns&& tuple_columns) {
- columns.reserve(tuple_columns.size());
- for (auto& column : tuple_columns) {
- if (is_column_const(*column)) {
- LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
- }
- columns.push_back(std::move(column));
- }
-}
-
-ColumnStruct::Ptr ColumnStruct::create(Columns& columns) {
+ColumnStruct::Ptr ColumnStruct::create(const Columns& columns) {
for (const auto& column : columns) {
if (is_column_const(*column)) {
LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
}
}
- auto column_struct = ColumnStruct::create(columns);
+ auto column_struct = ColumnStruct::create(MutableColumns());
+ column_struct->columns.assign(columns.begin(), columns.end());
return column_struct;
}
-ColumnStruct::Ptr ColumnStruct::create(TupleColumns& tuple_columns) {
+ColumnStruct::Ptr ColumnStruct::create(const TupleColumns& tuple_columns) {
for (const auto& column : tuple_columns) {
if (is_column_const(*column)) {
LOG(FATAL) << "ColumnStruct cannot have ColumnConst as its element";
}
}
- auto column_struct = ColumnStruct::create(tuple_columns);
+ auto column_struct = ColumnStruct::create(MutableColumns());
+ column_struct->columns = tuple_columns;
return column_struct;
}
@@ -221,6 +203,14 @@ void ColumnStruct::update_hash_with_value(size_t n, SipHash& hash) const {
// }
// }
+void ColumnStruct::insert_indices_from(const IColumn& src, const int* indices_begin,
+ const int* indices_end) {
+ const ColumnStruct& src_concrete = assert_cast<const ColumnStruct&>(src);
+ for (size_t i = 0; i < columns.size(); ++i) {
+ columns[i]->insert_indices_from(src_concrete.get_column(i), indices_begin, indices_end);
+ }
+}
+
// const char * ColumnStruct::skip_serialized_in_arena(const char * pos) const {
// for (const auto & column : columns) {
// pos = column->skip_serialized_in_arena(pos);
diff --git a/be/src/vec/columns/column_struct.h b/be/src/vec/columns/column_struct.h
index 75dade874d..a66d91f3e0 100644
--- a/be/src/vec/columns/column_struct.h
+++ b/be/src/vec/columns/column_struct.h
@@ -79,8 +79,6 @@ private:
template <bool positive>
struct Less;
- ColumnStruct(Columns&& columns);
- ColumnStruct(TupleColumns&& tuple_columns);
explicit ColumnStruct(MutableColumns&& mutable_columns);
ColumnStruct(const ColumnStruct&) = default;
@@ -89,14 +87,14 @@ public:
* Use IColumn::mutate in order to make mutable column and mutate shared nested columns.
*/
using Base = COWHelper<IColumn, ColumnStruct>;
- static Ptr create(Columns& columns);
- static Ptr create(MutableColumns& columns);
- static Ptr create(TupleColumns& columns);
+ static Ptr create(const Columns& columns);
+ static Ptr create(const TupleColumns& columns);
static Ptr create(Columns&& arg) { return create(arg); }
- template <typename... Args>
- static MutablePtr create(Args&&... args) {
- return Base::create(std::forward<Args>(args)...);
+ template <typename Arg,
+ typename = typename std::enable_if<std::is_rvalue_reference<Arg&&>::value>::type>
+ static MutablePtr create(Arg&& arg) {
+ return Base::create(std::forward<Arg>(arg));
}
std::string get_name() const override;
@@ -131,9 +129,7 @@ public:
// void update_hash_fast(SipHash & hash) const override;
void insert_indices_from(const IColumn& src, const int* indices_begin,
- const int* indices_end) override {
- LOG(FATAL) << "insert_indices_from not implemented";
- }
+ const int* indices_end) override;
void get_permutation(bool reverse, size_t limit, int nan_direction_hint,
Permutation& res) const override {
diff --git a/be/src/vec/data_types/data_type.cpp b/be/src/vec/data_types/data_type.cpp
index 7a0d67d4a4..360fa48a98 100644
--- a/be/src/vec/data_types/data_type.cpp
+++ b/be/src/vec/data_types/data_type.cpp
@@ -145,6 +145,8 @@ PGenericType_TypeId IDataType::get_pdata_type(const IDataType* data_type) {
return PGenericType::HLL;
case TypeIndex::Array:
return PGenericType::LIST;
+ case TypeIndex::Struct:
+ return PGenericType::STRUCT;
case TypeIndex::FixedLengthObject:
return PGenericType::FIXEDLENGTHOBJECT;
case TypeIndex::JSONB:
diff --git a/be/src/vec/data_types/data_type_factory.cpp b/be/src/vec/data_types/data_type_factory.cpp
index 7eec1eb2fd..553f117a48 100644
--- a/be/src/vec/data_types/data_type_factory.cpp
+++ b/be/src/vec/data_types/data_type_factory.cpp
@@ -338,6 +338,20 @@ DataTypePtr DataTypeFactory::create_data_type(const PColumnMeta& pcolumn) {
case PGenericType::FIXEDLENGTHOBJECT:
nested = std::make_shared<DataTypeFixedLengthObject>();
break;
+ case PGenericType::STRUCT: {
+ size_t col_size = pcolumn.children_size();
+ DCHECK(col_size >= 1);
+ DataTypes dataTypes;
+ Strings names;
+ dataTypes.reserve(col_size);
+ names.reserve(col_size);
+ for (size_t i = 0; i < col_size; i++) {
+ dataTypes.push_back(create_data_type(pcolumn.children(i)));
+ names.push_back(pcolumn.name());
+ }
+ nested = std::make_shared<DataTypeStruct>(dataTypes, names);
+ break;
+ }
default: {
LOG(FATAL) << fmt::format("Unknown data type: {}", pcolumn.type());
return nullptr;
diff --git a/be/src/vec/data_types/data_type_struct.cpp b/be/src/vec/data_types/data_type_struct.cpp
index 8fd2179bfa..4364f5c34f 100644
--- a/be/src/vec/data_types/data_type_struct.cpp
+++ b/be/src/vec/data_types/data_type_struct.cpp
@@ -232,6 +232,49 @@ String DataTypeStruct::get_name_by_position(size_t i) const {
return names[i - 1];
}
+int64_t DataTypeStruct::get_uncompressed_serialized_bytes(const IColumn& column,
+ int be_exec_version) const {
+ auto ptr = column.convert_to_full_column_if_const();
+ const auto& struct_column = assert_cast<const ColumnStruct&>(*ptr.get());
+ DCHECK(elems.size() == struct_column.tuple_size());
+
+ int64_t bytes = 0;
+ for (size_t i = 0; i < elems.size(); ++i) {
+ bytes += elems[i]->get_uncompressed_serialized_bytes(struct_column.get_column(i),
+ be_exec_version);
+ }
+ return bytes;
+}
+
+char* DataTypeStruct::serialize(const IColumn& column, char* buf, int be_exec_version) const {
+ auto ptr = column.convert_to_full_column_if_const();
+ const auto& struct_column = assert_cast<const ColumnStruct&>(*ptr.get());
+ DCHECK(elems.size() == struct_column.tuple_size());
+
+ for (size_t i = 0; i < elems.size(); ++i) {
+ buf = elems[i]->serialize(struct_column.get_column(i), buf, be_exec_version);
+ }
+ return buf;
+}
+
+const char* DataTypeStruct::deserialize(const char* buf, IColumn* column,
+ int be_exec_version) const {
+ auto* struct_column = assert_cast<ColumnStruct*>(column);
+ DCHECK(elems.size() == struct_column->tuple_size());
+
+ for (size_t i = 0; i < elems.size(); ++i) {
+ buf = elems[i]->deserialize(buf, &struct_column->get_column(i), be_exec_version);
+ }
+ return buf;
+}
+
+void DataTypeStruct::to_pb_column_meta(PColumnMeta* col_meta) const {
+ IDataType::to_pb_column_meta(col_meta);
+ for (size_t i = 0; i < elems.size(); ++i) {
+ elems[i]->to_pb_column_meta(col_meta->add_children());
+ }
+}
+
bool DataTypeStruct::text_can_contain_only_valid_utf8() const {
return std::all_of(elems.begin(), elems.end(),
[](auto&& elem) { return elem->text_can_contain_only_valid_utf8(); });
diff --git a/be/src/vec/data_types/data_type_struct.h b/be/src/vec/data_types/data_type_struct.h
index 9405544b76..88255de38c 100644
--- a/be/src/vec/data_types/data_type_struct.h
+++ b/be/src/vec/data_types/data_type_struct.h
@@ -90,20 +90,11 @@ public:
std::optional<size_t> try_get_position_by_name(const String& name) const;
String get_name_by_position(size_t i) const;
- [[noreturn]] int64_t get_uncompressed_serialized_bytes(const IColumn& column,
- int be_exec_version) const override {
- LOG(FATAL) << "get_uncompressed_serialized_bytes not implemented";
- }
-
- [[noreturn]] char* serialize(const IColumn& column, char* buf,
- int be_exec_version) const override {
- LOG(FATAL) << "serialize not implemented";
- }
-
- [[noreturn]] const char* deserialize(const char* buf, IColumn* column,
- int be_exec_version) const override {
- LOG(FATAL) << "serialize not implemented";
- }
+ int64_t get_uncompressed_serialized_bytes(const IColumn& column,
+ int be_exec_version) const override;
+ char* serialize(const IColumn& column, char* buf, int be_exec_version) const override;
+ const char* deserialize(const char* buf, IColumn* column, int be_exec_version) const override;
+ void to_pb_column_meta(PColumnMeta* col_meta) const override;
// bool is_parametric() const { return true; }
// SerializationPtr do_get_default_serialization() const override;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/09: test
Posted by xu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xuyang pushed a commit to branch struct-type
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d2d0960a3dcac19c889dfd3ef768bc0d5eb906d3
Author: xy720 <xu...@baidu.com>
AuthorDate: Thu Nov 17 19:01:17 2022 +0800
test
---
test_file | 1 +
1 file changed, 1 insertion(+)
diff --git a/test_file b/test_file
new file mode 100644
index 0000000000..63d3c2d75e
--- /dev/null
+++ b/test_file
@@ -0,0 +1 @@
+just for test
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org