You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/02/10 07:20:38 UTC
[GitHub] [incubator-doris] chaoyli opened a new issue #2872: Release Notes
0.12.0
chaoyli opened a new issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872
Release Notes 0.12.0
# New Features
## Segment V2
We have re-implemented the underlying data storage format, using `Page-Based` logical concepts to organize different data types or index types. We call this storage format `Segment V2`.
Based on the new storage format, we have introduced a series of new features, including dictionary compression and bitmap indexing, to improve Doris's read and write performance.
[#1305]
## Bitmap Index
On the basis of Segment V2, Doris implements the function of adding bitmap indexes to the specified columns. This feature significantly enhances Doris's query performance for any combination of multi-dimensional scenarios.
[#2487] [#2050] [#2319]
## Spark on Doris
With spark-doris-connector, users can now query data stored in Doris directly through Spark. Better meet users' federal query needs.
[#2228]
## Support Loading ORC format file
Supports reading and loading ORC format files stored on HDFS.
[#2554]
## Create partition automatically
[Dynamic Partition] Support for automatically adding partitions
Doris can now automatically create partitions through scheduled tasks, reducing user maintenance costs for partition operations.
[#2262]
## Grouping Set SQL syntax
Supports `Grouping Set` SQL syntax. Can further reduce the complexity of writing SQL for data analysts.
[#2293]
# Enhancement
## Doris on ES Performance Enhancement
Significantly enhanced query performance for Doris on Es. Doris on ES is probably the best open source "SQL on ES" solution.
[#2237] [#2575]
## Bitmap aggregate type support BIGINT
The integer type supported by the Bitmap aggregation method has been extended from 32 bits to 64 bits. Further strengthen the scope of application of Doris's precise deduplication function.
[#2772]
## Supports conversion of more column types
Now Doris support conversion of more column types by altering table.
Float -> Double
Datetime <-> Date
Int -> Date
Varchar -> TINTINT/SMALLINT/INT/BIGINT/LARGEINT/FLOAT/DOUBLE/DATE
[#2310] [#2393] [#2481] [#2489] [#2501]
## NIO-based MySQL protocol connection layer
A new MySQL protocol connection layer based on NIO is implemented, which can support highly concurrent connection requests.
[Experimental, enable it by setting FE config `mysql_service_nio_enabled` to true]
[#2603]
## More convenient information viewing
Now we support adding filter conditions for following stmts:
* SHOW ALTER TABLE [#2380]
* SHOW PARTITIONS [#2553]
* SHOW FUNCTIONS [#2880]
# New Built-in Functions
* `bitmap_or` [#2649]
* `bitmap_and` [#2649]
* `bitamp_to_string` [#2731]
* `bitmap_from_string` [#2731]
* `bitmap_contains` [#2752]
* `bitmap_has_any` [#2752]
* `bitmap_union_count` [#2425]
* `bitmap_hash` [#2439]
* `intersect_count` [#2418]
* `ends_with` [#2746]
* `curdate` [#2521]
# Other
* Support aggregation type of REPLACE_IF_NOT_NULL [#2127]
* Introduce GSON library to serialize the meta data in FE [#2343]
* Support correlated non-scalar subquery [#2468]
* Support `ALTER VIEW` stmt [#2522]
* Support `replication_num` setting for table level [#2737]
* SupportBroker reading config from hdfs-site.xml [#2148]
* Support `date_add` function for partition pruning [#2154]
* Optimize compaction strategy of tablet on BE [#2473]
* Support decompressing csv file with deflate format in hdfs broker load [#2583]
* Support sql mode [2083]
* Modify the authorization checking logic [#2372]
* Improve SkipList memory usage tracking [#2359]
## Thirdparty
* Bump Apache Maven version to 3.6.3 in Docker dev image [#2395]
* Bump JMockit version to 1.48 [#2423]
* Bump Apache Arrow version to 0.15.1 [#2657] [#2769]
# API Change
## Default Table Storage Model
Change default storage model from `AGGREGATE KEY` to `DUPLICATE KEY`
[#2318]
## Forbid Cluster Related Operations
Forbid cluster related operations:
* create/drop cluster
* add free backend/add backend to cluster/decommission cluster balance
* change the backends num of cluster
* link/migration db
[#2429]
# Credits
Thanks to everyone who contributed to this release!
@EmmyMiao87
@HangyuanLiu
@HappenLee
@IanWatsons
@JustinTsui
@Lishi-hub
@MLikeWater
@Seaven
@WingsGo
@Wsyzxxn
@Youngwb
@blackfox1983
@caiconghui
@chaoyli
@firetree01
@frwrdt
@gaodayue
@imay
@infearOnTheWay
@kangkaisen
@kangpinghuang
@landon-dai
@lingbin
@liutang123
@lshmouse
@marising
@morningman
@shengyunyao
@vagetablechicken
@vinson0526
@wangbo
@wkhappy1
@wubiaoi
@wutiangan
@wuyunfeng
@xhg-tech
@xy720
@yangzhg
@yuanfeng0905
@zhouhaibing089
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on issue #2872: Release
Notes 0.12.0
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-601504895
Support materialized view instead of Support aggregated materialized view framework
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on issue #2872: Release Notes
0.12.0
Posted by GitBox <gi...@apache.org>.
gaodayue commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-584629275
> What do you mean incompatible ? Why there is an incompatible feature on going?
It means incompatible with the segment_v2 file format in master branch. It's OK because segment_v2 is not released yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2872: Release Notes
0.12.0
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-601503965
The incompatibility has been solved. Restart release 0.12.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangkaisen commented on issue #2872: Release
Notes 0.12.0
Posted by GitBox <gi...@apache.org>.
kangkaisen commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-583997030
API change should include https://github.com/apache/incubator-doris/pull/2256
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #2872: Release
Notes 0.12.0
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-602993818
> Is it a bug or a feature? If it's not a bug, we'd better include this in the `API Change` of release note.
`SHOW PARTITIONS` is now supporting `ORDER BY` clause.
But I think this should be changed back to the origin to avoid your problem.
Could you submit a PR for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on issue #2872: Release Notes
0.12.0
Posted by GitBox <gi...@apache.org>.
gaodayue commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-602992980
I just notice that the default result ordering of SHOW PARTITIONS statements is changed from `ascending PartitionId` in 0.11 to `descending PartitionId`. This causes some of our existing tools to break because they relies on the ascending behavior of 0.11.
Is it a bug or a feature? If it's not a bug, we'd better include this in the `API Change` of release note.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on issue #2872: Release Notes
0.12.0
Posted by GitBox <gi...@apache.org>.
gaodayue commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-584063379
The ongoing work to support complex data type may introduce incompatible changes to V2 format, so the release should pause until we finalized V2 format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on issue #2872: Release Notes
0.12.0
Posted by GitBox <gi...@apache.org>.
gaodayue commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-603016690
> But I think this should be changed back to the origin to avoid your problem.
Could you submit a PR for this?
Sure, I submitted #3184 for it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 edited a comment on issue #2872:
Release Notes 0.12.0
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 edited a comment on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-601504895
@chaoyli Support materialized view on duplicate mode instead of Support aggregated materialized view framework
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on issue #2872: Release
Notes 0.12.0
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-601505227
@chaoyli Please include the new feature of query in tpc-ds. The feature list is https://github.com/apache/incubator-doris/projects/6#card-34831609
The grouping set, sql mode, intersect, except, non-scalar subquery
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #2872: Release Notes
0.12.0
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-601506515
> @chaoyli Please include the new feature of query in tpc-ds. The feature list is https://github.com/apache/incubator-doris/projects/6#card-34831609
>
> The grouping set, sql mode, intersect, except, non-scalar subquery
all have added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #2872: Release
Notes 0.12.0
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #2872: Release Notes 0.12.0
URL: https://github.com/apache/incubator-doris/issues/2872#issuecomment-584169634
> The ongoing work to support complex data type may introduce incompatible changes to V2 format, so the release should pause until we finalized V2 format.
What do you mean `incompatible` ? Why there is an incompatible feature on going?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org