You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@doris.apache.org by GuoLei Yi <yi...@gmail.com> on 2022/10/09 03:52:16 UTC

Discussion about Doris 1.1.3

In this release, Doris Team has fixed more than 80 issues or performance
improvement since 1.1.2. This release is a bugfix release on 1.1 and all
users are encouraged to upgrade to this release.

I have created an issue about this release
https://github.com/apache/doris/issues/13199, please modify this issue
directly if I missed some important bugs for features.
Features

   -

   Support escape identifiers for sqlserver and postgresql in ODBC table.
   -

   Could use Parquet as output file format.

Improvements

   -

   Optimize flush policy to avoid small segments [Enhancement](load)
   optimize flush policy to avoid small segments #12706
   <https://github.com/apache/doris/pull/12706>
   -

   Refactor runtime filter to reduce the prepare time.
[improvement](runtimefilter)
   shorter time prepare consumes #13127
   <https://github.com/apache/doris/pull/13127>
   -

   Lots of memory control related issues during query or load process.
[fix](memtracker)
   Remove consume mem tracker GC and fix mem limit exceed error format
    #12682 <https://github.com/apache/doris/pull/12682> [fix](memtracker)
   Process physical mem check does not include tc/jemalloc allocator cache
    #12688 <https://github.com/apache/doris/pull/12688> [fix](memtracker)
   Fix memory leak by calling boost::stacktrace in mem hook #12708
   <https://github.com/apache/doris/pull/12708> [Bugfix](mem) Fix memory
   limit check may overflow #12776
   <https://github.com/apache/doris/pull/12776> [fix](memtracker) Fix
   thread mem tracker try consume accuracy #12782
   <https://github.com/apache/doris/pull/12782> [fix](memtracker) Refactor
   load channel mem tracker to improve accuracy #12791
   <https://github.com/apache/doris/pull/12791> [fix](memtracker) Introduce
   orphan mem tracker to verify memory tracking accuracy #12794
   <https://github.com/apache/doris/pull/12794> [fix](streamload&sink)
   release and allocate memory in the same tracker #12820
   <https://github.com/apache/doris/pull/12820> [fix](memtracker) Fix bug
   that cache consume causes inaccurate process tracker #12932
   <https://github.com/apache/doris/pull/12932> [fix](memtracker) Remove
   mem tracker record mem pool actual memory usage #12954
   <https://github.com/apache/doris/pull/12954> [branch-1.1-lts](cherry-pick)
   Fix bug that cache consume causes inaccurate process tracker #12951
   <https://github.com/apache/doris/pull/12951>

Bug Fix

   -

   Core dump on compaction with largeint [Bug][Compaction] core dump on
   compaction with largeint #10094
   <https://github.com/apache/doris/pull/10094>
   -

   Grouping sets cause be core or return wrong results [fix](grouping sets)
   columns in grouping sets should not be in selectstmt‘s AggFunc #12313
   <https://github.com/apache/doris/pull/12313>
   -

   PREAGGREGATION flag in orthogonal_bitmap_union_count operator is
wrong [fix](planner)
   fix orthogonal_bitmap_union_count planner : wrong PREAGGREGATION #12581
   <https://github.com/apache/doris/pull/12581>
   -

   Level1Iterator should release iterators in heap and it may cause memory
   leak (storage) Level1Iterator should release iterators in heap #12592
   <https://github.com/apache/doris/pull/12592>
   -

   Fix decommission failure with 2 BEs and existing colocation table
[fix](colocation)
   fix decommission failure with 2 BEs and colocation table #12644
   <https://github.com/apache/doris/pull/12644>
   -

   BE may core dump because of stack-buffer-overflow when
   TBrokerOpenReaderResponse too large [Bugfix](stack_over_flow) fix be may
   core dump because of stack-buffer-overflow when TBrokerOpenReaderResponse
   too large #12658 <https://github.com/apache/doris/pull/12658>
   -

   BE may OOM during load when error code -238 occurs [fix](memory) Fix BE
   OOM when load -238 fail #12666
   <https://github.com/apache/doris/pull/12666>
   -

   Fix wrong child expression of lead function [Bug](lead) fix wrong child
   expression of lead function #12587
   <https://github.com/apache/doris/pull/12587>
   -

   Fix intersect query failed in row storage code [fix](intersect) fix
   intersect query failed in row storage code #12712
   <https://github.com/apache/doris/pull/12712>
   -

   Fix wrong result produced by CURDATE()/CURRENT_DATE() function [Bug](date)
   Fix wrong result produced by date function #12720
   <https://github.com/apache/doris/pull/12720>
   -

   Fix lateral view explode_split with temp table bug [fix](lateral_view)
   fix lateral view explode_split with temp table #12643
   <https://github.com/apache/doris/pull/12643>
   -

   Bucket shuffle join plan is wrong in two same table [Bug](bucket
   shuffle) fix error bucket shuffle join plan in two same table #12930
   <https://github.com/apache/doris/pull/12930>
   -

   Fix bug that tablet version may be wrong when doing alter and load
[fix](alter-load)
   fix bug that tablet version may be wrong when doing alter and load #13070
   <https://github.com/apache/doris/pull/13070>
   -

   BE core when load data using broker with md5sum()/sm3sum() [fix](Broker
   Load)fix bug when load data using broker with md5sum()/sm3sum() #13009
   <https://github.com/apache/doris/pull/13009>

Behavior Changes

   - Disable PageCache and ChunkAllocator by default to reduce memory
   usage. User could enable this by changing disable_storage_page_cache and
   chunk_reserved_bytes_limit.



-- 
Best Regards,
Yi Guo Lei
SelectDB
The Best Cloud Native Platform for Apache Doris
Tel:134-3991-0228
Email:yiguolei@gmail.com

Re: Discussion about Doris 1.1.3

Posted by Morningman <mo...@163.com>.

Thanks for your work and glad to see the good maintainance of v1.1 lets.

> 在 2022年10月9日，21:42，Jiafeng.Zhang <zh...@gmail.com> 写道：
> 
> +1
> 
> 
> Looking forward to it, there are too many problems with 1.1.2 and this
> version is urgently needed
> 
> 
> GuoLei Yi <yi...@gmail.com>于2022年10月9日 周日11:52写道：
> 
>> In this release, Doris Team has fixed more than 80 issues or performance
>> improvement since 1.1.2. This release is a bugfix release on 1.1 and all
>> users are encouraged to upgrade to this release.
>> 
>> I have created an issue about this release
>> https://github.com/apache/doris/issues/13199, please modify this issue
>> directly if I missed some important bugs for features.
>> Features
>> 
>>   -
>> 
>>   Support escape identifiers for sqlserver and postgresql in ODBC table.
>>   -
>> 
>>   Could use Parquet as output file format.
>> 
>> Improvements
>> 
>>   -
>> 
>>   Optimize flush policy to avoid small segments [Enhancement](load)
>>   optimize flush policy to avoid small segments #12706
>>   <https://github.com/apache/doris/pull/12706>
>>   -
>> 
>>   Refactor runtime filter to reduce the prepare time.
>> [improvement](runtimefilter)
>>   shorter time prepare consumes #13127
>>   <https://github.com/apache/doris/pull/13127>
>>   -
>> 
>>   Lots of memory control related issues during query or load process.
>> [fix](memtracker)
>>   Remove consume mem tracker GC and fix mem limit exceed error format
>>    #12682 <https://github.com/apache/doris/pull/12682> [fix](memtracker)
>>   Process physical mem check does not include tc/jemalloc allocator cache
>>    #12688 <https://github.com/apache/doris/pull/12688> [fix](memtracker)
>>   Fix memory leak by calling boost::stacktrace in mem hook #12708
>>   <https://github.com/apache/doris/pull/12708> [Bugfix](mem) Fix memory
>>   limit check may overflow #12776
>>   <https://github.com/apache/doris/pull/12776> [fix](memtracker) Fix
>>   thread mem tracker try consume accuracy #12782
>>   <https://github.com/apache/doris/pull/12782> [fix](memtracker) Refactor
>>   load channel mem tracker to improve accuracy #12791
>>   <https://github.com/apache/doris/pull/12791> [fix](memtracker)
>> Introduce
>>   orphan mem tracker to verify memory tracking accuracy #12794
>>   <https://github.com/apache/doris/pull/12794> [fix](streamload&sink)
>>   release and allocate memory in the same tracker #12820
>>   <https://github.com/apache/doris/pull/12820> [fix](memtracker) Fix bug
>>   that cache consume causes inaccurate process tracker #12932
>>   <https://github.com/apache/doris/pull/12932> [fix](memtracker) Remove
>>   mem tracker record mem pool actual memory usage #12954
>>   <https://github.com/apache/doris/pull/12954>
>> [branch-1.1-lts](cherry-pick)
>>   Fix bug that cache consume causes inaccurate process tracker #12951
>>   <https://github.com/apache/doris/pull/12951>
>> 
>> Bug Fix
>> 
>>   -
>> 
>>   Core dump on compaction with largeint [Bug][Compaction] core dump on
>>   compaction with largeint #10094
>>   <https://github.com/apache/doris/pull/10094>
>>   -
>> 
>>   Grouping sets cause be core or return wrong results [fix](grouping sets)
>>   columns in grouping sets should not be in selectstmt‘s AggFunc #12313
>>   <https://github.com/apache/doris/pull/12313>
>>   -
>> 
>>   PREAGGREGATION flag in orthogonal_bitmap_union_count operator is
>> wrong [fix](planner)
>>   fix orthogonal_bitmap_union_count planner : wrong PREAGGREGATION #12581
>>   <https://github.com/apache/doris/pull/12581>
>>   -
>> 
>>   Level1Iterator should release iterators in heap and it may cause memory
>>   leak (storage) Level1Iterator should release iterators in heap #12592
>>   <https://github.com/apache/doris/pull/12592>
>>   -
>> 
>>   Fix decommission failure with 2 BEs and existing colocation table
>> [fix](colocation)
>>   fix decommission failure with 2 BEs and colocation table #12644
>>   <https://github.com/apache/doris/pull/12644>
>>   -
>> 
>>   BE may core dump because of stack-buffer-overflow when
>>   TBrokerOpenReaderResponse too large [Bugfix](stack_over_flow) fix be may
>>   core dump because of stack-buffer-overflow when
>> TBrokerOpenReaderResponse
>>   too large #12658 <https://github.com/apache/doris/pull/12658>
>>   -
>> 
>>   BE may OOM during load when error code -238 occurs [fix](memory) Fix BE
>>   OOM when load -238 fail #12666
>>   <https://github.com/apache/doris/pull/12666>
>>   -
>> 
>>   Fix wrong child expression of lead function [Bug](lead) fix wrong child
>>   expression of lead function #12587
>>   <https://github.com/apache/doris/pull/12587>
>>   -
>> 
>>   Fix intersect query failed in row storage code [fix](intersect) fix
>>   intersect query failed in row storage code #12712
>>   <https://github.com/apache/doris/pull/12712>
>>   -
>> 
>>   Fix wrong result produced by CURDATE()/CURRENT_DATE() function
>> [Bug](date)
>>   Fix wrong result produced by date function #12720
>>   <https://github.com/apache/doris/pull/12720>
>>   -
>> 
>>   Fix lateral view explode_split with temp table bug [fix](lateral_view)
>>   fix lateral view explode_split with temp table #12643
>>   <https://github.com/apache/doris/pull/12643>
>>   -
>> 
>>   Bucket shuffle join plan is wrong in two same table [Bug](bucket
>>   shuffle) fix error bucket shuffle join plan in two same table #12930
>>   <https://github.com/apache/doris/pull/12930>
>>   -
>> 
>>   Fix bug that tablet version may be wrong when doing alter and load
>> [fix](alter-load)
>>   fix bug that tablet version may be wrong when doing alter and load
>> #13070
>>   <https://github.com/apache/doris/pull/13070>
>>   -
>> 
>>   BE core when load data using broker with md5sum()/sm3sum() [fix](Broker
>>   Load)fix bug when load data using broker with md5sum()/sm3sum() #13009
>>   <https://github.com/apache/doris/pull/13009>
>> 
>> Behavior Changes
>> 
>>   - Disable PageCache and ChunkAllocator by default to reduce memory
>>   usage. User could enable this by changing disable_storage_page_cache and
>>   chunk_reserved_bytes_limit.
>> 
>> 
>> 
>> --
>> Best Regards,
>> Yi Guo Lei
>> SelectDB
>> The Best Cloud Native Platform for Apache Doris
>> Tel:134-3991-0228
>> Email:yiguolei@gmail.com
>> 
> -- 
> 张家峰
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org

Re: Discussion about Doris 1.1.3

Posted by "Jiafeng.Zhang" <zh...@gmail.com>.

+1


Looking forward to it, there are too many problems with 1.1.2 and this
version is urgently needed


GuoLei Yi <yi...@gmail.com>于2022年10月9日 周日11:52写道：

> In this release, Doris Team has fixed more than 80 issues or performance
> improvement since 1.1.2. This release is a bugfix release on 1.1 and all
> users are encouraged to upgrade to this release.
>
> I have created an issue about this release
> https://github.com/apache/doris/issues/13199, please modify this issue
> directly if I missed some important bugs for features.
> Features
>
>    -
>
>    Support escape identifiers for sqlserver and postgresql in ODBC table.
>    -
>
>    Could use Parquet as output file format.
>
> Improvements
>
>    -
>
>    Optimize flush policy to avoid small segments [Enhancement](load)
>    optimize flush policy to avoid small segments #12706
>    <https://github.com/apache/doris/pull/12706>
>    -
>
>    Refactor runtime filter to reduce the prepare time.
> [improvement](runtimefilter)
>    shorter time prepare consumes #13127
>    <https://github.com/apache/doris/pull/13127>
>    -
>
>    Lots of memory control related issues during query or load process.
> [fix](memtracker)
>    Remove consume mem tracker GC and fix mem limit exceed error format
>     #12682 <https://github.com/apache/doris/pull/12682> [fix](memtracker)
>    Process physical mem check does not include tc/jemalloc allocator cache
>     #12688 <https://github.com/apache/doris/pull/12688> [fix](memtracker)
>    Fix memory leak by calling boost::stacktrace in mem hook #12708
>    <https://github.com/apache/doris/pull/12708> [Bugfix](mem) Fix memory
>    limit check may overflow #12776
>    <https://github.com/apache/doris/pull/12776> [fix](memtracker) Fix
>    thread mem tracker try consume accuracy #12782
>    <https://github.com/apache/doris/pull/12782> [fix](memtracker) Refactor
>    load channel mem tracker to improve accuracy #12791
>    <https://github.com/apache/doris/pull/12791> [fix](memtracker)
> Introduce
>    orphan mem tracker to verify memory tracking accuracy #12794
>    <https://github.com/apache/doris/pull/12794> [fix](streamload&sink)
>    release and allocate memory in the same tracker #12820
>    <https://github.com/apache/doris/pull/12820> [fix](memtracker) Fix bug
>    that cache consume causes inaccurate process tracker #12932
>    <https://github.com/apache/doris/pull/12932> [fix](memtracker) Remove
>    mem tracker record mem pool actual memory usage #12954
>    <https://github.com/apache/doris/pull/12954>
> [branch-1.1-lts](cherry-pick)
>    Fix bug that cache consume causes inaccurate process tracker #12951
>    <https://github.com/apache/doris/pull/12951>
>
> Bug Fix
>
>    -
>
>    Core dump on compaction with largeint [Bug][Compaction] core dump on
>    compaction with largeint #10094
>    <https://github.com/apache/doris/pull/10094>
>    -
>
>    Grouping sets cause be core or return wrong results [fix](grouping sets)
>    columns in grouping sets should not be in selectstmt‘s AggFunc #12313
>    <https://github.com/apache/doris/pull/12313>
>    -
>
>    PREAGGREGATION flag in orthogonal_bitmap_union_count operator is
> wrong [fix](planner)
>    fix orthogonal_bitmap_union_count planner : wrong PREAGGREGATION #12581
>    <https://github.com/apache/doris/pull/12581>
>    -
>
>    Level1Iterator should release iterators in heap and it may cause memory
>    leak (storage) Level1Iterator should release iterators in heap #12592
>    <https://github.com/apache/doris/pull/12592>
>    -
>
>    Fix decommission failure with 2 BEs and existing colocation table
> [fix](colocation)
>    fix decommission failure with 2 BEs and colocation table #12644
>    <https://github.com/apache/doris/pull/12644>
>    -
>
>    BE may core dump because of stack-buffer-overflow when
>    TBrokerOpenReaderResponse too large [Bugfix](stack_over_flow) fix be may
>    core dump because of stack-buffer-overflow when
> TBrokerOpenReaderResponse
>    too large #12658 <https://github.com/apache/doris/pull/12658>
>    -
>
>    BE may OOM during load when error code -238 occurs [fix](memory) Fix BE
>    OOM when load -238 fail #12666
>    <https://github.com/apache/doris/pull/12666>
>    -
>
>    Fix wrong child expression of lead function [Bug](lead) fix wrong child
>    expression of lead function #12587
>    <https://github.com/apache/doris/pull/12587>
>    -
>
>    Fix intersect query failed in row storage code [fix](intersect) fix
>    intersect query failed in row storage code #12712
>    <https://github.com/apache/doris/pull/12712>
>    -
>
>    Fix wrong result produced by CURDATE()/CURRENT_DATE() function
> [Bug](date)
>    Fix wrong result produced by date function #12720
>    <https://github.com/apache/doris/pull/12720>
>    -
>
>    Fix lateral view explode_split with temp table bug [fix](lateral_view)
>    fix lateral view explode_split with temp table #12643
>    <https://github.com/apache/doris/pull/12643>
>    -
>
>    Bucket shuffle join plan is wrong in two same table [Bug](bucket
>    shuffle) fix error bucket shuffle join plan in two same table #12930
>    <https://github.com/apache/doris/pull/12930>
>    -
>
>    Fix bug that tablet version may be wrong when doing alter and load
> [fix](alter-load)
>    fix bug that tablet version may be wrong when doing alter and load
> #13070
>    <https://github.com/apache/doris/pull/13070>
>    -
>
>    BE core when load data using broker with md5sum()/sm3sum() [fix](Broker
>    Load)fix bug when load data using broker with md5sum()/sm3sum() #13009
>    <https://github.com/apache/doris/pull/13009>
>
> Behavior Changes
>
>    - Disable PageCache and ChunkAllocator by default to reduce memory
>    usage. User could enable this by changing disable_storage_page_cache and
>    chunk_reserved_bytes_limit.
>
>
>
> --
> Best Regards,
> Yi Guo Lei
> SelectDB
> The Best Cloud Native Platform for Apache Doris
> Tel:134-3991-0228
> Email:yiguolei@gmail.com
>
-- 
张家峰