You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2023/04/26 18:36:42 UTC
[GitHub] [doris] xinyiZzz opened a new pull request, #19126: [fix](memory) Fix AggFunc memory leak due to incorrect destroy
xinyiZzz opened a new pull request, #19126:
URL: https://github.com/apache/doris/pull/19126
# Proposed changes
Issue Number: close #xxx
## Problem summary
Asan memory leak:
```
Direct leak of 192 byte(s) in 6 object(s) allocated from:
#0 0x5598a4329bee in malloc (/mnt/hdd01/dorisTestEnv/VEC_ASAN/be/lib/doris_be+0x14cd1bee) (BuildId: 0e35cb6207c3ab7c)
#1 0x5598a48e85c5 in Allocator<false, true>::alloc(unsigned long, unsigned long) /doris/be/src/vec/common/all
ocator.h:157:27
#2 0x5598a500adb6 in doris::vectorized::Allocator_<void* phmap::priv::Allocate<8ul, doris::vectorized::Allocator_<unsigned long>>(doris::vectorized::Alloc
ator_<unsigned long>*, unsigned long)::M>::allocate(unsigned long) /doris/be/src/vec/common/hash_table/phmap_fwd_
decl.h:39:73
#3 0x5598a500ad6e in phmap::allocator_traits<doris::vectorized::Allocator_<void* phmap::priv::Allocate<8ul, doris::vectorized::Allocator_<unsigned long>>(
doris::vectorized::Allocator_<unsigned long>*, unsigned long)::M>>::allocate(doris::vectorized::Allocator_<void* phmap::priv::Allocate<8ul, doris::vectorized:
:Allocator_<unsigned long>>(doris::vectorized::Allocator_<unsigned long>*, unsigned long)::M>&, unsigned long) /d
oris/thirdparty/installed/include/parallel_hashmap/phmap_base.h:1468:18
#4 0x5598a500a919 in void* phmap::priv::Allocate<8ul, doris::vectorized::Allocator_<unsigned long>>(doris::vectorized::Allocator_<unsigned long>*, unsigne
d long) /doris/thirdparty/installed/include/parallel_hashmap/phmap_base.h:4356:13
...
#10 0x5598a726fe40 in phmap::flat_hash_set<unsigned long, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, doris::vectorized::Allocator_<unsigne
d long>>::operator=(phmap::flat_hash_set<unsigned long, phmap::Hash<unsigned long>, phmap::EqualTo<unsigned long>, doris::vectorized::Allocator_<unsigned long
>> const&) /doris/thirdparty/installed/include/parallel_hashmap/phmap.h:4577:7
#11 0x5598a727e7a4 in doris::HyperLogLog::merge(doris::HyperLogLog const&) /doris/be/src/olap/hll.cpp:89:23
#12 0x5598ab7a544e in doris::vectorized::AggregateFunctionHLLData::add(doris::vectorized::IColumn const*, unsigned long) /mnt/hdd01/repo_center/doris_bran
ch-2.0-alpha/doris/be/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h:59:17
#13 0x5598ab7a09a0 in doris::vectorized::AggregateFunctionHLLUnion<doris::vectorized::AggregateFunctionHLLUnionImpl<doris::vectorized::AggregateFunctionHL
LData>>::add(char*, doris::vectorized::IColumn const**, unsigned long, doris::vectorized::Arena*) const /doris/be
/src/vec/aggregate_functions/aggregate_function_hll_union_agg.h:104:27
#14 0x5598a5dfe3b1 in doris::MultiBlockMerger::merge(std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>,
std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>>> const&, doris::RowsetWriter*, unsigned long*) doris/be/src/olap/schema_change.cpp:107:52
#15 0x5598a5dbc8d2 in doris::VSchemaChangeWithSorting::_internal_sorting(std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::
vectorized::Block>>, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block>>>> const&, doris::Version const&,
long, std::shared_ptr<doris::Tablet>, doris::RowsetTypePB, doris::SegmentsOverlapPB, std::shared_ptr<doris::Rowset>*) /mnt/hdd01/repo_center/doris_branch-2.0-
alpha/doris/be/src/olap/schema_change.cpp:565:5
#16 0x5598a5dbb304 in doris::VSchemaChangeWithSorting::_inner_process(std::shared_ptr<doris::RowsetReader>, doris::RowsetWriter*, std::shared_ptr<doris::T
ablet>, std::shared_ptr<doris::TabletSchema>)::$_3::operator()() const /doris/be/src/olap/schema_change.cpp:489:9
#17 0x5598a5dba862 in doris::VSchemaChangeWithSorting::_inner_process(std::shared_ptr<doris::RowsetReader>, doris::RowsetWriter*, std::shared_ptr<doris::T
ablet>, std::shared_ptr<doris::TabletSchema>) /doris/be/src/olap/schema_change.cpp:533:5
#18 0x5598a5e0b069 in doris::SchemaChange::process(std::shared_ptr<doris::RowsetReader>, doris::RowsetWriter*, std::shared_ptr<doris::Tablet>, std::shared
_ptr<doris::Tablet>, std::shared_ptr<doris::TabletSchema>) /doris/be/src/olap/schema_change.h:87:9
#19 0x5598a5dd2db5 in doris::SchemaChangeHandler::_convert_historical_rowsets(doris::SchemaChangeHandler::SchemaChangeParams const&) /mnt/hdd01/repo_cente
r/doris_branch-2.0-alpha/doris/be/src/olap/schema_change.cpp:1598:33
#20 0x5598a5dcad14 in doris::SchemaChangeHandler::_do_process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) /mnt/hdd01/repo_center/doris_branch-2.0-alp
ha/doris/be/src/olap/schema_change.cpp:1124:15
#21 0x5598a5dc54db in doris::SchemaChangeHandler::process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) /d
oris/be/src/olap/schema_change.cpp:857:18
#22 0x5598a69d35ed in doris::EngineAlterTabletTask::execute() /doris/be/src/olap/task/engine_alter_tablet_tas
k.cpp:40:18
#23 0x5598a5d08d6c in doris::StorageEngine::execute_task(doris::EngineTask*) /doris/be/src/olap/storage_engin
e.cpp:1001:5
#24 0x5598a5e9a550 in doris::TaskWorkerPool::_alter_tablet(doris::TAgentTaskRequest const&, long, doris::TTaskType::type, doris::TFinishTaskRequest*) /mnt
/hdd01/repo_center/doris_branch-2.0-alpha/doris/be/src/agent/task_worker_pool.cpp:680:42
#25 0x5598a5e7e44b in doris::TaskWorkerPool::_alter_tablet_worker_thread_callback() /doris/be/src/agent/task_
worker_pool.cpp:640:17
```
```
#0 0x5598a4363ded in operator new(unsigned long) (/mnt/hdd01/dorisTestEnv/VEC_ASAN/be/lib/doris_be+0x14d0bded) (BuildId: 0e35cb6207c3ab7c)
#1 0x5598ae1606a1 in COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnVector<long>> COWHelper<doris::vectorized::ColumnVectorHelper,
doris::vectorized::ColumnVector<long>>::create<>() /doris/be/src/vec/common/cow.h:412:27
#2 0x5598ae160268 in doris::vectorized::ColumnVector<long>::clone_resized(unsigned long) const /doris/be/src/
vec/columns/column_vector.cpp:319:16
#3 0x5598a477c0f9 in doris::vectorized::IColumn::clone_empty() const /doris/be/src/vec/columns/column.h:117:5
3
#4 0x5598ae632232 in doris::vectorized::ColumnWithTypeAndName::clone_empty() const /doris/be/src/vec/core/col
umn_with_type_and_name.cpp:34:30
#5 0x5598ae5e1d41 in doris::vectorized::Block::clone_empty() const /doris/be/src/vec/core/block.cpp:508:25
#6 0x5598b32ac19e in doris::vectorized::AggregateFunctionSortData::AggregateFunctionSortData(std::vector<doris::vectorized::SortColumnDescription, std::al
locator<doris::vectorized::SortColumnDescription>>, doris::vectorized::Block const&) /doris/be/src/vec/aggregate_
functions/aggregate_function_sort.h:39:60
#7 0x5598b32a895f in doris::vectorized::AggregateFunctionSort<doris::vectorized::AggregateFunctionSortData>::create(char*) const /mnt/hdd01/repo_center/do
ris_branch-2.0-alpha/doris/be/src/vec/aggregate_functions/aggregate_function_sort.h:160:21
#8 0x5598b32a9d3f in doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionSort<doris::vectorized::AggregateFunctionSortData>>::
deserialize_vec(char*, doris::vectorized::ColumnString const*, doris::vectorized::Arena*, unsigned long) const /d
oris/be/src/vec/aggregate_functions/aggregate_function.h:335:48
#9 0x5598b32aa00f in doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionSort<doris::vectorized::AggregateFunctionSortData>>::
deserialize_from_column(char*, doris::vectorized::IColumn const&, doris::vectorized::Arena*, unsigned long) const /mnt/hdd01/repo_center/doris_branch-2.0-alph
a/doris/be/src/vec/aggregate_functions/aggregate_function.h:342:9
```
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #19126: [fix](memory) Fix AggFunc memory leak due to incorrect destroy
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #19126:
URL: https://github.com/apache/doris/pull/19126#issuecomment-1523893701
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #19126: [fix](memory) Fix AggFunc memory leak due to incorrect destroy
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #19126:
URL: https://github.com/apache/doris/pull/19126
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on a diff in pull request #19126: [fix](memory) Fix AggFunc memory leak due to incorrect destroy
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #19126:
URL: https://github.com/apache/doris/pull/19126#discussion_r1178264636
##########
be/src/util/defer_op.h:
##########
@@ -40,4 +40,9 @@ class Defer {
T _closure;
};
+// Nested use Defer, variable name concat line number
+#define DEFER_CONCAT(n, ...) const auto defer##n = Defer([&]() { __VA_ARGS__; })
Review Comment:
warning: macro is not used [clang-diagnostic-unused-macros]
```cpp
#define DEFER_CONCAT(n, ...) const auto defer##n = Defer([&]() { __VA_ARGS__; })
^
```
##########
be/src/vec/aggregate_functions/aggregate_function.h:
##########
@@ -47,6 +48,16 @@ using DataTypes = std::vector<DataTypePtr>;
using AggregateDataPtr = char*;
using ConstAggregateDataPtr = const char*;
+#define SAFE_CREATE(create, destroy) \
Review Comment:
warning: macro is not used [clang-diagnostic-unused-macros]
```cpp
#define SAFE_CREATE(create, destroy) \
^
```
##########
be/src/util/defer_op.h:
##########
@@ -40,4 +40,9 @@
T _closure;
};
+// Nested use Defer, variable name concat line number
+#define DEFER_CONCAT(n, ...) const auto defer##n = Defer([&]() { __VA_ARGS__; })
+#define DEFER_FWD(n, ...) DEFER_CONCAT(n, __VA_ARGS__)
Review Comment:
warning: macro is not used [clang-diagnostic-unused-macros]
```cpp
#define DEFER_FWD(n, ...) DEFER_CONCAT(n, __VA_ARGS__)
^
```
##########
be/src/util/defer_op.h:
##########
@@ -40,4 +40,9 @@
T _closure;
};
+// Nested use Defer, variable name concat line number
+#define DEFER_CONCAT(n, ...) const auto defer##n = Defer([&]() { __VA_ARGS__; })
+#define DEFER_FWD(n, ...) DEFER_CONCAT(n, __VA_ARGS__)
+#define DEFER(...) DEFER_FWD(__LINE__, __VA_ARGS__)
Review Comment:
warning: macro is not used [clang-diagnostic-unused-macros]
```cpp
#define DEFER(...) DEFER_FWD(__LINE__, __VA_ARGS__)
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #19126: [fix](memory) Fix AggFunc memory leak due to incorrect destroy
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #19126:
URL: https://github.com/apache/doris/pull/19126#issuecomment-1524581655
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #19126: [fix](memory) Fix AggFunc memory leak due to incorrect destroy
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #19126:
URL: https://github.com/apache/doris/pull/19126#issuecomment-1524438197
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org