You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/10/29 09:16:56 UTC
[GitHub] [incubator-doris] zuochunwei opened a new pull request #6971: hll optimize: trace memory usage, new explicit data when really need
zuochunwei opened a new pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971
reduce hll memory occupied：
replace uint64_t _explicit_data[160*2] with uint64_t*
new memory for explicit data when really needed
trace HLL memory usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
zuochunwei commented on a change in pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r745402142
##########
File path: be/src/olap/hll.h
##########
@@ -33,6 +33,7 @@ class Slice;
const static int HLL_COLUMN_PRECISION = 14;
const static int HLL_ZERO_COUNT_BITS = (64 - HLL_COLUMN_PRECISION);
const static int HLL_EXPLICIT_INT64_NUM = 160;
+const static int HLL_EXPLICIT_INT64_NUM_DOUBLE = HLL_EXPLICIT_INT64_NUM*2;
Review comment:
fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
zuochunwei commented on a change in pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r745396863
##########
File path: be/src/olap/hll.h
##########
@@ -33,6 +33,7 @@ class Slice;
const static int HLL_COLUMN_PRECISION = 14;
const static int HLL_ZERO_COUNT_BITS = (64 - HLL_COLUMN_PRECISION);
const static int HLL_EXPLICIT_INT64_NUM = 160;
+const static int HLL_EXPLICIT_INT64_NUM_DOUBLE = HLL_EXPLICIT_INT64_NUM*2;
Review comment:
do you means i should add space between HLL_EXPLICIT_INT64_NUM and *?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
zuochunwei commented on pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#issuecomment-955984076
> Hi @zuochunwei Could you please create an issue to describe more detail about how this works and how much memory can be reduced after this PR?
each HLL object will reduce nearly 2k bytes，because 160*2*sizeof(uint64_t) result 2560，almost 2k bytes.
so when use tool to import data，if the table have hundreds of millions records, we generate one hll object for each record, the memory usage will increase hundreds Gigabytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zuochunwei commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
zuochunwei commented on a change in pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r745398893
##########
File path: be/src/olap/hll.h
##########
@@ -154,7 +174,7 @@ class HyperLogLog {
HllDataType _type = HLL_DATA_EMPTY;
uint32_t _explicit_data_num = 0;
- uint64_t _explicit_data[HLL_EXPLICIT_INT64_NUM * 2];
+ uint64_t* _explicit_data = nullptr;
Review comment:
no need replace with unique_ptr, keep style consistency with uint8_t *_registers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
HappenLee commented on a change in pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r744450847
##########
File path: be/src/olap/hll.h
##########
@@ -154,7 +174,7 @@ class HyperLogLog {
HllDataType _type = HLL_DATA_EMPTY;
uint32_t _explicit_data_num = 0;
- uint64_t _explicit_data[HLL_EXPLICIT_INT64_NUM * 2];
+ uint64_t* _explicit_data = nullptr;
Review comment:
directly use the raw ptr is dangerous and complex. Rethink should we replace it by std::unique_ptr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
morningman commented on pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#issuecomment-955897530
Hi @zuochunwei Could you please create an issue to describe more detail about how this works and how much memory can be reduced after this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#issuecomment-966793653
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#issuecomment-966239701
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#issuecomment-962862614
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HappenLee commented on a change in pull request #6971: hll optimize: trace memory usage, new explicit data when really need
Posted by GitBox <gi...@apache.org>.
HappenLee commented on a change in pull request #6971:
URL: https://github.com/apache/incubator-doris/pull/6971#discussion_r744450364
##########
File path: be/src/olap/hll.h
##########
@@ -33,6 +33,7 @@ class Slice;
const static int HLL_COLUMN_PRECISION = 14;
const static int HLL_ZERO_COUNT_BITS = (64 - HLL_COLUMN_PRECISION);
const static int HLL_EXPLICIT_INT64_NUM = 160;
+const static int HLL_EXPLICIT_INT64_NUM_DOUBLE = HLL_EXPLICIT_INT64_NUM*2;
Review comment:
HLL_EXPLICIT_INT64_NUM * 2
##########
File path: be/src/olap/hll.h
##########
@@ -154,7 +174,7 @@ class HyperLogLog {
HllDataType _type = HLL_DATA_EMPTY;
uint32_t _explicit_data_num = 0;
- uint64_t _explicit_data[HLL_EXPLICIT_INT64_NUM * 2];
+ uint64_t* _explicit_data = nullptr;
Review comment:
directly use the raw ptr is dangerous and complex. Rethink shoule we replace is use std::unique_ptr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org