You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "devilcony (via GitHub)" <gi...@apache.org> on 2024/01/31 09:47:38 UTC
[I] [Bug] Failed when CREATE MATERIALIZED VIEW or ADD ROLLUP due to OOM, exec_mem_limit does not work [doris]
devilcony opened a new issue, #30649:
URL: https://github.com/apache/doris/issues/30649
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
doris-2.0.2-rc05
### What's Wrong?
My table have 4 HLL columns, if CREATE MATERIALIZED VIEW or ADD ROLLUP use 2 or above HLL columns then create failed , if use only 1 HLL column then success. I doubt this is a memory issue.
SHOW ALTER TABE MATERIALIZED VIEW -Msg shows:
errCode = 2, detailMessage = rollup tasks failed on same tablet reach threshold [[ALTER], signature: 165636, backendId: 11268, tablet id: 165636], reason=task type: ALTER, status_code: INTERNAL_ERROR, status_message: [(10.195.8.176)[E-1602]SchemaChange check row nums failed], backendId: Backend [id=11268, host=10.195.8.176, heartbeatPort=9050, alive=true, lastStartTime=2024-01-31 14:21:00, tags: {location=default}], signature: 165636
BE logs shows:
W0131 14:58:47.489673 518774 beta_rowset_reader.cpp:289] failed to read next block: [MEM_LIMIT_EXCEEDED]PreCatch error code:11, [E11] Memory limit exceeded:[MEM_LIMIT_EXCEEDED]failed alloc size 32.00 B, exceeded tracker:<EngineAlterTabletTask#baseTabletI
d=165200:newTabletId=165636>, limit 2.00 GB, peak used 2.00 GB, current used 2.00 GB, exec node:<>, execute msg:Allocator mem tracker check failed. backend 10.195.8.176 process memory used 4.91 GB, limit 24.85 GB. Can `set exec_mem_limit=8G` to change li
mit, details see be.INFO.
I have run `set global exec_mem_limit=8589934592` to extend memory, but did not work, the log still shows limit 2.00 GB.
### What You Expected?
Create successfully
### How to Reproduce?
CREATE TABLE IF NOT EXISTS test_hll_rollup
(
d1 VARCHAR(64)
,d2 DATE
,d3 INT
,d4 VARCHAR(64)
,d5 VARCHAR(1024)
,r1 BIGINT SUM
,r2 HLL HLL_UNION
,r3 HLL HLL_UNION
,r4 HLL HLL_UNION
,r5 HLL HLL_UNION
,r6 INT SUM
,r7 BIGINT REPLACE
,r8 DATETIME MAX
)
AGGREGATE KEY(d1,d2,d3,d4,d5)
DISTRIBUTED BY HASH(d1)
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);
insert about 200000000 rows into this table;
alter table test_hll_rollup add rollup test_hll_rollup_1(d1,d2,r1,r2,r3,r4);
SHOW ALTER TABLE MATERIALIZED VIEW;
### Anything Else?
https://github.com/apache/doris/issues/29724 maybe samilar with this but im not sure
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Failed when CREATE MATERIALIZED VIEW or ADD ROLLUP due to OOM, exec_mem_limit does not work [doris]
Posted by "devilcony (via GitHub)" <gi...@apache.org>.
devilcony closed issue #30649: [Bug] Failed when CREATE MATERIALIZED VIEW or ADD ROLLUP due to OOM, exec_mem_limit does not work
URL: https://github.com/apache/doris/issues/30649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Failed when CREATE MATERIALIZED VIEW or ADD ROLLUP due to OOM, exec_mem_limit does not work [doris]
Posted by "devilcony (via GitHub)" <gi...@apache.org>.
devilcony commented on issue #30649:
URL: https://github.com/apache/doris/issues/30649#issuecomment-1982914502
It works after I set this param, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Failed when CREATE MATERIALIZED VIEW or ADD ROLLUP due to OOM, exec_mem_limit does not work [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on issue #30649:
URL: https://github.com/apache/doris/issues/30649#issuecomment-1920454154
memlimit for schema change is set by memory_limitation_per_thread_for_schema_change_bytes in be.conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org