You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/11/18 09:20:10 UTC
[GitHub] [druid] liubo-it opened a new issue #11944: ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS
liubo-it opened a new issue #11944:
URL: https://github.com/apache/druid/issues/11944
@asdf2014 @fjy @jihoonson
Hello, I'm using compact task to compress segment. I found an error, memory overflow, but I don't know what caused it. Can you give me a direction?
### compact task
### Affected Version
0.19.0
### Task error logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #11944: ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #11944:
URL: https://github.com/apache/druid/issues/11944#issuecomment-974382299
What is your average segment size at the moment and how many rows per segment does it have on an average? If there are multiple sketch fields with atleast 8k in size, 13M rows per segment can create very large segments which might not be the goal.
You could always throw more heap at the compaction tasks proportional to the total segment size it is trying to compact but it would be useful to first estimate how big of a segment would 13M rows create and if this estimated size is within the recommended size for segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liubo-it closed issue #11944: ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS
Posted by GitBox <gi...@apache.org>.
liubo-it closed issue #11944:
URL: https://github.com/apache/druid/issues/11944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liubo-it commented on issue #11944: ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS
Posted by GitBox <gi...@apache.org>.
liubo-it commented on issue #11944:
URL: https://github.com/apache/druid/issues/11944#issuecomment-973830771
@a2l007 `2021-08-19 to 2021-08-20` we have about `40G` to `50 G` of data
yes ! `MaxBytesInMemory` is set to -1 because our dataSource Metrics has many Sketch fields,I'm not sure if I set the `MaxRowsPerSegment` too big, should I make `Xmx` big enough? You still need to adjust the `off-heap` memory? thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liubo-it commented on issue #11944: ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS
Posted by GitBox <gi...@apache.org>.
liubo-it commented on issue #11944:
URL: https://github.com/apache/druid/issues/11944#issuecomment-975042426
I think I know what to do. I'll keep trying. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #11944: ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS
Posted by GitBox <gi...@apache.org>.
a2l007 commented on issue #11944:
URL: https://github.com/apache/druid/issues/11944#issuecomment-973003377
@liubo-it What is the total size of all the segments for that datasource for the interval `2021-08-19 to 2021-08-20`? Also could you please let us know why is `maxBytesInMemory` set to -1? Is it because you have sketch aggs?
I suspect that the maxRowsPerSegment might be too large in this case which might be using up all the heap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org