You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/06/14 09:16:08 UTC
[GitHub] [incubator-druid] quenlang commented on issue #7824: Kafka index
service use a lot of direct memory during segment publish
quenlang commented on issue #7824: Kafka index service use a lot of direct memory during segment publish
URL: https://github.com/apache/incubator-druid/issues/7824#issuecomment-502035506
I found a lot of ```Allocating new littleEndByteBuf``` operations in the log.
Finally, allocated 333023 littleEndByteBuf and the usage of direct memory arrived 18GB, and OS killed this task.
The ingested data was very small, just 2GB in six hours, and all the dimensions have very low cardinality.
Why the task need so many direct memory?
@himanshug
Can you give me some advices? Thank you so much!
```
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,020]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,021]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,022]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,023]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,024]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,025]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,026]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,027]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,028]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,029]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org