You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@iotdb.apache.org by "Houliang Qi (Jira)" <ji...@apache.org> on 2020/12/07 03:49:00 UTC
[jira] [Commented] (IOTDB-1005) [Distributed] Make the frame size
of thrift scalable
[ https://issues.apache.org/jira/browse/IOTDB-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17244936#comment-17244936 ]
Houliang Qi commented on IOTDB-1005:
------------------------------------
As we can see from the log, the auto-resize of the max frame size bring another problem, which will cause OOM, as we can see from the profile, the byte[] array consume about 18GB, which is generated by thrift thread.
this analyze alose brings another one problem, its reasonable that the max thread client num is 10000？
I think we should do the following two things:
1. There must be a segmentation mechanism to divide large requests into small requests, as far I know, apache HBase default max frame is 2MB,
2. The maximum number of threads needs to be reduced(now the default value is 1000), but some tests may be needed to set the appropriate size.
[^log.out]!image-2020-12-07-10-52-44-343.png!
> [Distributed] Make the frame size of thrift scalable
> ----------------------------------------------------
>
> Key: IOTDB-1005
> URL: https://issues.apache.org/jira/browse/IOTDB-1005
> Project: Apache IoTDB
> Issue Type: Improvement
> Components: Core/Cluster
> Reporter: Houliang Qi
> Assignee: Houliang Qi
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-12-07-10-52-44-343.png, log.out
>
>
> As described in IOTDB-998[1], when the snapshot size larger than the max frame size, it would fail when do snapshot, another way is to make the frame size of thrift is scalable. it would be easier than IOTDB-998
> [1]https://issues.apache.org/jira/browse/IOTDB-998
--
This message was sent by Atlassian Jira
(v8.3.4#803005)