You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@iotdb.apache.org by Trevor Hart <tr...@ope.nz> on 2021/04/09 02:28:18 UTC
High CPU Usage
Hi Team,
I am still struggling with high CPU usage on V11.2. My CPU usage creeps up after a few days and basically got to 100% today and was unusable.
If I restart the iotDB server then its fine for a few days. This Linux server has 2 cores and 4Gb RAM. The database is running 24x7.
The data ingestion is typically small. Maybe a 500 points per minute. I can never see any data ingestion backing up unless the CPU is over 90%.
Questions;
1. Does anyone else see this behaviour?
2. Are there any maintenance tasks I should call daily? Should I be calling Flush daily?
3. Should I reduce the WAL threshold?
4. I am using compaction_strategy=LEVEL_COMPACTION
5. And have set enable_unseq_compaction=false
Is there anything else I can try or do I just need bigger hardware?
Thanks
Trevor
Re: High CPU Usage
Posted by Jialin Qiao <qi...@apache.org>.
Glad to hear that! Thanks for the feedback.
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Trevor Hart <tr...@ope.nz> 于2021年4月20日周二 上午8:35写道:
> Just reporting back on this topic again.
>
>
>
> The server has been up ~11 days now and CPU is still sitting around 4%
> constantly. Very happy with that!
>
>
> Thanks
>
> Trevor Hart
Re: High CPU Usage
Posted by Trevor Hart <tr...@ope.nz>.
Just reporting back on this topic again.
The server has been up ~11 days now and CPU is still sitting around 4% constantly. Very happy with that!
Thanks
Trevor Hart
Re: High CPU Usage
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
Glad to see the feedback :D
The mail list does not support attachments, you could upload it and share a
download link.
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Trevor Hart <tr...@ope.nz> 于2021年4月11日周日 上午6:47写道:
> 0.11.3 RC3 seems much happier so far. There was an initial spike in CPU
> after I upgradef to 11.3 but it has now settled down.
>
> I will keep an eye on it over the next week - see attachment.
>
> Thanks
> Trevor Hart
>
> ---- On Sat, 10 Apr 2021 15:45:36 +1200 * trevor@ope.nz <tr...@ope.nz> *
> wrote ----
>
> Thanks everyone for the replies, I have deployed 0.11.3 RC3. I will try
> that before I try altering the level compaction.
>
> Thanks
> Trevor Hart
>
>
>
> ---- On Fri, 09 Apr 2021 23:29:07 +1200 *Xiangdong Huang
> <sainthxd@gmail.com <sa...@gmail.com>>* wrote ----
>
> And remember to set the strategy back after diagnosis...
>
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao <qi...@apache.org> 于2021年4月9日周五 下午7:24写道:
>
> > Hi,
> >
> > This may due to the Compaction. To verify the problem, change the
> compaction_strategy
> > to NO_COMPACTION.
> >
> > This is fixed in the 0.11.3 RC3[1]. you could try this version :)
> >
> > [1] https://dist.apache.org/repos/dist/dev/iotdb/0.11.3/rc3
> >
> > Thanks,
> > —————————————————
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> >
> > Trevor Hart <tr...@ope.nz> 于2021年4月9日周五 上午10:29写道:
> >
> >> Hi Team,
> >>
> >> I am still struggling with high CPU usage on V11.2. My CPU usage creeps
> >> up after a few days and basically got to 100% today and was unusable.
> >>
> >> If I restart the iotDB server then its fine for a few days. This Linux
> >> server has 2 cores and 4Gb RAM. The database is running 24x7.
> >>
> >> The data ingestion is typically small. Maybe a 500 points per minute. I
> >> can never see any data ingestion backing up unless the CPU is over 90%.
> >>
> >> Questions;
> >>
> >> 1. Does anyone else see this behaviour?
> >> 2. Are there any maintenance tasks I should call daily? Should I be
> >> calling Flush daily?
> >> 3. Should I reduce the WAL threshold?
> >> 4. I am using compaction_strategy=LEVEL_COMPACTION
> >> 5. And have set enable_unseq_compaction=false
> >>
> >> Is there anything else I can try or do I just need bigger hardware?
> >>
> >>
> >>
> >>
> >>
> >> Thanks
> >> Trevor
> >>
> >>
> >>
>
>
>
>
>
Re: High CPU Usage
Posted by Trevor Hart <tr...@ope.nz>.
0.11.3 RC3 seems much happier so far. There was an initial spike in CPU after I upgradef to 11.3 but it has now settled down.I will keep an eye on it over the next week - see attachment. Thanks Trevor Hart ---- On Sat, 10 Apr 2021 15:45:36 +1200 trevor@ope.nz wrote ----Thanks everyone for the replies, I have deployed 0.11.3 RC3. I will try that before I try altering the level compaction.Thanks Trevor Hart---- On Fri, 09 Apr 2021 23:29:07 +1200 Xiangdong Huang <sa...@gmail.com> wrote ----And remember to set the strategy back after diagnosis... ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao <qi...@apache.org> 于2021年4月9日周五 下午7:24写道: > Hi, > > This may due to the Compaction. To verify the problem, change the compaction_strategy > to NO_COMPACTION. > > This is fixed in the 0.11.3 RC3[1]. you could try this version :) > > [1] https://dist.apache.org/repos/dist/dev/iotdb/0.11.3/rc3 > > Thanks, > ————————————————— > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > Trevor Hart <tr...@ope.nz> 于2021年4月9日周五 上午10:29写道: > >> Hi Team, >> >> I am still struggling with high CPU usage on V11.2. My CPU usage creeps >> up after a few days and basically got to 100% today and was unusable. >> >> If I restart the iotDB server then its fine for a few days. This Linux >> server has 2 cores and 4Gb RAM. The database is running 24x7. >> >> The data ingestion is typically small. Maybe a 500 points per minute. I >> can never see any data ingestion backing up unless the CPU is over 90%. >> >> Questions; >> >> 1. Does anyone else see this behaviour? >> 2. Are there any maintenance tasks I should call daily? Should I be >> calling Flush daily? >> 3. Should I reduce the WAL threshold? >> 4. I am using compaction_strategy=LEVEL_COMPACTION >> 5. And have set enable_unseq_compaction=false >> >> Is there anything else I can try or do I just need bigger hardware? >> >> >> >> >> >> Thanks >> Trevor >> >> >>
Re: High CPU Usage
Posted by Trevor Hart <tr...@ope.nz>.
Thanks everyone for the replies, I have deployed 0.11.3 RC3. I will try that before I try altering the level compaction.
Thanks
Trevor Hart
---- On Fri, 09 Apr 2021 23:29:07 +1200 Xiangdong Huang <sa...@gmail.com> wrote ----
And remember to set the strategy back after diagnosis...
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Jialin Qiao <ma...@apache.org> 于2021年4月9日周五 下午7:24写道:
> Hi,
>
> This may due to the Compaction. To verify the problem, change the compaction_strategy
> to NO_COMPACTION.
>
> This is fixed in the 0.11.3 RC3[1]. you could try this version :)
>
> [1] https://dist.apache.org/repos/dist/dev/iotdb/0.11.3/rc3
>
> Thanks,
> —————————————————
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>
> Trevor Hart <ma...@ope.nz> 于2021年4月9日周五 上午10:29写道:
>
>> Hi Team,
>>
>> I am still struggling with high CPU usage on V11.2. My CPU usage creeps
>> up after a few days and basically got to 100% today and was unusable.
>>
>> If I restart the iotDB server then its fine for a few days. This Linux
>> server has 2 cores and 4Gb RAM. The database is running 24x7.
>>
>> The data ingestion is typically small. Maybe a 500 points per minute. I
>> can never see any data ingestion backing up unless the CPU is over 90%.
>>
>> Questions;
>>
>> 1. Does anyone else see this behaviour?
>> 2. Are there any maintenance tasks I should call daily? Should I be
>> calling Flush daily?
>> 3. Should I reduce the WAL threshold?
>> 4. I am using compaction_strategy=LEVEL_COMPACTION
>> 5. And have set enable_unseq_compaction=false
>>
>> Is there anything else I can try or do I just need bigger hardware?
>>
>>
>>
>>
>>
>> Thanks
>> Trevor
>>
>>
>>
Re: High CPU Usage
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
> Are there any maintenance tasks I should call daily? Should I be calling
Flush daily?
No need
>Should I reduce the WAL threshold?
No need
If you still meet this problem in 0.11.3 or after setting compaction to
NO_COMPACTION,
please use VisualVM or JProfiler to record the CPU while it is 100%, and
share the snapshot. Then, we will find the problem.
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Xiangdong Huang <sa...@gmail.com> 于2021年4月9日周五 下午7:29写道:
> And remember to set the strategy back after diagnosis...
>
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao <qi...@apache.org> 于2021年4月9日周五 下午7:24写道:
>
> > Hi,
> >
> > This may due to the Compaction. To verify the problem, change the
> compaction_strategy
> > to NO_COMPACTION.
> >
> > This is fixed in the 0.11.3 RC3[1]. you could try this version :)
> >
> > [1] https://dist.apache.org/repos/dist/dev/iotdb/0.11.3/rc3
> >
> > Thanks,
> > —————————————————
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> >
> > Trevor Hart <tr...@ope.nz> 于2021年4月9日周五 上午10:29写道:
> >
> >> Hi Team,
> >>
> >> I am still struggling with high CPU usage on V11.2. My CPU usage creeps
> >> up after a few days and basically got to 100% today and was unusable.
> >>
> >> If I restart the iotDB server then its fine for a few days. This Linux
> >> server has 2 cores and 4Gb RAM. The database is running 24x7.
> >>
> >> The data ingestion is typically small. Maybe a 500 points per minute. I
> >> can never see any data ingestion backing up unless the CPU is over 90%.
> >>
> >> Questions;
> >>
> >> 1. Does anyone else see this behaviour?
> >> 2. Are there any maintenance tasks I should call daily? Should I be
> >> calling Flush daily?
> >> 3. Should I reduce the WAL threshold?
> >> 4. I am using compaction_strategy=LEVEL_COMPACTION
> >> 5. And have set enable_unseq_compaction=false
> >>
> >> Is there anything else I can try or do I just need bigger hardware?
> >>
> >>
> >>
> >>
> >>
> >> Thanks
> >> Trevor
> >>
> >>
> >>
>
Re: High CPU Usage
Posted by Xiangdong Huang <sa...@gmail.com>.
And remember to set the strategy back after diagnosis...
-----------------------------------
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Jialin Qiao <qi...@apache.org> 于2021年4月9日周五 下午7:24写道:
> Hi,
>
> This may due to the Compaction. To verify the problem, change the compaction_strategy
> to NO_COMPACTION.
>
> This is fixed in the 0.11.3 RC3[1]. you could try this version :)
>
> [1] https://dist.apache.org/repos/dist/dev/iotdb/0.11.3/rc3
>
> Thanks,
> —————————————————
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>
> Trevor Hart <tr...@ope.nz> 于2021年4月9日周五 上午10:29写道:
>
>> Hi Team,
>>
>> I am still struggling with high CPU usage on V11.2. My CPU usage creeps
>> up after a few days and basically got to 100% today and was unusable.
>>
>> If I restart the iotDB server then its fine for a few days. This Linux
>> server has 2 cores and 4Gb RAM. The database is running 24x7.
>>
>> The data ingestion is typically small. Maybe a 500 points per minute. I
>> can never see any data ingestion backing up unless the CPU is over 90%.
>>
>> Questions;
>>
>> 1. Does anyone else see this behaviour?
>> 2. Are there any maintenance tasks I should call daily? Should I be
>> calling Flush daily?
>> 3. Should I reduce the WAL threshold?
>> 4. I am using compaction_strategy=LEVEL_COMPACTION
>> 5. And have set enable_unseq_compaction=false
>>
>> Is there anything else I can try or do I just need bigger hardware?
>>
>>
>>
>>
>>
>> Thanks
>> Trevor
>>
>>
>>
Re: High CPU Usage
Posted by Jialin Qiao <qi...@apache.org>.
Hi,
This may due to the Compaction. To verify the problem, change the
compaction_strategy
to NO_COMPACTION.
This is fixed in the 0.11.3 RC3[1]. you could try this version :)
[1] https://dist.apache.org/repos/dist/dev/iotdb/0.11.3/rc3
Thanks,
—————————————————
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
Trevor Hart <tr...@ope.nz> 于2021年4月9日周五 上午10:29写道:
> Hi Team,
>
> I am still struggling with high CPU usage on V11.2. My CPU usage creeps up
> after a few days and basically got to 100% today and was unusable.
>
> If I restart the iotDB server then its fine for a few days. This Linux
> server has 2 cores and 4Gb RAM. The database is running 24x7.
>
> The data ingestion is typically small. Maybe a 500 points per minute. I
> can never see any data ingestion backing up unless the CPU is over 90%.
>
> Questions;
>
> 1. Does anyone else see this behaviour?
> 2. Are there any maintenance tasks I should call daily? Should I be
> calling Flush daily?
> 3. Should I reduce the WAL threshold?
> 4. I am using compaction_strategy=LEVEL_COMPACTION
> 5. And have set enable_unseq_compaction=false
>
> Is there anything else I can try or do I just need bigger hardware?
>
>
>
>
>
> Thanks
> Trevor
>
>
>