You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by Manuel Sopena Ballesteros <ma...@garvan.org.au> on 2019/04/19 11:37:34 UTC
tablet server not using all resources availables
Dear Kudu community,
We have a test kudu cluster with 3 vms and each one of them has a master and a tablet server.
Each vm has:
x50 vcpus
500 MB RAM
4TB virtual disks with xfs on local hypervisor nvme drives
We started an ETL process using apache spark and realised that the vms are not fully utilised (~15 vcpus are used and the iowait time is 0). That tell us that kudu is not using all the resources available on the machine and the system is not fully optimised.
Question:
How could we configure the tablet server to fully utilize the resources available?
Is it a good practice to run multiple tablet server on the same vm?
thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
RE: tablet server not using all resources availables
Posted by Manuel Sopena Ballesteros <ma...@garvan.org.au>.
Hi Adar,
Apologies, 500MB of RAM was a type, the real unit is 500GB of RAM
I will play with parameters maintenance_manager_num_thread and will provide feedback
Thank you very much
Manuel
-----Original Message-----
From: Adar Lieber-Dembo [mailto:adar@cloudera.com]
Sent: Monday, April 22, 2019 6:57 AM
To: user@kudu.apache.org
Subject: Re: tablet server not using all resources availables
500 MB RAM per VM is very low. Is this a typo? If not, it's the first
thing you should tackle.
As for your questions:
- You can look into increasing --maintenance_manager_num_threads. We
typically recommend a 1:3 ratio of threads to number of disks, but
that's when using spinning disks; with fast nvme disks, you can
explore 1:1, 2:1, or higher. If each VM has multiple disks, you should
let Kudu stripe its data across all of the disks (via --fs_data_dirs).
You should also make sure that your tables are sufficiently
partitioned so as to take advantage of every machine in the cluster. I
recommend you read through
https://kudu.apache.org/docs/schema_design.html if you haven't
already.
- You should definitely NOT run multiple tservers on the same VM.
Besides crippling your fault tolerance (unless you go out of your way
to set up location awareness), it's just not necessary: each tserver
is architected to take advantage of all of the machine's resources if
you let it).
On Fri, Apr 19, 2019 at 4:37 AM Manuel Sopena Ballesteros
<ma...@garvan.org.au> wrote:
>
> Dear Kudu community,
>
> We have a test kudu cluster with 3 vms and each one of them has a master and a tablet server.
>
> Each vm has:
> x50 vcpus
> 500 MB RAM
> 4TB virtual disks with xfs on local hypervisor nvme drives
>
> We started an ETL process using apache spark and realised that the vms are not fully utilised (~15 vcpus are used and the iowait time is 0). That tell us that kudu is not using all the resources available on the machine and the system is not fully optimised.
>
> Question:
>
> How could we configure the tablet server to fully utilize the resources available?
> Is it a good practice to run multiple tablet server on the same vm?
>
> thank you very much
>
>
>
>
> NOTICE
> Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
Re: tablet server not using all resources availables
Posted by Adar Lieber-Dembo <ad...@cloudera.com>.
500 MB RAM per VM is very low. Is this a typo? If not, it's the first
thing you should tackle.
As for your questions:
- You can look into increasing --maintenance_manager_num_threads. We
typically recommend a 1:3 ratio of threads to number of disks, but
that's when using spinning disks; with fast nvme disks, you can
explore 1:1, 2:1, or higher. If each VM has multiple disks, you should
let Kudu stripe its data across all of the disks (via --fs_data_dirs).
You should also make sure that your tables are sufficiently
partitioned so as to take advantage of every machine in the cluster. I
recommend you read through
https://kudu.apache.org/docs/schema_design.html if you haven't
already.
- You should definitely NOT run multiple tservers on the same VM.
Besides crippling your fault tolerance (unless you go out of your way
to set up location awareness), it's just not necessary: each tserver
is architected to take advantage of all of the machine's resources if
you let it).
On Fri, Apr 19, 2019 at 4:37 AM Manuel Sopena Ballesteros
<ma...@garvan.org.au> wrote:
>
> Dear Kudu community,
>
> We have a test kudu cluster with 3 vms and each one of them has a master and a tablet server.
>
> Each vm has:
> x50 vcpus
> 500 MB RAM
> 4TB virtual disks with xfs on local hypervisor nvme drives
>
> We started an ETL process using apache spark and realised that the vms are not fully utilised (~15 vcpus are used and the iowait time is 0). That tell us that kudu is not using all the resources available on the machine and the system is not fully optimised.
>
> Question:
>
> How could we configure the tablet server to fully utilize the resources available?
> Is it a good practice to run multiple tablet server on the same vm?
>
> thank you very much
>
>
>
>
> NOTICE
> Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.