You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Kiran Kumar Nandihalli Shetru (CONT)" <ki...@capitalone.com> on 2018/12/27 20:51:30 UTC
Drill performance tuning
Hi Team,
I have the below condition:
1. I have a cluster of ec2 instances with 10 nodes of m4.4x large size.
Meaning, each m4.4x large node would have 16 vCPUs and 64 G memory with 300
IOPs. Since I mentioned in override-conf all the 10 IPs, and because I am
able to see all the drill instances in one page, if i go to one node
instance IP in the UI (http://one of the IPs in 10 nodes:8047), I am assume
i am running drill in cluster of 10 ec2 instances/nodes
2. I have installed zookeeper on 9 of those same 10 nodes and 1 drillbit in
each of those 10 nodes. It means, I am running 10 drillbits in 10 node
cluster. 9 instances of zookeeper in the same 10 nodes cluster.
3. free -h command on each ec2 instance/node always varies depending what I
am doing on that instance
4. However the memory configurations always gives me different messages on
setting and available free memory or max proc memory.
[WARN] You have an allocation of 26 GB that is currently unused from a
total of 64 GB. You can increase your existing memory configuration to use
this extra memory
DRILLBIT_MAX_PROC_MEM=64G
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=29G
DRILLBIT_CODE_CACHE_SIZE=1024m
if I change the config according to the above message, sometimes i get this
[INFO] Attempting to start up Drill with the following settings
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=39G
DRILLBIT_CODE_CACHE_SIZE=1024m
[WARN] Total Memory Allocation for Drillbit (48GB) exceeds available free
memory (45GB)
[WARN] Drillbit will start up, but can potentially crash due to
oversubscribing of system memory.
# free -h
total used free shared buff/cache
available
Mem: 62G 7.0G 45G 73M 10G
54G
Swap: 0B 0B 0B
So, can't really determine on what basis the calculations are done. Whether
it is based out free memory or on set max proc mem.
However, i have these questions.
1. For an ec2 instance which has 64 GB memory, what should be the value of
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=39G
DRILLBIT_CODE_CACHE_SIZE=1024m
planner.memory.max_query_memory_per_node
planner.memory.percent_per_query
planner.width.max_per_node
drill.exec.memory.operator.output_batch_size
Are there any additional settings that I need to consider, to get the
full benefit of large cluster that I have, in order to get faster results.
The data size I want to work upon is in terms of TBs, but currently i am
not able to execute queries on MBs of partioned tables also.
To create a table with 7 GB data, it takes 140.493 seconds
To create partitioned table with 7 GB data, it takes 4029.633 seconds
To select values from a partitioned table, i get this 3,036 rows selected
(34.639 seconds)
2. How does drill distribute this configuration across all the nodes? I
mean, if i provide 39 GB of direct memory, is it only to that node or the
whole cluster? why direct memory is not shown in UI as per the parameter
that we set in drill-env.sh
3. I had the problem of select query execution earlier, that said i can't
find XYZ file across cluster. So, i configured EFS for all the data files
to reside in one place that is now mounted to all the nodes in the cluster.
Some times, I can't select values from partitioned tables. The query
doesn't respond at all.
Response above would be highly appreciated.
--
*Kind Regards*
*Kiran Kumar NS*
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Drill performance tuning
Posted by "Kiran Kumar Nandihalli Shetru (CONT)" <ki...@capitalone.com>.
Hi Team,
>
> I have the below condition:
>
> 1. I have a cluster of ec2 instances with 10 nodes of m4.4x large size.
> Meaning, each m4.4x large node would have 16 vCPUs and 64 G memory with 300
> IOPs. Since I mentioned in override-conf all the 10 IPs, and because I am
> able to see all the drill instances in one page, if i go to one node
> instance IP in the UI (http://one of the IPs in 10 nodes:8047), I am
> assume i am running drill in cluster of 10 ec2 instances/nodes
> 2. I have installed zookeeper on 9 of those same 10 nodes and 1 drillbit
> in each of those 10 nodes. It means, I am running 10 drillbits in 10 node
> cluster. 9 instances of zookeeper in the same 10 nodes cluster.
> 3. free -h command on each ec2 instance/node always varies depending what
> I am doing on that instance
> 4. However the memory configurations always gives me different messages on
> setting and available free memory or max proc memory.
>
> [WARN] You have an allocation of 26 GB that is currently unused from a
> total of 64 GB. You can increase your existing memory configuration to use
> this extra memory
>
> DRILLBIT_MAX_PROC_MEM=64G
>
> DRILL_HEAP=8G
>
> DRILL_MAX_DIRECT_MEMORY=29G
>
> DRILLBIT_CODE_CACHE_SIZE=1024m
>
>
> if I change the config according to the above message, sometimes i get this
>
> [INFO] Attempting to start up Drill with the following settings
>
> DRILL_HEAP=8G
>
> DRILL_MAX_DIRECT_MEMORY=39G
>
> DRILLBIT_CODE_CACHE_SIZE=1024m
>
> [WARN] Total Memory Allocation for Drillbit (48GB) exceeds available free
> memory (45GB)
>
> [WARN] Drillbit will start up, but can potentially crash due to
> oversubscribing of system memory.
>
> # free -h
>
> total used free shared buff/cache
> available
>
> Mem: 62G 7.0G 45G 73M 10G
> 54G
>
> Swap: 0B 0B 0B
>
> So, can't really determine on what basis the calculations are done.
> Whether it is based out free memory or on set max proc mem.
>
> However, i have these questions.
> 1. For an ec2 instance which has 64 GB memory, what should be the value of
>
> DRILL_HEAP=8G
>
> DRILL_MAX_DIRECT_MEMORY=39G
>
> DRILLBIT_CODE_CACHE_SIZE=1024m
>
> planner.memory.max_query_memory_per_node
>
> planner.memory.percent_per_query
>
> planner.width.max_per_node
>
> drill.exec.memory.operator.output_batch_size
>
>
> Are there any additional settings that I need to consider, to get the
> full benefit of large cluster that I have, in order to get faster results.
> The data size I want to work upon is in terms of TBs, but currently i am
> not able to execute queries on MBs of partioned tables also.
>
>
> To create a table with 7 GB data, it takes 140.493 seconds
>
> To create partitioned table with 7 GB data, it takes 4029.633 seconds
>
> To select values from a partitioned table, i get this 3,036 rows selected
> (34.639 seconds)
>
>
>
> 2. How does drill distribute this configuration across all the nodes? I
> mean, if i provide 39 GB of direct memory, is it only to that node or the
> whole cluster? why direct memory is not shown in UI as per the parameter
> that we set in drill-env.sh
>
>
> 3. I had the problem of select query execution earlier, that said i can't
> find XYZ file across cluster. So, i configured EFS for all the data files
> to reside in one place that is now mounted to all the nodes in the cluster.
> Some times, I can't select values from partitioned tables. The query
> doesn't respond at all.
>
> Response above would be highly appreciated.
>
> --
> *Kind Regards*
> *Kiran Kumar NS*
>
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.