You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by nawfal osman <na...@corporate-groupe.com> on 2021/09/23 07:44:23 UTC
RE: Soucis Drill
Hi Support
Below the drill issues that we have
1. Slowness issues when running two queries the same time ( please find attached an Example of querie)
Start of execution querie 20210815: 09: 26: 23 to 09: 33 ==> about 6min30s
Start of execution of the date 15 and 16 has been launched at the same time: 09:38 to 09:48 ==> 10 minutes of exécution
2 - Sometimes we get the error (Unable to allocate buffer of size 134217728 due to memory limit (191739611). Current allocation: 78880768)
Can you please help up to troubleshoot theses issues.
Kind Regards
________________________________
De : Amine Elmekki <am...@corporate-groupe.com>
Envoyé : mercredi 22 septembre 2021 09:08
À : nawfal osman <na...@corporate-groupe.com>
Cc : Zakaria Rahoui <za...@corporate-groupe.com>
Objet : RE: Soucis Drill
Merci
Tu envois ces éléments à user@drill.apache.org<ma...@drill.apache.org>
From: nawfal osman <na...@corporate-groupe.com>
Sent: Tuesday, 21 September 2021 10:15
To: Amine Elmekki <am...@corporate-groupe.com>
Cc: Zakaria Rahoui <za...@corporate-groupe.com>
Subject: RE: Soucis Drill
Bonjour ;
Ci-dessous mon retour :
1. Pb de lenteur lorsqu’on lance deux projets en //.
Eléments à envoyer : Request, date début, date fin (en la lançant seule et avec deux req en //)
Début d'exécution de la journée du 20210815 : 09 :26 :23 au 09 :33 ==> environ 6min30s
Début d'exécution quand on a lancé la journée du 15 et 16 en même temps : 09:38 à 09:48 ==> 10 minutes d'exécution
En jointure la requête utilisée
1. Problème de jointure de drill. Eléments à envoyer :
* Requête
* Fausse résultat
* Correct résultat qu’on doit obtenir
Pour ce cas de figure, je n'arrive toujours pas à trouver un cas ; je suis toujours en train de chercher ( je vous ferai un retour dans la journée )
3.D’autres problèmes…
* Parfois on a l'erreur ( Unable to allocate buffer of size 134217728 due to memory limit (191739611). Current allocation: 78880768 ) ; mais le support MapR nous a envoyer un lien pour troubleshooting : Troubleshooting Guide: DRILL OutOfMemory error and possible workarounds
https://support.datafabric.hpe.com/s/article/Troubleshooting-DRILL-OutOfMemory-error-and-possible-workarounds?language=en_US
Cordialement
________________________________
De : Amine Elmekki <am...@corporate-groupe.com>>
Envoyé : lundi 20 septembre 2021 18:31
À : nawfal osman <na...@corporate-groupe.com>>
Cc : Zakaria Rahoui <za...@corporate-groupe.com>>
Objet : Soucis Drill
Nawfal,
Besoin des cas réel des problèmes de drill :
1. Pb de lenteur lorsqu’on lance deux projets en //.
Eléments à envoyer : Request, date début, date fin (en la lançant seule et avec deux req en //)
1. Problème de jointure de drill. Eléments à envoyer :
* Requête
* Fausse résultat
* Correct résultat qu’on doit obtenir
1. D’autres problèmes…
Re: Soucis Drill
Posted by James Turton <dz...@apache.org>.
I think you probably mean options like
planner.width.max_per_node
planner.width.max_per_query
in your last response, given you mention values of 3 and 28.
That aside, I can't see anything in the query profile you shared that
should be any obstacle for your cluster, or even for a single laptop. I
would debug this by investigating memory allocation (including to
processes other than Drill) on the host machines, as well as swap
thrashing. Note that a lot of this investigation is at the operating
system level, rather than in Drill.
Out of interest, you can also try copying the CSV files you're querying
here to your laptop, or some other idle machine, where you query them
with drill-embedded. This last step would give you an answer to: "how
quickly can a small, unloaded, single node Drill complete this query?".
I expect less than 30 seconds, with time to write the results out being
a big part of that.
On 2021/09/28 18:11, nawfal osman wrote:
> Hi
>
> Please find atatched the json profile of the query
>
>
> Have you made any changes to Drill memory options such
> as|planner.memory.max_query_memory_per_node|?
>
> Answer : yes we use this parameter sometimes we ran queries with
> the parameter '3' and some other query with '28'
>
>
> Kinds Regards :
> ------------------------------------------------------------------------
> *De :* James Turton <dz...@apache.org>
> *Envoyé :* mardi 28 septembre 2021 14:52
> *À :* user@drill.apache.org <us...@drill.apache.org>; nawfal osman
> <na...@corporate-groupe.com>
> *Cc :* Zakaria Rahoui <za...@corporate-groupe.com>; Amine
> Elmekki <am...@corporate-groupe.com>
> *Objet :* Re: Soucis Drill
> Plenty of direct memory there, 50Gb on every Drillbit, far in excess
> of the modest 134Mb that Drill says it can't get. Can you share a
> JSON profile from an execution of this query (Web UI -> Profiles ->
> Completed queries -> (choose query) -> Full JSON Profile?
>
> Have you made any changes to Drill memory options such as
> |planner.memory.max_query_memory_per_node|?
>
> On 2021/09/28 13:55, nawfal osman wrote:
>> Hi
>>
>> Please notice we have 10 nodes (mapr11-mapr15 ) ( mapr31-35)
>>
>> please find below the output of the command : free -h of the node
>> mapr15 :
>> [mapr@mapr15 conf]$ free -h
>> total used free shared buff/cache
>> available
>> Mem: 125G 97G 3.1G 1.3G 25G
>> 24G
>> Swap: 117G 54G
>>
>>
>>
>> Find below the output of the command svc :
>> [mapr@mapr15 conf]$ maprcli node list -columns svc
>> 2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ vmsplice
>> 406 err = 1 10.148.0.73:7222
>>
>>
>> hostname service ip
>> mapr11
>> cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver
>> 10.148.0.44,10.32.243.44
>> mapr12
>> data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver
>> 10.148.0.45,10.32.243.45
>> mapr13
>> httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver
>> 10.148.0.46,10.32.243.46
>> mapr14
>> cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
>> 10.148.0.47
>> mapr15
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.48,10.32.243.48
>> mapr31
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.70
>> mapr32
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.71
>> mapr33
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.72
>> mapr34
>> cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
>> 10.148.0.73
>> mapr35
>> data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>>
>>
>> For the memory setting of drill ; please find attached the conf of
>> all nodes
>>
>>
>>
>> Kind Regards;
>
RE: Soucis Drill
Posted by nawfal osman <na...@corporate-groupe.com>.
Hi
Please find atatched the json profile of the query
Have you made any changes to Drill memory options such as planner.memory.max_query_memory_per_node?
Answer : yes we use this parameter sometimes we ran queries with the parameter '3' and some other query with '28'
Kinds Regards :
________________________________
De : James Turton <dz...@apache.org>
Envoyé : mardi 28 septembre 2021 14:52
À : user@drill.apache.org <us...@drill.apache.org>; nawfal osman <na...@corporate-groupe.com>
Cc : Zakaria Rahoui <za...@corporate-groupe.com>; Amine Elmekki <am...@corporate-groupe.com>
Objet : Re: Soucis Drill
Plenty of direct memory there, 50Gb on every Drillbit, far in excess of the modest 134Mb that Drill says it can't get. Can you share a JSON profile from an execution of this query (Web UI -> Profiles -> Completed queries -> (choose query) -> Full JSON Profile?
Have you made any changes to Drill memory options such as planner.memory.max_query_memory_per_node?
On 2021/09/28 13:55, nawfal osman wrote:
Hi
Please notice we have 10 nodes (mapr11-mapr15 ) ( mapr31-35)
please find below the output of the command : free -h of the node mapr15 :
[mapr@mapr15 conf]$ free -h
total used free shared buff/cache available
Mem: 125G 97G 3.1G 1.3G 25G 24G
Swap: 117G 54G
Find below the output of the command svc :
[mapr@mapr15 conf]$ maprcli node list -columns svc
2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ vmsplice 406 err = 1 10.148.0.73:7222
hostname service ip
mapr11 cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver 10.148.0.44,10.32.243.44
mapr12 data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver 10.148.0.45,10.32.243.45
mapr13 httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver 10.148.0.46,10.32.243.46
mapr14 cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver 10.148.0.47
mapr15 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.48,10.32.243.48
mapr31 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.70
mapr32 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.71
mapr33 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.72
mapr34 cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver 10.148.0.73
mapr35 data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
For the memory setting of drill ; please find attached the conf of all nodes
Kind Regards;
Re: Soucis Drill
Posted by James Turton <dz...@apache.org>.
I do also see mapr15 reporting a *lot* of data in swap. It looks like
the RAM on that box could be over allocated. I haven't tested this but
I'm pretty sure it's going to be better to let Drill intentionally spill
when it thinks RAM is constrained than to tell it can have more RAM than
there is available and let it thrash the swap partition. Swapping
during query processing could also be consistent with your point 1
(performance degradation when two queries are run) if the two queries
are competing for seeks from the storage device hosting the swap. I
recommend running `smem -atk` to see which processes have been swapped
out and watching a monitoring tool that can show swapping counters like
major page faults per second during query execution.
On 2021/09/28 15:52, James Turton wrote:
> Plenty of direct memory there, 50Gb on every Drillbit, far in excess
> of the modest 134Mb that Drill says it can't get. Can you share a
> JSON profile from an execution of this query (Web UI -> Profiles ->
> Completed queries -> (choose query) -> Full JSON Profile?
>
> Have you made any changes to Drill memory options such as
> |planner.memory.max_query_memory_per_node|?
>
> On 2021/09/28 13:55, nawfal osman wrote:
>> Hi
>>
>> Please notice we have 10 nodes (mapr11-mapr15 ) ( mapr31-35)
>>
>> please find below the output of the command : free -h of the node
>> mapr15 :
>> [mapr@mapr15 conf]$ free -h
>> total used free shared buff/cache
>> available
>> Mem: 125G 97G 3.1G 1.3G 25G
>> 24G
>> Swap: 117G 54G
>>
>>
>>
>> Find below the output of the command svc :
>> [mapr@mapr15 conf]$ maprcli node list -columns svc
>> 2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ vmsplice
>> 406 err = 1 10.148.0.73:7222
>>
>>
>> hostname service ip
>> mapr11
>> cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver
>> 10.148.0.44,10.32.243.44
>> mapr12
>> data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver
>> 10.148.0.45,10.32.243.45
>> mapr13
>> httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver
>> 10.148.0.46,10.32.243.46
>> mapr14
>> cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
>> 10.148.0.47
>> mapr15
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.48,10.32.243.48
>> mapr31
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.70
>> mapr32
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.71
>> mapr33
>> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>> 10.148.0.72
>> mapr34
>> cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
>> 10.148.0.73
>> mapr35
>> data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>>
>>
>> For the memory setting of drill ; please find attached the conf of
>> all nodes
>>
>>
>>
>> Kind Regards;
>
>
Re: Soucis Drill
Posted by James Turton <dz...@apache.org>.
Plenty of direct memory there, 50Gb on every Drillbit, far in excess of
the modest 134Mb that Drill says it can't get. Can you share a JSON
profile from an execution of this query (Web UI -> Profiles -> Completed
queries -> (choose query) -> Full JSON Profile?
Have you made any changes to Drill memory options such as
|planner.memory.max_query_memory_per_node|?
On 2021/09/28 13:55, nawfal osman wrote:
> Hi
>
> Please notice we have 10 nodes (mapr11-mapr15 ) ( mapr31-35)
>
> please find below the output of the command : free -h of the node
> mapr15 :
> [mapr@mapr15 conf]$ free -h
> total used free shared buff/cache
> available
> Mem: 125G 97G 3.1G 1.3G 25G
> 24G
> Swap: 117G 54G
>
>
>
> Find below the output of the command svc :
> [mapr@mapr15 conf]$ maprcli node list -columns svc
> 2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ vmsplice 406
> err = 1 10.148.0.73:7222
>
>
> hostname service ip
> mapr11
> cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver
> 10.148.0.44,10.32.243.44
> mapr12
> data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver
> 10.148.0.45,10.32.243.45
> mapr13
> httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver
> 10.148.0.46,10.32.243.46
> mapr14
> cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
> 10.148.0.47
> mapr15
> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
> 10.148.0.48,10.32.243.48
> mapr31
> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
> 10.148.0.70
> mapr32
> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
> 10.148.0.71
> mapr33
> fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
> 10.148.0.72
> mapr34
> cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver
> 10.148.0.73
> mapr35
> data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
>
>
> For the memory setting of drill ; please find attached the conf of
> all nodes
>
>
>
> Kind Regards;
RE: Soucis Drill
Posted by nawfal osman <na...@corporate-groupe.com>.
Hi
Please notice we have 10 nodes (mapr11-mapr15 ) ( mapr31-35)
please find below the output of the command : free -h of the node mapr15 :
[mapr@mapr15 conf]$ free -h
total used free shared buff/cache available
Mem: 125G 97G 3.1G 1.3G 25G 24G
Swap: 117G 54G
Find below the output of the command svc :
[mapr@mapr15 conf]$ maprcli node list -columns svc
2021-09-28 12:42:30,6130 INFO:Unable to set F_SETPIPE_SZ vmsplice 406 err = 1 10.148.0.73:7222
hostname service ip
mapr11 cldb,data-access-gateway,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,fileserver,nfs,gateway,apiserver 10.148.0.44,10.32.243.44
mapr12 data-access-gateway,fileserver,elasticsearch,mastgateway,nodemanager,kibana,drill-bits,collectd,hoststats,fluentd,grafana,resourcemanager,historyserver,nfs,opentsdb,gateway,apiserver 10.148.0.45,10.32.243.45
mapr13 httpfs,hs2,data-access-gateway,fileserver,mastgateway,nodemanager,kibana,hcat,drill-bits,hoststats,fluentd,hivemeta,grafana,hue,nfs,gateway,apiserver 10.148.0.46,10.32.243.46
mapr14 cldb,mastgateway,nodemanager,drill-bits,hbasethrift,oozie,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver 10.148.0.47
mapr15 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.48,10.32.243.48
mapr31 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.70
mapr32 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.71
mapr33 fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver 10.148.0.72
mapr34 cldb,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,fileserver,nfs,apiserver 10.148.0.73
mapr35 data-access-gateway,fileserver,mastgateway,nodemanager,drill-bits,collectd,hoststats,fluentd,hbaserest,nfs,apiserver
For the memory setting of drill ; please find attached the conf of all nodes
Kind Regards;
________________________________
De : James Turton <dz...@apache.org>
Envoyé : mardi 28 septembre 2021 11:50
À : user@drill.apache.org <us...@drill.apache.org>; nawfal osman <na...@corporate-groupe.com>
Cc : Zakaria Rahoui <za...@corporate-groupe.com>; Amine Elmekki <am...@corporate-groupe.com>
Objet : Re: Soucis Drill
I would try to resolve 2 first, since a major contributor to 1 could be spilling to disk. What memory settings are running Drill with?
On 2021/09/23 09:44, nawfal osman wrote:
Hi Support
Below the drill issues that we have
1. Slowness issues when running two queries the same time ( please find attached an Example of querie)
Start of execution querie 20210815: 09: 26: 23 to 09: 33 ==> about 6min30s
Start of execution of the date 15 and 16 has been launched at the same time: 09:38 to 09:48 ==> 10 minutes of exécution
2 - Sometimes we get the error (Unable to allocate buffer of size 134217728 due to memory limit (191739611). Current allocation: 78880768)
Can you please help up to troubleshoot theses issues.
Kind Regards
________________________________
De : Amine Elmekki <am...@corporate-groupe.com>
Envoyé : mercredi 22 septembre 2021 09:08
À : nawfal osman <na...@corporate-groupe.com>
Cc : Zakaria Rahoui <za...@corporate-groupe.com>
Objet : RE: Soucis Drill
Merci
Tu envois ces éléments à user@drill.apache.org<ma...@drill.apache.org>
From: nawfal osman <na...@corporate-groupe.com>
Sent: Tuesday, 21 September 2021 10:15
To: Amine Elmekki <am...@corporate-groupe.com>
Cc: Zakaria Rahoui <za...@corporate-groupe.com>
Subject: RE: Soucis Drill
Bonjour ;
Ci-dessous mon retour :
1. Pb de lenteur lorsqu’on lance deux projets en //.
Eléments à envoyer : Request, date début, date fin (en la lançant seule et avec deux req en //)
Début d'exécution de la journée du 20210815 : 09 :26 :23 au 09 :33 ==> environ 6min30s
Début d'exécution quand on a lancé la journée du 15 et 16 en même temps : 09:38 à 09:48 ==> 10 minutes d'exécution
En jointure la requête utilisée
1. Problème de jointure de drill. Eléments à envoyer :
* Requête
* Fausse résultat
* Correct résultat qu’on doit obtenir
Pour ce cas de figure, je n'arrive toujours pas à trouver un cas ; je suis toujours en train de chercher ( je vous ferai un retour dans la journée )
3.D’autres problèmes…
* Parfois on a l'erreur ( Unable to allocate buffer of size 134217728 due to memory limit (191739611). Current allocation: 78880768 ) ; mais le support MapR nous a envoyer un lien pour troubleshooting : Troubleshooting Guide: DRILL OutOfMemory error and possible workarounds
https://support.datafabric.hpe.com/s/article/Troubleshooting-DRILL-OutOfMemory-error-and-possible-workarounds?language=en_US
Cordialement
________________________________
De : Amine Elmekki <am...@corporate-groupe.com>>
Envoyé : lundi 20 septembre 2021 18:31
À : nawfal osman <na...@corporate-groupe.com>>
Cc : Zakaria Rahoui <za...@corporate-groupe.com>>
Objet : Soucis Drill
Nawfal,
Besoin des cas réel des problèmes de drill :
1. Pb de lenteur lorsqu’on lance deux projets en //.
Eléments à envoyer : Request, date début, date fin (en la lançant seule et avec deux req en //)
1. Problème de jointure de drill. Eléments à envoyer :
* Requête
* Fausse résultat
* Correct résultat qu’on doit obtenir
1. D’autres problèmes…
Re: Soucis Drill
Posted by James Turton <dz...@apache.org>.
I would try to resolve 2 first, since a major contributor to 1 could be
spilling to disk. What memory settings are running Drill with?
On 2021/09/23 09:44, nawfal osman wrote:
> Hi Support
>
> Below the drill issues that we have
> 1. Slowness issues when running two queries the same time ( please
> find attached an Example of querie)
>
> Start of execution querie 20210815: 09: 26: 23 to 09: 33 ==> about
> 6min30s
>
> Start of execution of the date 15 and 16 has been launched at the
> same time: 09:38 to 09:48 ==> 10 minutes of exécution
>
> 2 - Sometimes we get the error (Unable to allocate buffer of size
> 134217728 due to memory limit (191739611). Current allocation: 78880768)
>
> Can you please help up to troubleshoot theses issues.
>
> Kind Regards
> ------------------------------------------------------------------------
> *De :* Amine Elmekki <am...@corporate-groupe.com>
> *Envoyé :* mercredi 22 septembre 2021 09:08
> *À :* nawfal osman <na...@corporate-groupe.com>
> *Cc :* Zakaria Rahoui <za...@corporate-groupe.com>
> *Objet :* RE: Soucis Drill
>
> Merci
>
> Tu envois ces éléments à user@drill.apache.org
> <ma...@drill.apache.org>
>
> *From:*nawfal osman <na...@corporate-groupe.com>
> *Sent:* Tuesday, 21 September 2021 10:15
> *To:* Amine Elmekki <am...@corporate-groupe.com>
> *Cc:* Zakaria Rahoui <za...@corporate-groupe.com>
> *Subject:* RE: Soucis Drill
>
> Bonjour ;
>
> Ci-dessous mon retour :
>
> 1. *Pb de lenteur lorsqu’on lance deux projets en //.*
>
> *Eléments à envoyer : Request, date début, date fin (en la lançant
> seule et avec deux req en //)*
>
> Début d'exécution de la journée du 20210815 : 09 :26 :23 au 09 :33
> ==> environ 6min30s
>
> Début d'exécution quand on a lancé la journée du 15 et 16 en même
> temps : 09:38 à 09:48 ==> 10 minutes d'exécution
>
> En jointure la requête utilisée
>
> 2. *Problème de jointure de drill. Eléments à envoyer :*
>
> * *Requête*
> * *Fausse résultat*
> * *Correct résultat qu’on doit obtenir*
>
> Pour ce cas de figure, je n'arrive toujours pas à trouver un cas ; je
> suis toujours en train de chercher ( je vous ferai un retour dans la
> journée )
>
> ****3*.**D’autres problèmes…*
>
> * Parfois on a l'erreur ( Unable to allocate buffer of size
> 134217728 due to memory limit (191739611). Current allocation:
> 78880768 ) ; mais le support MapR nous a envoyer un lien pour
> troubleshooting : Troubleshooting Guide: DRILL OutOfMemory error
> and possible workarounds
> https://support.datafabric.hpe.com/s/article/Troubleshooting-DRILL-OutOfMemory-error-and-possible-workarounds?language=en_US
> <https://support.datafabric.hpe.com/s/article/Troubleshooting-DRILL-OutOfMemory-error-and-possible-workarounds?language=en_US>
>
> Cordialement
>
> ------------------------------------------------------------------------
>
> *De :*Amine Elmekki <amine.elmekki@corporate-groupe.com
> <ma...@corporate-groupe.com>>
> *Envoyé :* lundi 20 septembre 2021 18:31
> *À :* nawfal osman <nawfal.osman@corporate-groupe.com
> <ma...@corporate-groupe.com>>
> *Cc :* Zakaria Rahoui <zakaria.rahoui@corporate-groupe.com
> <ma...@corporate-groupe.com>>
> *Objet :* Soucis Drill
>
> Nawfal,
>
> Besoin des cas réel des problèmes de drill :
>
> 1. Pb de lenteur lorsqu’on lance deux projets en //.
>
> Eléments à envoyer : Request, date début, date fin (en la lançant
> seule et avec deux req en //)
>
> 2. Problème de jointure de drill. Eléments à envoyer :
>
> * Requête
> * Fausse résultat
> * Correct résultat qu’on doit obtenir
>
> 3. D’autres problèmes…
>