You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Giovanni Conte <gi...@gmail.com> on 2019/03/12 13:03:10 UTC
RESOURCE ERROR: External Sort encountered an error while spilling to disk
Hello,
I am doing a LAG and a SUM query over a PCAP dataset of 300Mb.
I get this error:
RESOURCE ERROR: External Sort encountered an error while spilling to disk
Then I changetd
planner.memory.max_query_memory_per_node ---> 8589934592
and with this I am able to perform the LAG but not the SUM.
I have no problem of memory since I am working on a server with 72 cores
and 256 GB of RAM.
Which other parameter should I change to avoid the RESOURCE ERROR?
Can I go over 8589934592?
Thank you very much,
Giovanni
Re: RESOURCE ERROR: External Sort encountered an error while spilling
to disk
Posted by Khurram Faraaz <kf...@mapr.com>.
Can you also share the stack trace from drillbit.log for the below error
RESOURCE ERROR: External Sort encountered an error while spilling to disk
And what is the version of Drill that you are running ?
Please share the table definition and the number of rows in that table,
which is being queried.
Thanks,
Khurram
On Tue, Mar 12, 2019 at 12:17 PM Boaz Ben-Zvi <bo...@apache.org> wrote:
> Hi Giovanni,
>
> The error given by the External-Sort indicates a problem while
> spilling the excess memory into disk.
>
> When you enlarged the memory (from the default 2GB) to 8GB the LAG query
> may succeeded without spilling, hence circumvented the issue.
>
> Yes, you can keep enlarging the memory and run w/o spilling, but better
> check and fix the root issue.
>
> How is your spilling configured - check the filesystem -
> "drill.exec.spill.fs" and the directories - "drill.exec.spill.directories"
>
> The default is the local filesystem, and into /tmp . It is possible
> that very little disk space is available for /tmp .
>
> Thanks,
>
> -- Boaz
>
> On 3/12/19 6:03 AM, Giovanni Conte wrote:
> > Hello,
> > I am doing a LAG and a SUM query over a PCAP dataset of 300Mb.
> > I get this error:
> > RESOURCE ERROR: External Sort encountered an error while spilling to disk
> >
> > Then I changetd
> > planner.memory.max_query_memory_per_node ---> 8589934592
> > and with this I am able to perform the LAG but not the SUM.
> >
> > I have no problem of memory since I am working on a server with 72 cores
> > and 256 GB of RAM.
> > Which other parameter should I change to avoid the RESOURCE ERROR?
> > Can I go over 8589934592?
> > Thank you very much,
> >
> > Giovanni
> >
>
Re: RESOURCE ERROR: External Sort encountered an error while spilling
to disk
Posted by Boaz Ben-Zvi <bo...@apache.org>.
Hi Giovanni,
The error given by the External-Sort indicates a problem while
spilling the excess memory into disk.
When you enlarged the memory (from the default 2GB) to 8GB the LAG query
may succeeded without spilling, hence circumvented the issue.
Yes, you can keep enlarging the memory and run w/o spilling, but better
check and fix the root issue.
How is your spilling configured - check the filesystem -
"drill.exec.spill.fs" and the directories - "drill.exec.spill.directories"
The default is the local filesystem, and into /tmp . It is possible
that very little disk space is available for /tmp .
Thanks,
-- Boaz
On 3/12/19 6:03 AM, Giovanni Conte wrote:
> Hello,
> I am doing a LAG and a SUM query over a PCAP dataset of 300Mb.
> I get this error:
> RESOURCE ERROR: External Sort encountered an error while spilling to disk
>
> Then I changetd
> planner.memory.max_query_memory_per_node ---> 8589934592
> and with this I am able to perform the LAG but not the SUM.
>
> I have no problem of memory since I am working on a server with 72 cores
> and 256 GB of RAM.
> Which other parameter should I change to avoid the RESOURCE ERROR?
> Can I go over 8589934592?
> Thank you very much,
>
> Giovanni
>