[jira] [Created] (DRILL-6813) Inner-Join between two tar.gz files throwing error- One or more nodes ran out of memory while executing the query

["Ashish Pancholi (JIRA)" <ji...@apache.org>]

Ashish Pancholi created DRILL-6813:
--------------------------------------

             Summary: Inner-Join between two tar.gz files throwing error- One or more nodes ran out of memory while executing the query
                 Key: DRILL-6813
                 URL: https://issues.apache.org/jira/browse/DRILL-6813
             Project: Apache Drill
          Issue Type: Bug
    Affects Versions: 1.14.0
         Environment: [^TBL_MOREDATA1-20180924181406.tar.gz]
            Reporter: Ashish Pancholi


I am using `Apache Drill` version `1.14` on `windows` system having 8 GB RAM and running Drill using the command:
{code:java}
sqlline.bat -u "jdbc:drill:zk=local"{code}
I am trying to `execute` a `join query` on two `compressed` and `archived` `CSV` files.

Query:
{code:java}
SELECT * FROM dfs.`C:\Users\admin\Desktop\DRILL_FILES\csvFileParquet\TBL_MOREDATA-20180924181406.tar.gz` AS Table0 INNER JOIN dfs.`C:\Users\admin\Desktop\DRILL_FILES\csvFileParquet\TBL_MOREDATA1-20180924181406.tar.gz` AS Table1 ON Table0.columns[0]=Table1.columns[0]{code}
But an out of memory, error occurred:
{code:java}
org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query. Unable to allocate buffer of size 131072 (rounded from 86104) due to memory limit (630194176). Current allocation: 630108434 Fragment 0:0 [Error Id: 585c0644-5fd5-446e-b9b3-d48e0771eb2a on DESKTOP-SM3E3KM:31010]{code}
To resolve the issue, I tried to update `config\drill-env.sh` file but the issue remains the same and it looks like updating the script file does not reflect the changes because I am trying to increase the DIRECT MEMORY beyond the system memory (RAM), every time drill starts up peacefully. Not even complaining that you have exceeded the memory, therefore, it looks like the changes are not reflecting.
{code:java}
export DRILLBIT_MAX_PROC_MEM=12G
export DRILL_HEAP=2G
export DRILL_MAX_DIRECT_MEMORY=10G{code}
whereas my system's main memory is only 8 GB.

Please help me to resolve the out of memory error. I had even run the below queries, in order to follow the troubleshooting instructions but the issue remains the same.
{code:java}
alter session set `planner.enable_hashagg` = false; 
alter session set `planner.enable_hashjoin` = false;
alter session set planner.width.max_per_node=3; 
alter system set planner.width.max_per_query = 100;{code}
 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)