[jira] [Created] (DRILL-4118) Drillbit Foreman shutdowns while executing complex query on large amount of data

["Shankar (JIRA)" <ji...@apache.org>]

Shankar created DRILL-4118:
------------------------------

             Summary: Drillbit Foreman shutdowns while executing complex query on large amount of data
                 Key: DRILL-4118
                 URL: https://issues.apache.org/jira/browse/DRILL-4118
             Project: Apache Drill
          Issue Type: Test
    Affects Versions: 1.2.0
            Reporter: Shankar


h4.{color:DarkCyan}*System config for POC:*{color}
* Servers => AWS instances
* Total Servers => 3
* instance Type => c4.xlarge
* vCPU => 4
* Memory => 7.5 GB
* Storage Type => EBS
* OS => CentOS-6.6 ( x64 architecture)

h4.{color:DarkCyan}*Data :*{color}
* DataSize = 15 GB GZ compressed ( equivalent of 150GB of uncompressed data)
* Type of Data => json format (1 json per line)
* Persistent storage => HDFS
* Data Frequency => 1 Day data only. ( file names are divided by each hour)

h4.{color:DarkCyan}*How we setup Apache drill :*{color}
# Version = Apache Drill 1.2.0
# Setup using default configurations on all 3 nodes.
# used Drill shell to query.
# Drill Web-Console to analyze the queries.


h4.{color:green}*Query-1 (total counts):*{color}
We had run simple query for *1 hour data*.Below is the query :

 - select count(`timestamp`) from dfs.`/tmp/hadoop/20151120-10.json.gz`

- Query has taken something around 120 seconds and it ran successfully.
- cpu load => 1.5 (on an avg per node)
- memory used => 3gb (on an avg per node)



h4.{color:green}*Query-2 (distinct counts)  :*{color}
We had run simple query for *1 hour data*.Below is the query :

- select count( distinct `timestamp`) from dfs.`/tmp/hadoop/20151120-10.json.gz`

-  Query has taken something around 200 seconds and it ran successfully.
- cpu load => 5.5 (on an avg per node)
- memory used => 3.9gb (on an avg per node)



h4.{color:green}*Query-3 (create table using filter)  :*{color}

We had run simple query for *1 day data*.Below is the query :

- create table tmp as select col1, col2 from dfs.`/tmp/hadoop`
where col like '%filter-text%'

- All columns are string in natures.
- Query has taken something around 340 seconds and it ran successfully.
- cpu load => 6.2 (on an avg per node)
- memory used => 4.2gb (on an avg per node)



h4.{color:red}*Query-4 (complex query with filters) :*{color}
We had run query for *1 day data*.Below is the query :

select
count( distinct case when col like '%filter-text%' then sessions end ) as new_col_01,
count( distinct case when col like '%filter-text%' then sessions end ) as new_col_02,
------------------
------------------
------------------
count( distinct case when col like '%filter-text%' then sessions end ) as new_col_15
from dfs.`/tmp/hadoop`

-- All columns are string in natures.
-- filters conditions are different for each count clauses.
-- {color:red}from drill shell => *seemed query were still running*{color}
-- {color:red}from logs => *drillbit Foreman shutdown*{color}
- cpu load => *85.x* (on an avg per node)
- memory used => *6.6gb* (on an avg per node)

{color:red}=> Error from Log file of drillbit Foreman node{color}
----------------------------------------------------

2015-11-20 18:53:59,185 [29b058ba-2c2c-2c7b-d380-00fb51af47c2:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 41ms total, 41.774180ms avg, 41ms max.
2015-11-20 18:53:59,185 [29b058ba-2c2c-2c7b-d380-00fb51af47c2:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 7.217000 μs, Latest start: 7.217000 μs, Average start: 7.217000 μs .
2015-11-20 19:06:07,320 [Drillbit-ShutdownHook#0] INFO  o.apache.drill.exec.server.Drillbit - Received shutdown request.

----------------------------------------------------




h4.*Questions are:*

# Could you please tell me solution for above error ?
# Does drill-bit is needed high end servers to process large amount of data ?
# Does drill bit works well if we scale our servers horizontally with low system configurations (say 4 virtual CPU's, 8gb memory) and process large amount of data?
# Does drill bit works well if we scale our servers horizontally with low system configurations (say 8 virtual CPU's, 16gb memory) and process large amount of data?
# And finally please provide me the well tuned configuration. 







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)