You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Jacques Nadeau (JIRA)" <ji...@apache.org> on 2016/01/20 16:32:39 UTC
[jira] [Resolved] (DRILL-4278) Memory leak when using LIMIT
[ https://issues.apache.org/jira/browse/DRILL-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jacques Nadeau resolved DRILL-4278.
-----------------------------------
Resolution: Fixed
Assignee: Jacques Nadeau
Fix Version/s: 1.5.0
Resolved in 43414e1e2731c85a52febaaedebfa2f392dd6d2f
> Memory leak when using LIMIT
> ----------------------------
>
> Key: DRILL-4278
> URL: https://issues.apache.org/jira/browse/DRILL-4278
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - RPC
> Affects Versions: 1.4.0, 1.5.0
> Environment: OS X
> 0: jdbc:drill:zk=local> select * from sys.version;
> +----------+-------------------------------------------+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+
> | version | commit_id | commit_message | commit_time | build_email | build_time |
> +----------+-------------------------------------------+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+
> | 1.4.0 | 32b871b24c7b69f59a1d2e70f444eed6e599e825 | [maven-release-plugin] prepare release drill-1.4.0 | 08.12.2015 @ 00:24:59 PST | venki.korukanti@gmail.com | 08.12.2015 @ 01:14:39 PST |
> +----------+-------------------------------------------+-----------------------------------------------------+----------------------------+----------------------------+----------------------------+
> 0: jdbc:drill:zk=local> select * from sys.options where status <> 'DEFAULT';
> +-----------------------------+-------+---------+----------+----------+-------------+-----------+------------+
> | name | kind | type | status | num_val | string_val | bool_val | float_val |
> +-----------------------------+-------+---------+----------+----------+-------------+-----------+------------+
> | planner.slice_target | LONG | SYSTEM | CHANGED | 10 | null | null | null |
> | planner.width.max_per_node | LONG | SYSTEM | CHANGED | 5 | null | null | null |
> +-----------------------------+-------+---------+----------+----------+-------------+-----------+------------+
> 2 rows selected (0.16 seconds)
> Reporter: jean-claude
> Assignee: Jacques Nadeau
> Fix For: 1.5.0
>
>
> copy the parquet files in the samples directory so that you have a 12 or so
> $ ls -lha /apache-drill-1.4.0/sample-data/nationsMF/
> nationsMF1.parquet
> nationsMF2.parquet
> nationsMF3.parquet
> create a file with a few thousand lines like these
> select * from dfs.`/Users/jccote/apache-drill-1.4.0/sample-data/nationsMF` limit 500;
> start drill
> $ /apache-drill-1.4.0/bin/drill-embeded
> reduce the slice target size to force drill to use multiple fragment/threads
> jdbc:drill:zk=local> system set planner.slice_target=10;
> now run the list of queries from the file your created above
> jdbc:drill:zk=local> !run /Users/jccote/test-memory-leak-using-limit.sql
> the java heap space keeps going up until the old space is at 100% and eventually you get an OutOfMemoryException in drill
> $ jstat -gccause 86850 5s
> S0 S1 E O M CCS YGC YGCT FGC FGCT GCT LGCC GCC
> 0.00 0.00 100.00 100.00 98.56 96.71 2279 26.682 240 458.139 484.821 GCLocker Initiated GC Ergonomics
> 0.00 0.00 100.00 99.99 98.56 96.71 2279 26.682 242 461.347 488.028 Allocation Failure Ergonomics
> 0.00 0.00 100.00 99.99 98.56 96.71 2279 26.682 245 466.630 493.311 Allocation Failure Ergonomics
> 0.00 0.00 100.00 99.99 98.56 96.71 2279 26.682 247 470.020 496.702 Allocation Failure Ergonomics
> If you do the same test but do not use the LIMIT then the memory usage does not go up.
> If you add a where clause so that no results are returned, then the memory usage does not go up.
> Something with the RPC layer?
> Also it seems sensitive to the number of fragments/threads. If you limit it to one fragment/thread the memory usage goes up much slower.
> I have used parquet files and CSV files. In either case the behaviour is the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)