[jira] [Commented] (IMPALA-662) Frontend runs out of memory when generating many scan ranges

["Tim Armstrong (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210532#comment-17210532 ] 

Tim Armstrong commented on IMPALA-662:
--------------------------------------

I think we should add a boolean flag --unlock_impala_test_flags that allows setting weird values for query options. We can then set that on the test cluster that's launched with start-impala-cluster.py.

Then if that's not set, have a lower bound on MAX_SCAN_RANGE_LENGTH of, say, 2MB.

> Frontend runs out of memory when generating many scan ranges
> ------------------------------------------------------------
>
>                 Key: IMPALA-662
>                 URL: https://issues.apache.org/jira/browse/IMPALA-662
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Frontend
>    Affects Versions: Impala 1.2
>            Reporter: Skye Wanderman-Milne
>            Priority: Major
>              Labels: crash, downgraded, ramp-up
>
> When I set the scan range length to 2 bytes and queried a 200MB table, the frontend churned for several minutes (during which time I could not cancel the query) before the JVM ran out of memory and took down the impalad process. It would be nice if the frontend could somehow bail in this situation, or at least be cancellable. This is an extreme case and I'm not sure at what point the FE falls over, but this could theoretically be an issue with a sane scan range length and many files (or just a huge amount of data).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org