[jira] [Commented] (PHOENIX-5987) PointLookup may cost too much memory

["Chen Feng (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248724#comment-17248724 ] 

Chen Feng commented on PHOENIX-5987:
------------------------------------

Optimize the sql is acceptable, but as least we should throw a specific exception (or a WARN) to tell the user.

In our progress, we spent sever days to figure out the cause of OOM.

> PointLookup may cost too much memory
> ------------------------------------
>
>                 Key: PHOENIX-5987
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5987
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Chen Feng
>            Priority: Major
>
> When all rowkeys are covered in where conditions, Phoenix uses point look up to switch "a huge range scan" to "multi single key scans".
>  However, when #scans are too huge, Phoenix quick exhausts the memory.
> We meet such condition in hour product environment as follow:
>  We have a table with five primary keys like k1, k2, ..., k5, all key types are UNSIGNED_LONG.
>  The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and k4 in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
>  We have l=600, m=800 and n=1000, #scans of look up is 1*1*600*800*1000=480,000,000.
>  Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is 480,000,000 * 50bytes = 25GB.
>  25GB exceeds the JMX configuration and causes OOM exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)