[jira] [Updated] (PHOENIX-5987) PointLookup may cost too much memory

["Chen Feng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chen Feng updated PHOENIX-5987:
-------------------------------
    Description: 
When all rowkeys are covered in where conditions, Phoenix uses point look up to switch "a huge range scan" to "multi single key scans".
 However, when #scans are too huge, Phoenix quick exhausts the memory.

We meet such condition in hour product environment as follow:
 We have a table with five primary keys like k1, k2, ..., k5, all key types are UNSIGNED_LONG.
 The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and k4 in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
 We have l=600, m=800 and n=1000, #scans of look up is 1*1*600*800*1000=480,000,000.
 Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is 480,000,000 * 50bytes = 25GB.
 25GB exceeds the JMX configuration and causes OOM exception.

  was:
When all rowkeys are covered in where conditions, Phoenix use point look up to switch "a huge range scan" to "multi single key scans".
However, the number single key scans are too huge, it quick exhausts the memory.

We meet such condition in hour product environment as follow:
We have a table with five primary keys like k1, k2, ..., k5, all key types are UNSIGNED_LONG.
The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and k4 in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
We have l=600, m=800 and n=1000, so the possible number of look up scans is 1*1*600*800*1000=480,000,000.
Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is 480,000,000 * 50bytes = 25GB.
25GB exceeds the JMX configuration and causes OOM exception.


> PointLookup may cost too much memory
> ------------------------------------
>
>                 Key: PHOENIX-5987
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5987
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Chen Feng
>            Priority: Major
>
> When all rowkeys are covered in where conditions, Phoenix uses point look up to switch "a huge range scan" to "multi single key scans".
>  However, when #scans are too huge, Phoenix quick exhausts the memory.
> We meet such condition in hour product environment as follow:
>  We have a table with five primary keys like k1, k2, ..., k5, all key types are UNSIGNED_LONG.
>  The query sql like " ... where k1 = 1 and k2 = 1 and k3 in (1,2,3,...,l) and k4 in (1,2,3,...,m) and k5 in (1,2,3,...,n)"
>  We have l=600, m=800 and n=1000, #scans of look up is 1*1*600*800*1000=480,000,000.
>  Each scan rowkey costs 5*8=40 bytes. Therefore the total memory cost is 480,000,000 * 50bytes = 25GB.
>  25GB exceeds the JMX configuration and causes OOM exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)