You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "Lars Hofhansl (JIRA)" <ji...@apache.org> on 2014/02/11 03:39:21 UTC
[jira] [Commented] (PHOENIX-34) Insufficient memory exception on
join when RHS rows count > 250K
[ https://issues.apache.org/jira/browse/PHOENIX-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13897473#comment-13897473 ]
Lars Hofhansl commented on PHOENIX-34:
--------------------------------------
We should not give Phoenix 50% of the heap on a RegionServer.
By default a RegionServer is configured with 40% of the heap to the memstore and another 40% to the blockcache, and 20% for "everything else".
5-10% to Phoenix seems like a good default. When more is needed other parts of the RegionServer need to reconfigured.
Also with the hash-join types Phoenix supports I had assumed we'd limit the smaller hashed to 10k rows or so.
It's nice to support a wider range of scenarios, but we need to accept the realities of a RegionServer, where heap is already at a premium.
(I think we need to generally think through Phoenix's resource consumption anyway and if/how they scale with the number of available resources. For example the number of a threads to use for a query should not be fixed, etc.)
> Insufficient memory exception on join when RHS rows count > 250K
> -----------------------------------------------------------------
>
> Key: PHOENIX-34
> URL: https://issues.apache.org/jira/browse/PHOENIX-34
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 3.0.0
> Environment: HBase 0.94.14, r1543222, Hadoop 1.0.4, r1393290, 2 RS + 1 Master, Heap 4GB per RS
> Reporter: Mujtaba Chohan
> Fix For: 3.0.0
>
>
> Join fails when rows count of RHS table is >250K. Detail on table schema is and performance numbers with different LHS/RHS row count is on http://phoenix-bin.github.io/client/performance/phoenix-20140210023154.htm.
> James comment:
> So that's with a 4GB heap allowing Phoenix to use 50% of it. With a pretty narrow table: 3 KV columns of 30bytes. Topping out at 250K is a bit low. I wonder if our memory estimation matches reality.
> What do you think Maryann?
> How about filing a JIRA, Mujtaba. This is a good conversation to have on the dev list. Can we move it there, please?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)