[jira] [Commented] (PHOENIX-938) Use higher priority queue for index updates to prevent deadlock

["Jesse Yates (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056785#comment-14056785 ] 

Jesse Yates commented on PHOENIX-938:
-------------------------------------

Started looking at this today.... at first, 0.98.3 is pretty simple. Then it starts to get complicated as 0.98.4 completely changed how rpc scheduling is implemented (HBASE-11355). I don't know if we have the bandwidth to continually monitor all the possible changes to the scheduler code to support this. Further, as we look to real transactions, this implementation becomes somewhat moot; maybe we just leave the code as-is?

The nitty-gritty of the details is that 0.98.4 introduced the idea of an RpcExecutor (which is a great improvement over the current munging), but that isn't in 0.98.3, so we would either need to port that class to phoenix (losing any updates from the HBase community), but that's kinda already what I was doing with this patch, so maybe that's alright for now. 

Now, we could have a whole reflection framework to support the different HBase versions we are running (which becomes a testing pain, but doable) and then pick the most optimal one (0.98.4+ just uses RpcExecutor as-is, 0.98.3 uses the copied code, <=0.98.2 ignores). Or we can copy the changed implementations back and just use the same thing everywhere, but we loose out on changes... There really isn't a clean solution here :-/

Really, this stems from the RpcScheduler code being a private interface in HBase but wanting to leverage it outside HBase.

thoughts [~jamestaylor]?

> Use higher priority queue for index updates to prevent deadlock
> ---------------------------------------------------------------
>
>                 Key: PHOENIX-938
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-938
>             Project: Phoenix
>          Issue Type: Bug
>    Affects Versions: 4.0.0, 4.1
>            Reporter: James Taylor
>            Assignee: Jesse Yates
>             Fix For: 5.0.0, 4.1
>
>         Attachments: phoenix-938-4.0-v0.patch, phoenix-938-master-v0.patch, phoenix-938-master-v1.patch
>
>
> With our current global secondary indexing solution, a batched Put of table data causes a RS to do a batch Put to other RSs. This has the potential to lead to a deadlock if all RS are overloaded and unable to process the pending batched Put. To prevent this, we should use a higher priority queue to submit these Puts so that they're always processed before other Puts. This will prevent the potential for a deadlock under high load. Note that this will likely require some HBase 0.98 code changes and would not be feasible to implement for HBase 0.94.



--
This message was sent by Atlassian JIRA
(v6.2#6252)