[jira] [Commented] (HBASE-16524) Procedure v2 - Compute WALs cleanup on wal modification and not on every sync

["Appy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-16524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798436#comment-15798436 ] 

Appy commented on HBASE-16524:
------------------------------

Reviewed it just now. Looks good to me.
Should probably change the functions {{insert(BitSetNode node, long procId)}} and others to {{insert(long procId, BitSetNode nodeHint)}} because the first argument isn't mandatory, instead it's a hint to avoid lookup into map repeatedly.

One thing worth mentioning, wal cleanup logic gets triggered on every wal sync, (WalProcedureStore#syncLoop), so the effort here is to make it quick.
One case here is, if a procedure gets stuck and was last synced a while back, we keep holding intermediate wals. Once the stuck procedure finishes, it'll take some time to cleanup wals since each wal will be sequentially processed as 'holding' wall. This will delay the next sync a bit.
Just a heads up in case we see issues related to wal.

> Procedure v2 - Compute WALs cleanup on wal modification and not on every sync
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-16524
>                 URL: https://issues.apache.org/jira/browse/HBASE-16524
>             Project: HBase
>          Issue Type: Sub-task
>          Components: proc-v2
>    Affects Versions: 2.0.0
>            Reporter: Appy
>            Assignee: Matteo Bertozzi
>            Priority: Minor
>             Fix For: 2.0.0
>
>         Attachments: HBASE-16524-v2.patch, HBASE-16524-v3.patch, HBASE-16524-v4.patch, HBASE-16524-v5.patch, HBASE-16524-v6.patch, HBASE-16524.master.001.patch, HBASE-16524.master.002.patch, flame1.svg
>
>
> Fix performance regression introduced by HBASE-16094.
> Instead of scanning all the wals every time, we can rely on the insert/update/delete events we have.
> and since we want to delete the wals in order we can keep track of what is "holding" that wal, and take a hit on scanning all the trackers only when we remove the first log in the queue.
> e.g.
> WAL-1 [1, 2] 
> WAL-2 [1] -> "[2] is holding WAL-1"
> WAL-3 [2] -> "WAL 1 can be removed, recompute what is holding WAL-2"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)