[jira] [Updated] (HBASE-16094) Procedure v2 - Improve cleaning up of proc wals

["Appy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Appy updated HBASE-16094:
-------------------------
    Description: 
Avoid accumulating too many wals.
We remove logs in 3 cases:
 - No procedures are running. We can remove all the logs.
 - All procedures are updated/written in the last log. We can remove all the logs, aside the active one.	
 - Remove log if does not contains “active” procIds

https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865
The last point, Remove log if does not contains “active” procIds, needs to be improved. Basically, even if a log contains state of an active proc, we can delete it if a later log contains a newer state.


  was:
Avoid accumulating too many wals.
We remove logs in 3 cases:
 - No procedures are running. We can remove all the logs.
 - All procedures are updated/written in the last log. We can remove all the logs, aside the active one.	
 - Remove log if does not contains “active” procIds

https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865
Remove log if does not contains “active” procIds requires to be improved, since those procedures can be in earlier wals. An example that can be unit-tested is:
Insert 10 proc, Roll The Wal, Update the 10 proc
At the end of the 10 updates the removeInactiveLogs() should be able to remove the first WAL. We can probably do a diff on the updated tracker bitmap between wals.



> Procedure v2 - Improve cleaning up of proc wals
> -----------------------------------------------
>
>                 Key: HBASE-16094
>                 URL: https://issues.apache.org/jira/browse/HBASE-16094
>             Project: HBase
>          Issue Type: Sub-task
>          Components: proc-v2
>            Reporter: Appy
>             Fix For: 2.0.0
>
>
> Avoid accumulating too many wals.
> We remove logs in 3 cases:
>  - No procedures are running. We can remove all the logs.
>  - All procedures are updated/written in the last log. We can remove all the logs, aside the active one.	
>  - Remove log if does not contains “active” procIds
> https://github.com/apache/hbase/blob/master/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L865
> The last point, Remove log if does not contains “active” procIds, needs to be improved. Basically, even if a log contains state of an active proc, we can delete it if a later log contains a newer state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)