[jira] [Commented] (PHOENIX-5250) The accumulated wal files can't be cleaned

["Istvan Toth (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284146#comment-17284146 ] 

Istvan Toth commented on PHOENIX-5250:
--------------------------------------

Looking at the changes, but without running any tests, I get the impression that this cannot be fixed from the Phoenix side.

Fixing from the HBase side seems easy, if not free (again, not tested) by changing the FSWALEntry constructor:

{code:java}
    if (inMemstore) {
      // construct familyNames here to reduce the work of log sinker.
      Set<byte[]> families = edit.getFamilies();
      // Phoenix uses METAFAMILY entries to persist index operations to the WAL,
      // those must not be added to the memstore
      families.remove(WALEdit.METAFAMILY);
      this.familyNames = families != null ? families : collectFamilies(edit.getCells());
    } else {
      this.familyNames = Collections.emptySet();
    }
{code}


> The accumulated wal files can't be cleaned
> ------------------------------------------
>
>                 Key: PHOENIX-5250
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-5250
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Jaanai Zhang
>            Priority: Blocker
>             Fix For: 5.1.0
>
>         Attachments: image-2019-04-19-21-31-27-888.png
>
>
> Because of the modification of HBASE-20781,  the faked WALEdits won't be filtered, all WALEdits will be saved into Memstore with a status that inMemstore is true(uses WAL->append method).
> !image-2019-04-19-21-31-27-888.png|width=755,height=310!
> The family array of IndexedKeyValue is WALEdit.METAFAMILY that is used to describe a fake WALEdit, and it will put into Memstore with WALedits of data table during sync global index.
> WAL files can't be cleaned, except for restarting RS, Many WAL files will decrease the percent of disk free.  
> !https://gw.alicdn.com/tfscom/TB1n3cDQVzqK1RjSZFCXXbbxVXa.png|width=422,height=159!
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)