[jira] [Work started] (HBASE-22899) logging improvements for snapshot operations w/large manifests

["Rabi Kumar K C (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-22899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HBASE-22899 started by Rabi Kumar K C.
----------------------------------------------
> logging improvements for snapshot operations w/large manifests
> --------------------------------------------------------------
>
>                 Key: HBASE-22899
>                 URL: https://issues.apache.org/jira/browse/HBASE-22899
>             Project: HBase
>          Issue Type: Improvement
>          Components: snapshots
>            Reporter: Sean Busbey
>            Assignee: Rabi Kumar K C
>            Priority: Minor
>              Labels: beginner
>
> working through some issues with snapshotting in the presence of a substantial (millions-of) hfiles.
> log messages about the state of creating or restoring span multiple hundreds of MB of logs, so more context in individual lines would make it easier to orient myself.
> RestoreSnapshotHelper has a few different stanzas that essentially all say "I'm creating an HFileLink for this thing". e.g.
> {code}
>     for (SnapshotRegionManifest.FamilyFiles familyFiles: manifest.getFamilyFilesList()) {
>       Path familyDir = new Path(regionDir, familyFiles.getFamilyName().toStringUtf8());
>       for (SnapshotRegionManifest.StoreFile storeFile: familyFiles.getStoreFilesList()) {
>         LOG.info("Adding HFileLink " + storeFile.getName() + " to table=" + tableName);
>         restoreStoreFile(familyDir, snapshotRegionInfo, storeFile, createBackRefs);
>       }
>     }
> {code}
> If would be helpful if
> 1) these stanzas offered something in the logged text to tell them apart (I think the coincidentally differ in log level, but that's esoteric and easy to lose over time)
> 2) it would be nice if we included the snapshot name in the log message so I can more easily pull out all log lines related to the snapshot
> 3) these stanzas indicated "progress" of sorts by giving how many storefiles are present and where we are in that list.
> sort of like this bit we log when creating the snapshot manifest:
> {code}
>       // 2.2. iterate through all the store's files and create "references".
>       for (int i = 0, sz = storeFiles.size(); i < sz; i++) {
>         HStoreFile storeFile = storeFiles.get(i);
>         monitor.rethrowException();
>         // create "reference" to this store file.
>         LOG.debug("Adding reference for file (" + (i+1) + "/" + sz + "): " + storeFile.getPath());
>         visitor.storeFile(regionData, familyData, storeFile.getFileInfo());
>       }
> {code}
> And speaking of the above bit in SnapshotManifest, it would be nice if it included the name of the snapshot in the log message so it's easier to pull out all log lines related to the snapshot.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)