[jira] [Commented] (HBASE-21098) Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3

["Tyler Mi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16603792#comment-16603792 ] 

Tyler Mi commented on HBASE-21098:
----------------------------------

For TestSnapshotDFSTemporaryDirectory, through my debugging, it showed that the file system of both the working and root directory were DFS rather than the local filesystem, so I believe the test does indeed test DFS correctly. In addition, the precondition check only makes sure that the temporary directory is not in the HBase root directory (except for the default location) such that it would not wipe out HBase related data. But it does not protect against other directories outside of the HBase root directory, which is why caution must be exercised in choosing the directory.

Regarding completeSnapshot(), I agree it was really confusing logic and I've since commented on it! And you bring up a really good point regarding the use of .equals() for file systems in TakeSnapshotHandler::completeSnapshot(). Do you think it's not worth making the logic more complicated for the occasions that the rename would be faster?

Lastly, I do not have automated testing for testing with S3 and HDFS, all testing with S3 was done manually. I did not set-up automated testing due the need for proper S3 test buckets.

> Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3
> ---------------------------------------------------------------------------------
>
>                 Key: HBASE-21098
>                 URL: https://issues.apache.org/jira/browse/HBASE-21098
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0, 1.4.8, 2.1.1
>            Reporter: Tyler Mi
>            Priority: Major
>         Attachments: HBASE-21098.master.001.patch, HBASE-21098.master.002.patch, HBASE-21098.master.003.patch, HBASE-21098.master.004.patch, HBASE-21098.master.005.patch, HBASE-21098.master.006.patch, HBASE-21098.master.007.patch, HBASE-21098.master.008.patch, HBASE-21098.master.009.patch, HBASE-21098.master.010.patch
>
>
> When using Apache HBase, the snapshot feature can be used to make a point in time recovery. To do this, HBase creates a manifest of all the files in all of the Regions so that those files can be referenced again when a user restores a snapshot. With HBase's S3 storage mode, developers can store their data off-cluster on Amazon S3. However, utilizing S3 as a file system is inefficient in some operations, namely renames. Most Hadoop ecosystem applications use an atomic rename as a method of committing data. However, with S3, a rename is a separate copy and then a delete of every file which is no longer atomic and, in fact, quite costly. In addition, puts and deletes on S3 have latency issues that traditional filesystems do not encounter when manipulating the region snapshots to consolidate into a single manifest. When HBase on S3 users have a significant amount of regions, puts, deletes, and renames (the final commit stage of the snapshot) become the bottleneck causing snapshots to take many minutes or even hours to complete.
> The purpose of this patch is to increase the overall performance of snapshots while utilizing HBase on S3 through the use of a temporary directory for the snapshots that exists on a traditional filesystem like HDFS to circumvent the bottlenecks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)