You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hbase.apache.org by zh...@apache.org on 2021/12/23 04:23:16 UTC
[hbase] branch HBASE-26233 updated (f40f268 -> f919774)
This is an automated email from the ASF dual-hosted git repository.
zhangduo pushed a change to branch HBASE-26233
in repository https://gitbox.apache.org/repos/asf/hbase.git.
discard f40f268 HBASE-26540 Implement a tool to verify the lag of region replication (#3950)
discard a6b728a HBASE-26539 The default rpc timeout 200ms is too small for replicating meta edits (#3919)
discard 44b6483 HBASE-26538 Should find a way to clear the replication queue for a legacy region_replica_replication peer (#3918)
discard d6a5d34 HBASE-26486 Attach the new region replication framework design doc to git repo (#3884)
discard 724735c HBASE-26478 Update ref guide about the new region replication framework (#3885)
discard ac9258f HBASE-26481 Consider rolling upgrading from old region replication framework (#3880)
discard 88daafe HBASE-26417 Remove NoopReplicationQueueStorage (#3882)
discard fbe6665 HBASE-26456 Limit the size for one replicating (#3873)
discard d841ffd HBASE-26416 Implement a new method for region replication instead of using replay (#3864)
discard 47c35f0 HBASE-26457 Should not always clear all the failed replicas when getting a flush all request (#3850)
discard 6febbc8 HBASE-26448 Make sure we do not flush a region too frequently (#3847)
discard c894a76 HBASE-26449 The way we add or clear failedReplicas may have race (#3846)
discard 282477c HBASE-26413 Limit the total size of buffered region replication entries (#3844)
discard b3ac276 HBASE-26412 Handle sink failure in RegionReplicationSink (#3815)
discard 5d69d1c HBASE-26407 Introduce a region replication sink for sinking WAL edits to secondary replicas directly (#3807)
add ea0fe22 HBASE-26579 Set storage policy of recovered edits when hbase.wal.storage.type is configured (#3948)
add d399799 HBASE-26580 The message of StoreTooBusy is confused (#3949)
add 314e924 HBASE-26606 Upgrade log4j2 to 2.17.0 (#3965)
add 6818ec2 HBASE-26613 The logic of the method incrementIV in Encryption class has problem (#3968)
add 6aaef897 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic
add 43b40e9 HBASE-25988 Store the store file list by a file (#3578)
add 6e05376 HBASE-26079 Use StoreFileTracker when splitting and merging (#3617)
add 090b2fe HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656)
add 0ee1689 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666)
add 2052e80 HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665)
add 5ff0f98 HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681)
add fc4f6d1 HBASE-26280 Use store file tracker when snapshoting (#3685)
add 06db852 HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721)
add e4e7cf8 HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774)
add 08d1171 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749)
add 8bec26e HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700)
add a288365 HBASE-26271 Cleanup the broken store files under data directory (#3786)
add d00b5fa HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845)
add 771e552 HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot
add f16b7b1 HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942)
add bf258cd HBASE-26598 Fix excessive connections in MajorCompactor (#3961)
add dbdef59 HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967)
add 8309f86 HBASE-26407 Introduce a region replication sink for sinking WAL edits to secondary replicas directly (#3807)
add aec952e HBASE-26412 Handle sink failure in RegionReplicationSink (#3815)
add b24c2e8 HBASE-26413 Limit the total size of buffered region replication entries (#3844)
add e75b3a6 HBASE-26449 The way we add or clear failedReplicas may have race (#3846)
add 5651572 HBASE-26448 Make sure we do not flush a region too frequently (#3847)
add 8c810b3 HBASE-26457 Should not always clear all the failed replicas when getting a flush all request (#3850)
add 21ed55f HBASE-26416 Implement a new method for region replication instead of using replay (#3864)
add 1d9c339 HBASE-26456 Limit the size for one replicating (#3873)
add e42b906 HBASE-26417 Remove NoopReplicationQueueStorage (#3882)
add 02f514b HBASE-26481 Consider rolling upgrading from old region replication framework (#3880)
add 3c2b4c8 HBASE-26478 Update ref guide about the new region replication framework (#3885)
add d38195f HBASE-26486 Attach the new region replication framework design doc to git repo (#3884)
add b54aaf5 HBASE-26538 Should find a way to clear the replication queue for a legacy region_replica_replication peer (#3918)
add bb35a87 HBASE-26539 The default rpc timeout 200ms is too small for replicating meta edits (#3919)
add f919774 HBASE-26540 Implement a tool to verify the lag of region replication (#3950)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f40f268)
\
N -- N -- N refs/heads/HBASE-26233 (f919774)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../java/org/apache/hadoop/hbase/client/Admin.java | 44 +-
.../hadoop/hbase/client/AdminOverAsyncAdmin.java | 7 +-
.../org/apache/hadoop/hbase/client/AsyncAdmin.java | 14 +-
.../hadoop/hbase/client/AsyncHBaseAdmin.java | 6 +-
.../hbase/client/ColumnFamilyDescriptor.java | 5 +
.../client/ColumnFamilyDescriptorBuilder.java | 6 +
.../hadoop/hbase/client/RawAsyncHBaseAdmin.java | 24 +-
.../apache/hadoop/hbase/io/crypto/Encryption.java | 21 +-
.../hadoop/hbase/io/crypto/TestEncryption.java | 20 +
.../src/main/protobuf/server/master/Master.proto | 1 +
.../protobuf/server/master/MasterProcedure.proto | 1 +
.../StoreFileTracker.proto} | 16 +-
.../java/org/apache/hadoop/hbase/io/HFileLink.java | 63 +-
.../org/apache/hadoop/hbase/master/HMaster.java | 31 +-
.../hadoop/hbase/master/MasterRpcServices.java | 2 +-
.../assignment/MergeTableRegionsProcedure.java | 20 +-
.../assignment/SplitTableRegionProcedure.java | 40 +-
.../hbase/master/migrate/RollingUpgradeChore.java | 130 ++++
.../master/procedure/CloneSnapshotProcedure.java | 106 +--
.../master/procedure/CreateTableProcedure.java | 43 +-
.../master/procedure/DeleteTableProcedure.java | 95 +--
.../master/procedure/ModifyTableProcedure.java | 5 +
.../master/procedure/RestoreSnapshotProcedure.java | 7 +-
.../hbase/master/snapshot/SnapshotManager.java | 27 +-
.../hadoop/hbase/mob/DefaultMobStoreCompactor.java | 28 +-
.../hadoop/hbase/mob/DefaultMobStoreFlusher.java | 4 +-
.../regionserver/AbstractMultiFileWriter.java | 6 +-
.../hbase/regionserver/BrokenStoreFileCleaner.java | 202 ++++++
.../regionserver/CreateStoreFileWriterParams.java | 134 ++++
.../regionserver/DateTieredMultiFileWriter.java | 2 +-
.../hbase/regionserver/DateTieredStoreEngine.java | 5 +-
.../hbase/regionserver/DefaultStoreEngine.java | 5 +-
.../hbase/regionserver/DefaultStoreFlusher.java | 11 +-
.../hadoop/hbase/regionserver/HMobStore.java | 3 +-
.../hbase/regionserver/HRegionFileSystem.java | 50 +-
.../hadoop/hbase/regionserver/HRegionServer.java | 86 ++-
.../apache/hadoop/hbase/regionserver/HStore.java | 735 +++++----------------
.../hadoop/hbase/regionserver/StoreContext.java | 14 +
.../hadoop/hbase/regionserver/StoreEngine.java | 482 +++++++++++++-
.../hbase/regionserver/StoreFileManager.java | 9 +
.../hadoop/hbase/regionserver/StoreFlusher.java | 9 +-
.../hadoop/hbase/regionserver/StoreUtils.java | 37 +-
.../hbase/regionserver/StripeMultiFileWriter.java | 2 +-
.../hbase/regionserver/StripeStoreEngine.java | 9 +-
.../hbase/regionserver/StripeStoreFlusher.java | 9 +-
.../compactions/AbstractMultiOutputCompactor.java | 11 +-
.../hbase/regionserver/compactions/Compactor.java | 81 ++-
.../compactions/DateTieredCompactor.java | 6 +-
.../regionserver/compactions/DefaultCompactor.java | 25 +-
.../regionserver/compactions/StripeCompactor.java | 2 +-
.../storefiletracker/DefaultStoreFileTracker.java | 68 ++
.../FileBasedStoreFileTracker.java | 162 +++++
.../MigrateStoreFileTrackerProcedure.java | 48 ++
.../MigrationStoreFileTracker.java | 96 +++
.../storefiletracker/StoreFileListFile.java | 144 ++++
.../storefiletracker/StoreFileTracker.java | 97 +++
.../storefiletracker/StoreFileTrackerBase.java | 180 +++++
.../storefiletracker/StoreFileTrackerFactory.java | 351 ++++++++++
.../throttle/StoreHotnessProtector.java | 19 +-
.../access/SnapshotScannerHDFSAclHelper.java | 4 +-
.../hbase/snapshot/RestoreSnapshotHelper.java | 104 ++-
.../hadoop/hbase/snapshot/SnapshotManifest.java | 42 +-
.../util/compaction/MajorCompactionRequest.java | 54 +-
.../util/compaction/MajorCompactionTTLRequest.java | 33 +-
.../hbase/util/compaction/MajorCompactor.java | 6 +-
.../hbase/util/compaction/MajorCompactorTTL.java | 4 +-
.../org/apache/hadoop/hbase/wal/WALSplitUtil.java | 7 +-
.../org/apache/hadoop/hbase/TestIOFencing.java | 12 +-
.../org/apache/hadoop/hbase/client/TestAdmin.java | 6 +
.../org/apache/hadoop/hbase/client/TestAdmin3.java | 6 +
.../hbase/client/TestAsyncTableAdminApi.java | 6 +
.../hbase/client/TestAsyncTableAdminApi3.java | 6 +
.../TestCloneSnapshotFromClientCustomSFT.java | 71 ++
...oneSnapshotFromClientCloneLinksAfterDelete.java | 4 +-
.../hbase/client/TestMobSnapshotFromClient.java | 7 +-
.../hbase/client/TestSnapshotFromClient.java | 30 +-
.../hadoop/hbase/master/TestMasterFileSystem.java | 29 +-
.../migrate/TestMigrateStoreFileTracker.java | 107 +++
.../procedure/MasterProcedureTestingUtility.java | 7 +
...=> TestCloneSnapshotProcedureFileBasedSFT.java} | 20 +-
.../master/procedure/TestCreateTableProcedure.java | 33 +
.../master/procedure/TestDeleteTableProcedure.java | 66 --
.../hadoop/hbase/mob/FaultyMobStoreCompactor.java | 3 +-
.../regionserver/TestBrokenStoreFileCleaner.java | 225 +++++++
.../regionserver/TestCacheOnWriteInSchema.java | 6 +-
.../hbase/regionserver/TestCompactorMemLeak.java | 4 +-
.../hbase/regionserver/TestDefaultStoreEngine.java | 6 +-
.../regionserver/TestDirectStoreSplitsMerges.java | 32 +-
.../hadoop/hbase/regionserver/TestHRegion.java | 4 +-
.../hadoop/hbase/regionserver/TestHStore.java | 33 +-
.../hadoop/hbase/regionserver/TestHStoreFile.java | 19 +-
.../regionserver/TestMergesSplitsAddToTracker.java | 262 ++++++++
.../TestRegionMergeTransactionOnCluster.java | 6 +-
.../regionserver/TestStoreFileRefresherChore.java | 3 +-
.../regionserver/TestStoreScannerClosure.java | 6 +-
.../hbase/regionserver/TestStripeStoreEngine.java | 3 +
.../compactions/TestDateTieredCompactor.java | 12 +-
.../compactions/TestStripeCompactionPolicy.java | 12 +-
.../compactions/TestStripeCompactor.java | 12 +-
.../TestChangeStoreFileTracker.java | 242 +++++++
.../TestMigrationStoreFileTracker.java | 194 ++++++
.../TestRegionWithFileBasedStoreFileTracker.java | 108 +++
.../storefiletracker/TestStoreFileTracker.java | 61 ++
.../TestStoreFileTrackerFactory.java | 110 +++
.../hbase/rsgroup/VerifyingRSGroupAdmin.java | 5 +-
.../hbase/snapshot/MobSnapshotTestingUtils.java | 74 ++-
.../hbase/snapshot/SnapshotTestingUtils.java | 16 +-
.../compaction/TestMajorCompactionRequest.java | 20 +-
.../compaction/TestMajorCompactionTTLRequest.java | 18 +-
.../org/apache/hadoop/hbase/wal/TestWALSplit.java | 12 +
hbase-shell/src/main/ruby/hbase/admin.rb | 4 +-
hbase-shell/src/main/ruby/hbase_constants.rb | 1 +
.../src/main/ruby/shell/commands/clone_snapshot.rb | 6 +-
.../hadoop/hbase/thrift2/client/ThriftAdmin.java | 4 +-
pom.xml | 2 +-
.../asciidoc/_chapters/store_file_tracking.adoc | 145 ++++
src/main/asciidoc/book.adoc | 1 +
117 files changed, 4860 insertions(+), 1341 deletions(-)
copy hbase-protocol-shaded/src/main/protobuf/server/{rsgroup/RSGroup.proto => region/StoreFileTracker.proto} (78%)
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/master/migrate/RollingUpgradeChore.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/BrokenStoreFileCleaner.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CreateStoreFileWriterParams.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/DefaultStoreFileTracker.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/FileBasedStoreFileTracker.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrateStoreFileTrackerProcedure.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/MigrationStoreFileTracker.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileListFile.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTracker.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerBase.java
create mode 100644 hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/storefiletracker/StoreFileTrackerFactory.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClientCustomSFT.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/master/migrate/TestMigrateStoreFileTracker.java
copy hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/{TestSCPWithoutMetaWithoutZKCoordinated.java => TestCloneSnapshotProcedureFileBasedSFT.java} (62%)
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBrokenStoreFileCleaner.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMergesSplitsAddToTracker.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestChangeStoreFileTracker.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestMigrationStoreFileTracker.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestRegionWithFileBasedStoreFileTracker.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestStoreFileTracker.java
create mode 100644 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestStoreFileTrackerFactory.java
create mode 100644 src/main/asciidoc/_chapters/store_file_tracking.adoc