You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Alex Feinberg (JIRA)" <ji...@apache.org> on 2012/08/03 21:23:02 UTC
[jira] [Created] (HBASE-6508) Filter out edits at log split time
Alex Feinberg created HBASE-6508:
------------------------------------
Summary: Filter out edits at log split time
Key: HBASE-6508
URL: https://issues.apache.org/jira/browse/HBASE-6508
Project: HBase
Issue Type: Improvement
Components: master, regionserver, wal
Affects Versions: 0.89-fb
Reporter: Alex Feinberg
Assignee: Alex Feinberg
Fix For: 0.89-fb
At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
This patch does the following:
1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
2) Send the map to master it as a part of the region server report.
3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6508) Filter out edits at log split time
Posted by "Alex Feinberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Feinberg resolved HBASE-6508.
----------------------------------
Resolution: Fixed
Done. Will be merged to 89-fb overnight.
> Filter out edits at log split time
> ----------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450187#comment-13450187 ]
Hudson commented on HBASE-6508:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #163 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/163/])
HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
* /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Sergey Shelukhin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481750#comment-13481750 ]
Sergey Shelukhin commented on HBASE-6508:
-----------------------------------------
It appears that new message will not be processed correctly by the old server on new protocols.
Looks like we won't be porting this...
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Sergey Shelukhin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480330#comment-13480330 ]
Sergey Shelukhin commented on HBASE-6508:
-----------------------------------------
Hi. Any interest/objections with regard to porting this to 0.94?
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480377#comment-13480377 ]
stack commented on HBASE-6508:
------------------------------
[~sershe] Ted has a point. This patch adds methods to rpc Interfaces. You'd need to ensure that 0.94.2 client can talk to a 0.94.3 server and that a 0.94.3 master would work with 0.94.2 regionservers so can do a rolling restart.
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444809#comment-13444809 ]
Hudson commented on HBASE-6508:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #154 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/154/])
HBASE-6659 Port HBASE-6508 Filter out edits at log split time, revert (Revision 1378752)
HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631)
Result = FAILURE
Tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
* /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
Tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
* /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444217#comment-13444217 ]
Hudson commented on HBASE-6508:
-------------------------------
Integrated in HBase-TRUNK #3291 (See [https://builds.apache.org/job/HBase-TRUNK/3291/])
HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1378631)
Result = FAILURE
Tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
* /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480348#comment-13480348 ]
Ted Yu commented on HBASE-6508:
-------------------------------
Would this feature affect rolling restart ?
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480341#comment-13480341 ]
stack commented on HBASE-6508:
------------------------------
[~sershe] Anything that helps w/ MTTR is good. This is in trunk already I believe.
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log
split time
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449886#comment-13449886 ]
Hudson commented on HBASE-6508:
-------------------------------
Integrated in HBase-TRUNK #3310 (See [https://builds.apache.org/job/HBase-TRUNK/3310/])
HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684)
Result = FAILURE
tedyu :
Files :
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto
* /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
> [0.89-fb] Filter out edits at log split time
> --------------------------------------------
>
> Key: HBASE-6508
> URL: https://issues.apache.org/jira/browse/HBASE-6508
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver, wal
> Affects Versions: 0.89-fb
> Reporter: Alex Feinberg
> Assignee: Alex Feinberg
> Fix For: 0.89-fb
>
>
> At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region.
> This patch does the following:
> 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map.
> 2) Send the map to master it as a part of the region server report.
> 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region.
> 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time.
> 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report.
> This can reduce downtime when a regionserver goes down quite a bit.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira