You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Yu Sun (JIRA)" <ji...@apache.org> on 2016/12/06 11:11:31 UTC
[jira] [Updated] (HBASE-17138) Backport read-path offheap
(HBASE-11425) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yu Sun updated HBASE-17138:
---------------------------
Description:
From the [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] of sharing our experience and performance data of read-path offheap usage in Alibaba search, we could see people are positive to have HBASE-11425 in branch-1, so I'd like to create a JIRA and move the discussion and decision making here.
Echoing some comments from the mail thread:
Bryan:
Is the backported patch available anywhere? If it ends up not getting officially backported to branch-1 due to 2.0 around the corner, some of us who build our own deploy may want to integrate into our builds
Andrew:
Yes, please, the patches will be useful to the community even if we decide not to backport into an official 1.x release.
Enis:
I don't see any reason why we cannot backport to branch-1.
Ted:
Opening a JIRA would be fine. This makes it easier for people to obtain the patch(es)
Nick:
From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
support 1.x releases for some time... I would guess these will be
maintained until 2.2 at least. Therefore, offheap patches that have seen
production exposure seem like a reasonable candidate for backport, perhaps in a 1.4 or 1.5 release timeframe.
Anoop:
Because of some compatibility issues, we decide that this will be done in 2.0 only.. Ya as Andy said, it would be great to share the 1.x backported patches.
The following is all the jira ids we have back ported:
HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells (Ram)
HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and AbstractHFileReader; ditto for Scanners and BlockReader, etc.
HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
HBASE-13450 - Purge RawBytescomparator from the writers and readers for HBASE-10800 (Ram)
HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
HBASE-12048 Remove deprecated APIs from Filter.
HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], int, int.
HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with byte[],int,int args in favor of taking Cell arg.
HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int length) in favor of filterRowKey(Cell firstRowCell).
HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], int, int.
HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
HBASE-12059 Create hbase-annotations module
HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
HBASE-15679 Assertion on wrong variable in TestReplicationThrottler#testThrottling
HBASE-13931 Move Unsafe based operations to UnsafeAccess.
HBASE-12345 Unsafe based ByteBuffer Comparator.
HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int llength, byte[] right, int roffset, int rlength).
HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn
HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport this patch)
HBASE-13448 New Cell implementation with cached component offsets/lengths.
HBASE-13387 Add ByteBufferedCell an extension to Cell.
HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
HBASE-12650 Move ServerName to hbase-common module (partially backport this patch)
HBASE-12296 Filters should work with ByteBufferedCell.
HBASE-14120 ByteBufferUtils#compareTo small optimization.
HBASE-13510 - Purge ByteBloomFilter (Ram)
HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be easy to use in the CellComparators (Ram)
HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot path (Ram)
HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram)
HBASE-13307 Making methods under ScannerV2#next inlineable, faster
HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream.
HBASE-13977 - Convert getKey and related APIs to Cell (Ram)
HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit)
HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram)
HBASE-12084 Remove deprecated APIs from Result.
HBASE-12084 Remove deprecated APIs from Result - shell addendum
HBASE-13754 Allow non KeyValue Cell types also to oswrite.
HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi)
HBASE-13817 ByteBufferOuputStream - add writeInt support.
HBASE-12374 Change DBEs to work with new BB based cell.
HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position based reads
HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk code.(Jingcheng)
HBASE-13926 Close the scanner only after Call#setResponse.
HBASE-12295 Prevent block eviction under us if reads are in progress from the BBs (Ram)
HBASE-12295 - Addendum for multiGets to add the call back(Ram)
HBASE-14001 Optimize write(OutputStream out, boolean withTags) for SizeCachedNoTagsKeyValue.
HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and HBASE-12295 (ram)
HBASE-14202 Reduce garbage we create.
HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row (Ram)
HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram)
HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for ByteBufferedCells.
HBASE-12298 Support BB usage in PrefixTree (Ram)
HBASE-14398 - Create the fake keys required in the scan path to avoid copy to byte[] (Ram)
HBASE-14480 Small optimization in SingleByteBuff.
HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name (Anoop Sam John)
HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow.
HBASE-14832 Ensure write paths work with ByteBufferedCells in case of compaction (Ram)
HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Francke)
HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram)
HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum(Ram)
HBASE-12593 Tags to work with ByteBuffer.
HBASE-15077 Support OffheapKV write in compaction with out copying data on heap.
HBASE-14660 AssertionError found when using offheap BucketCache with assertion enabled (ram)
HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner> additionalScanners) API into Region interface (Anoop Sam John
HBASE-15735(also particial HBASE-13893) Tightening of the CP contract.
HBASE-15785 Unnecessary lock in ByteBufferArray.
HBASE-15760 TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner fails in master branch (Ram)
HBASE-15379 Fake cells created in read path not implementing SettableSequenceId
HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not implementing SettableSequenceId
HBASE-14940 Make our unsafe based ops more safe
HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar)
HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache.
HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache - Addendum.
HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram)
HBASE-16704 Scan will be broken while working with DBE and KeyValueCodecWithTags
the above jira ids are listed in chronological order I have backport，and there are also some patch i dont list, such as:
1. keep Cell api compatible with our existing code.
2. hfile format related compatibility issues.
3. client compatibility issue
was:
From the [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] of sharing our experience and performance data of read-path offheap usage in Alibaba search, we could see people are positive to have HBASE-11425 in branch-1, so I'd like to create a JIRA and move the discussion and decision making here.
Echoing some comments from the mail thread:
Bryan:
Is the backported patch available anywhere? If it ends up not getting officially backported to branch-1 due to 2.0 around the corner, some of us who build our own deploy may want to integrate into our builds
Andrew:
Yes, please, the patches will be useful to the community even if we decide not to backport into an official 1.x release.
Enis:
I don't see any reason why we cannot backport to branch-1.
Ted:
Opening a JIRA would be fine. This makes it easier for people to obtain the patch(es)
Nick:
From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
support 1.x releases for some time... I would guess these will be
maintained until 2.2 at least. Therefore, offheap patches that have seen
production exposure seem like a reasonable candidate for backport, perhaps in a 1.4 or 1.5 release timeframe.
Anoop:
Because of some compatibility issues, we decide that this will be done in 2.0 only.. Ya as Andy said, it would be great to share the 1.x backported patches.
> Backport read-path offheap (HBASE-11425) to branch-1
> ----------------------------------------------------
>
> Key: HBASE-17138
> URL: https://issues.apache.org/jira/browse/HBASE-17138
> Project: HBase
> Issue Type: Improvement
> Reporter: Yu Li
> Assignee: Yu Sun
>
> From the [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] of sharing our experience and performance data of read-path offheap usage in Alibaba search, we could see people are positive to have HBASE-11425 in branch-1, so I'd like to create a JIRA and move the discussion and decision making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting officially backported to branch-1 due to 2.0 around the corner, some of us who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0 only.. Ya as Andy said, it would be great to share the 1.x backported patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int llength, byte[] right, int roffset, int rlength).
> HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn
> HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport this patch)
> HBASE-13448 New Cell implementation with cached component offsets/lengths.
> HBASE-13387 Add ByteBufferedCell an extension to Cell.
> HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
> HBASE-12650 Move ServerName to hbase-common module (partially backport this patch)
> HBASE-12296 Filters should work with ByteBufferedCell.
> HBASE-14120 ByteBufferUtils#compareTo small optimization.
> HBASE-13510 - Purge ByteBloomFilter (Ram)
> HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be easy to use in the CellComparators (Ram)
> HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot path (Ram)
> HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram)
> HBASE-13307 Making methods under ScannerV2#next inlineable, faster
> HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream.
> HBASE-13977 - Convert getKey and related APIs to Cell (Ram)
> HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit)
> HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram)
> HBASE-12084 Remove deprecated APIs from Result.
> HBASE-12084 Remove deprecated APIs from Result - shell addendum
> HBASE-13754 Allow non KeyValue Cell types also to oswrite.
> HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi)
> HBASE-13817 ByteBufferOuputStream - add writeInt support.
> HBASE-12374 Change DBEs to work with new BB based cell.
> HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position based reads
> HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk code.(Jingcheng)
> HBASE-13926 Close the scanner only after Call#setResponse.
> HBASE-12295 Prevent block eviction under us if reads are in progress from the BBs (Ram)
> HBASE-12295 - Addendum for multiGets to add the call back(Ram)
> HBASE-14001 Optimize write(OutputStream out, boolean withTags) for SizeCachedNoTagsKeyValue.
> HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and HBASE-12295 (ram)
> HBASE-14202 Reduce garbage we create.
> HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row (Ram)
> HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram)
> HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for ByteBufferedCells.
> HBASE-12298 Support BB usage in PrefixTree (Ram)
> HBASE-14398 - Create the fake keys required in the scan path to avoid copy to byte[] (Ram)
> HBASE-14480 Small optimization in SingleByteBuff.
> HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name (Anoop Sam John)
> HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow.
> HBASE-14832 Ensure write paths work with ByteBufferedCells in case of compaction (Ram)
> HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Francke)
> HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram)
> HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum(Ram)
> HBASE-12593 Tags to work with ByteBuffer.
> HBASE-15077 Support OffheapKV write in compaction with out copying data on heap.
> HBASE-14660 AssertionError found when using offheap BucketCache with assertion enabled (ram)
> HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner> additionalScanners) API into Region interface (Anoop Sam John
> HBASE-15735(also particial HBASE-13893) Tightening of the CP contract.
> HBASE-15785 Unnecessary lock in ByteBufferArray.
> HBASE-15760 TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner fails in master branch (Ram)
> HBASE-15379 Fake cells created in read path not implementing SettableSequenceId
> HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not implementing SettableSequenceId
> HBASE-14940 Make our unsafe based ops more safe
> HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar)
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache.
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache - Addendum.
> HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram)
> HBASE-16704 Scan will be broken while working with DBE and KeyValueCodecWithTags
> the above jira ids are listed in chronological order I have backport，and there are also some patch i dont list, such as:
> 1. keep Cell api compatible with our existing code.
> 2. hfile format related compatibility issues.
> 3. client compatibility issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)