[jira] [Resolved] (HBASE-17138) Backport read-path offheap (HBASE-11425) to branch-1

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-17138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Kyle Purtell resolved HBASE-17138.
-----------------------------------------
      Assignee:     (was: Yu Sun)
    Resolution: Won't Fix

> Backport read-path offheap (HBASE-11425) to branch-1
> ----------------------------------------------------
>
>                 Key: HBASE-17138
>                 URL: https://issues.apache.org/jira/browse/HBASE-17138
>             Project: HBase
>          Issue Type: Improvement
>          Components: regionserver
>            Reporter: Yu Li
>            Priority: Major
>         Attachments: 0001-fix-EHB-511-Resolve-client-compatibility-issue-introduced-by-offheap-change.patch, 0001-to-EHB-446-offheap-hfile-format-should-keep-compatible-v3.patch, 0001-to-EHB-456-Cell-should-be-compatible-with-branch-1.1.2.patch
>
>
> From the [thread|http://mail-archives.apache.org/mod_mbox/hbase-user/201611.mbox/%3CCAM7-19%2Bn7cEiY4H9iLQ3N9V0NXppOPduZwk-hhgNLEaJfiV3kA%40mail.gmail.com%3E] of sharing our experience and performance data of read-path offheap usage in Alibaba search, we could see people are positive to have HBASE-11425 in branch-1, so I'd like to create a JIRA and move the discussion and decision making here.
> Echoing some comments from the mail thread:
> Bryan:
> Is the backported patch available anywhere? If it ends up not getting officially backported to branch-1 due to 2.0 around the corner, some of us who build our own deploy may want to integrate into our builds
> Andrew:
> Yes, please, the patches will be useful to the community even if we decide not to backport into an official 1.x release.
> Enis:
> I don't see any reason why we cannot backport to branch-1.
> Ted:
> Opening a JIRA would be fine. This makes it easier for people to obtain the patch(es)
> Nick:
> From the DISCUSS thread re: EOL of 1.1, it seems we'll continue to
> support 1.x releases for some time... I would guess these will be
> maintained until 2.2 at least. Therefore, offheap patches that have seen
> production exposure seem like a reasonable candidate for backport, perhaps in a 1.4 or 1.5 release timeframe.
> Anoop:
> Because of some compatibility issues, we decide that this will be done in 2.0 only..  Ya as Andy said, it would be great to share the 1.x backported patches.
> The following is all the jira ids we have back ported:
> HBASE-10930 Change Filters and GetClosestRowBeforeTracker to work with Cells (Ram)
> HBASE-13373 Squash HFileReaderV3 together with HFileReaderV2 and AbstractHFileReader; ditto for Scanners and BlockReader, etc.
> HBASE-13429 Remove deprecated seek/reseek methods from HFileScanner.
> HBASE-13450 - Purge RawBytescomparator from the writers and readers for HBASE-10800 (Ram)
> HBASE-13501 - Deprecate/Remove getComparator() in HRegionInfo.
> HBASE-12048 Remove deprecated APIs from Filter.
> HBASE-10800 - Use CellComparator instead of KVComparator (Ram)
> HBASE-13679 Change ColumnTracker and SQM to deal with Cell instead of byte[], int, int.
> HBASE-13642 Deprecate RegionObserver#postScannerFilterRow CP hook with byte[],int,int args in favor of taking Cell arg.
> HBASE-13641 Deperecate Filter#filterRowKey(byte[] buffer, int offset, int length) in favor of filterRowKey(Cell firstRowCell).
> HBASE-13827 Delayed scanner close in KeyValueHeap and StoreScanner.
> HBASE-13871 Change RegionScannerImpl to deal with Cell instead of byte[], int, int.
> HBASE-11911 Break up tests into more fine grained categories (Alex Newman)
> HBASE-12059 Create hbase-annotations module
> HBASE-12106 Move test annotations to test artifact (Enis Soztutar)
> HBASE-13916 Create MultiByteBuffer an aggregation of ByteBuffers.
> HBASE-15679 Assertion on wrong variable in TestReplicationThrottler#testThrottling
> HBASE-13931 Move Unsafe based operations to UnsafeAccess.
> HBASE-12345 Unsafe based ByteBuffer Comparator.
> HBASE-13998 Remove CellComparator#compareRows(byte[] left, int loffset, int llength, byte[] right, int roffset, int rlength).
> HBASE-13998 Remove CellComparator#compareRows()- Addendum to fix javadoc warn
> HBASE-13579 Avoid isCellTTLExpired() for NO-TAG cases (partially backport this patch)
> HBASE-13448 New Cell implementation with cached component offsets/lengths.
> HBASE-13387 Add ByteBufferedCell an extension to Cell.
> HBASE-13387 Add ByteBufferedCell an extension to Cell - addendum.
> HBASE-12650 Move ServerName to hbase-common module (partially backport this patch)
> HBASE-12296 Filters should work with ByteBufferedCell.
> HBASE-14120 ByteBufferUtils#compareTo small optimization.
> HBASE-13510 - Purge ByteBloomFilter (Ram)
> HBASE-13451 - Make the HFileBlockIndex blockKeys to Cells so that it could be easy to use in the CellComparators (Ram)
> HBASE-13614 - Avoid temp KeyOnlyKeyValue temp objects creations in read hot path (Ram)
> HBASE-13939 - Make HFileReaderImpl.getFirstKeyInBlock() to return a Cell (Ram)
> HBASE-13307 Making methods under ScannerV2#next inlineable, faster
> HBASE-14020 Unsafe based optimized write in ByteBufferOutputStream.
> HBASE-13977 - Convert getKey and related APIs to Cell (Ram)
> HBASE-11927 Use Native Hadoop Library for HFile checksum. (Apekshit)
> HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram)
> HBASE-12084 Remove deprecated APIs from Result.
> HBASE-12084 Remove deprecated APIs from Result - shell addendum
> HBASE-13754 Allow non KeyValue Cell types also to oswrite.
> HBASE-14047 - Cleanup deprecated APIs from Cell class (Ashish Singhi)
> HBASE-13817 ByteBufferOuputStream - add writeInt support.
> HBASE-12374 Change DBEs to work with new BB based cell.
> HBASE-14116 Change ByteBuff.getXXXStrictlyForward to relative position based reads
> HBASE-14073 TestRemoteTable.testDelete failed in the latest trunk code.(Jingcheng)
> HBASE-13926 Close the scanner only after Call#setResponse.
> HBASE-12295 Prevent block eviction under us if reads are in progress from the BBs (Ram)
> HBASE-12295 - Addendum for multiGets to add the call back(Ram)
> HBASE-14001 Optimize write(OutputStream out, boolean withTags) for SizeCachedNoTagsKeyValue.
> HBASE-14063 Use BufferBackedCell in read path after HBASE-12213 and HBASE-12295 (ram)
> HBASE-14202 Reduce garbage we create.
> HBASE-14099 StoreFile.passesKeyRangeFilter need not create Cells from the Scan's start and stop Row (Ram)
> HBASE-14144 - Bloomfilter path to work with Byte buffered cells (Ram)
> HBASE-14395 Short circuit last byte check in CellUtil#matchingXXX methods for ByteBufferedCells.
> HBASE-12298 Support BB usage in PrefixTree (Ram)
> HBASE-14398 - Create the fake keys required in the scan path to avoid copy to byte[] (Ram)
> HBASE-14480 Small optimization in SingleByteBuff.
> HBASE-14590 Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name (Anoop Sam John)
> HBASE-14636 Clear HFileScannerImpl#prevBlocks in between Compaction flow.
> HBASE-14832 Ensure write paths work with ByteBufferedCells in case of compaction (Ram)
> HBASE-14524 Short-circuit comparison of rows in CellComparator. (Lars Francke)
> HBASE-14188 - Read path optimizations after HBASE-11425 profiling (Ram)
> HBASE-14188- Read path optimizations after HBASE-11425 profiling-Addendum(Ram)
> HBASE-12593 Tags to work with ByteBuffer.
> HBASE-15077 Support OffheapKV write in compaction with out copying data on heap.
> HBASE-14660 AssertionError found when using offheap BucketCache with assertion enabled (ram)
> HBASE-15076 Add getScanner(Scan scan, List<KeyValueScanner> additionalScanners) API into Region interface (Anoop Sam John
> HBASE-15735(also particial HBASE-13893) Tightening of the CP contract.
> HBASE-15785 Unnecessary lock in ByteBufferArray.
> HBASE-15760 TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner fails in master branch (Ram)
> HBASE-15379 Fake cells created in read path not implementing SettableSequenceId
> HBASE-16609 Fake cells EmptyByteBufferedCell created in read path not implementing SettableSequenceId
> HBASE-14940 Make our unsafe based ops more safe
> HBASE-15063 Bug in MultiByteBuf#toBytes. (deepankar)
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache.
> HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache - Addendum.
> HBASE-15253 Small bug in CellUtil.matchingRow(Cell, byte[]) (Ram)
> HBASE-16704 Scan will be broken while working with DBE and KeyValueCodecWithTags
> the above jira ids are listed in chronological order I have backport，and there are also some patch i dont list, such as:
> 1. keep Cell api compatible with our existing code.
> 2. hfile format related compatibility issues.
> 3. client compatibility issue



--
This message was sent by Atlassian Jira
(v8.20.7#820007)