[jira] [Commented] (HBASE-15064) BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache

["deepankar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198473#comment-15198473 ] 

deepankar commented on HBASE-15064:
-----------------------------------

I am still seeing this exception on our servers, I think I found something, what I observe is that a couple of things, in the normal byte buffers (java.nio) the hasRemaining function uses the current position and limit {code}     /**
     * Tells whether there are any elements between the current position and
     * the limit. </p>
     *
     * @return  <tt>true</tt> if, and only if, there is at least one element
     *          remaining in this buffer
     */
    public final boolean hasRemaining() {
        return position < limit;
    }
{code}

But in the MultiByteBuff we have the hasRemaining is not taking care of limit 
{code}
 /**
   * Returns true if there are elements between the current position and the limit
   * @return true if there are elements, false otherwise
   */
  @Override
  public final boolean hasRemaining() {
    return this.curItem.hasRemaining() || this.curItemIndex < this.items.length - 1;
  }
{code}

Also the items array is not changed in the limit(int) function, this means there could be a scenario where the user has asked to limit at the end of first buffer, but the hasRemaining() will still return true, Is there any flaw in my logic here ?

Also in the limit(int) function  in the MultiByteBuff function we are doing 
{code}
  // Normally the limit will try to limit within the last BB item
    int limitedIndexBegin = this.itemBeginPos[this.limitedItemIndex];
    if (limit >= limitedIndexBegin && limit < this.itemBeginPos[this.limitedItemIndex + 1]) {
      this.items[this.limitedItemIndex].limit(limit - limitedIndexBegin);
      return this;
    }
{code}
here I think in the if statement isn't the logic be just {noformat} if (limit  > limitedIndexBegin  && limit < this.itemBeginPos[this.limitedItemIndex + 1]) {noformat} because if somebody is trying to limit at the place which is exactly at the boundary of the limitIndexBuffer then we are also including the last item which does not have any data as you are limiting at 0 (as limit == limitedIndexBegin, which is at the boundary), But then once you have read everything in the previous buffer if the client consults hasRemaining function this will return again true (as curIterm < no_of_items in array) but when you actually try to read anything we will throw BufferUnderFlowException because again the last element has no data. There is similar issue with {{getItemIndexfunction}} when again the {{elemIndex}} matches with the boundary

> BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache
> -------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-15064
>                 URL: https://issues.apache.org/jira/browse/HBASE-15064
>             Project: HBase
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 2.0.0
>            Reporter: deepankar
>            Assignee: Anoop Sam John
>            Priority: Critical
>             Fix For: 2.0.0
>
>         Attachments: HBASE-15064.patch
>
>
> While running the newer patches on our production system, I saw this error come couple of times 
> {noformat}
> ipc.RpcServer: Unexpected throwable object 
> 2016-01-01 16:42:56,090 ERROR [B.defaultRpcServer.handler=20,queue=20,port=60020] ipc.RpcServer: Unexpected throwable object 
> java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:500)
> at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:249)
> at org.apache.hadoop.hbase.nio.MultiByteBuff.get(MultiByteBuff.java:494)
> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decode(FastDiffDeltaEncoder.java:402) 
> at org.apache.hadoop.hbase.io.encoding.FastDiffDeltaEncoder$1.decodeNext(FastDiffDeltaEncoder.java:517) 
> at org.apache.hadoop.hbase.io.encoding.BufferedDataBlockEncoder$BufferedEncodedSeeker.next(BufferedDataBlockEncoder.java:815)
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:138)
> {noformat}
> Looking at the get code 
> {code}
> if (this.curItem.remaining() == 0) {
>       if (items.length - 1 == this.curItemIndex) {
>         // means cur item is the last one and we wont be able to read a long. Throw exception
>         throw new BufferUnderflowException();
>       }
>       this.curItemIndex++;
>       this.curItem = this.items[this.curItemIndex];
>     }
> return this.curItem.get();
> {code}
> Can the new currentItem have zero elements (position == limit), does it make sense to change the {{if}} to {{while}} ? {{while (this.curItem.remaining() == 0)}}. This logic is repeated may make sense abstract to a new function if we plan to change to  {{if}} to {{while}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)