[jira] [Comment Edited] (FLUME-2404) Default maxReadBufferBytes might cause OOM and cause scribe source exit

["chenshangan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14043190#comment-14043190 ] 

chenshangan edited comment on FLUME-2404 at 6/25/14 8:26 AM:
-------------------------------------------------------------

FrameBuffer#read
{code}
public boolean read() {
  if (state_ == FrameBufferState.READING_FRAME_SIZE) {
    // try to read the frame size completely
    if (!internalRead()) {
      return false;
    }
    // if the frame size has been read completely, then prepare to read the
    // actual frame.
    if (buffer_.remaining() == 0) {
      // pull out the frame size as an integer.
      int frameSize = buffer_.getInt(0);
      if (frameSize <= 0) {
        LOGGER.error("Read an invalid frame size of " + frameSize
            + ". Are you using TFramedTransport on the client side?");
        return false;
      }
      // if this frame will always be too large for this server, log the
      // error and close the connection.
      if (frameSize > MAX_READ_BUFFER_BYTES) {
        LOGGER.error("Read a frame size of " + frameSize
            + ", which is bigger than the maximum allowable buffer size for ALL connections.");
        return false;
      }
      // if this frame will push us over the memory limit, then return.
      // with luck, more memory will free up the next time around.
      if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
        return true;
      }
      // increment the amount of memory allocated to read buffers
      readBufferBytesAllocated.addAndGet(frameSize);
      // reallocate the readbuffer as a frame-sized buffer
      buffer_ = ByteBuffer.allocate(frameSize);
      state_ = FrameBufferState.READING_FRAME;
    } else {
      // this skips the check of READING_FRAME state below, since we can't
      // possibly go on to that state if there's data left to be read at
      // this one.
      return true;
    }
  }

{code}

default MAX_READ_BUFFER_BYTES is Long.MAX_VALUE，obviously it's problematic



was (Author: chenshangan521@163.com):
{code}
public boolean read() {
  if (state_ == FrameBufferState.READING_FRAME_SIZE) {
    // try to read the frame size completely
    if (!internalRead()) {
      return false;
    }
    // if the frame size has been read completely, then prepare to read the
    // actual frame.
    if (buffer_.remaining() == 0) {
      // pull out the frame size as an integer.
      int frameSize = buffer_.getInt(0);
      if (frameSize <= 0) {
        LOGGER.error("Read an invalid frame size of " + frameSize
            + ". Are you using TFramedTransport on the client side?");
        return false;
      }
      // if this frame will always be too large for this server, log the
      // error and close the connection.
      if (frameSize > MAX_READ_BUFFER_BYTES) {
        LOGGER.error("Read a frame size of " + frameSize
            + ", which is bigger than the maximum allowable buffer size for ALL connections.");
        return false;
      }
      // if this frame will push us over the memory limit, then return.
      // with luck, more memory will free up the next time around.
      if (readBufferBytesAllocated.get() + frameSize > MAX_READ_BUFFER_BYTES) {
        return true;
      }
      // increment the amount of memory allocated to read buffers
      readBufferBytesAllocated.addAndGet(frameSize);
      // reallocate the readbuffer as a frame-sized buffer
      buffer_ = ByteBuffer.allocate(frameSize);
      state_ = FrameBufferState.READING_FRAME;
    } else {
      // this skips the check of READING_FRAME state below, since we can't
      // possibly go on to that state if there's data left to be read at
      // this one.
      return true;
    }
  }

{code}

default MAX_READ_BUFFER_BYTES is Long.MAX_VALUE，obviously it's problematic


> Default maxReadBufferBytes might cause OOM and cause scribe source exit
> -----------------------------------------------------------------------
>
>                 Key: FLUME-2404
>                 URL: https://issues.apache.org/jira/browse/FLUME-2404
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.5.0
>            Reporter: chenshangan
>            Assignee: chenshangan
>         Attachments: FLUME-2404.patch
>
>
> We're using scribe source,  some hacker like infosec guys send some malicious frames to flume with the frame size field set to a very big integer, then the thrift server inside scribe source will exit due to OOM. Then scribe source will keep wait_close state and can not accept any connection.



--
This message was sent by Atlassian JIRA
(v6.2#6252)