Coprocessor coprocessor execution result saved in buffer as whole - why?

[Asaf Mesika <as...@gmail.com>]

Hi,

I've noticed that in 0.94.7, when you execute a coprocessor, the result
object is converted into a byte buffer, using write() method which is on
the result object.
So, if my result object is 500mb in size, another 500mb is consumed from
the heap, since it is converted to a byte buffer before sent over the wire.

I was wondering why the result object isn't streamed into the Connection.
It's write method gets a DataOutput which is essentially an OutputStream,
so it makes sense that its write will directly to the connection instead of
first to a byte buffer as a whole, and then get sent from there, by chunks.

Any Idea?


The code section is here in HBaseServer.java line 1442:

            call.setResponse(value,

              errorClass == null? Status.SUCCESS: Status.ERROR,

                errorClass, error);

And in HBaseServer$Call.setResponse() method:

      Writable result = null;

      if (value instanceof Writable) {

        result = (Writable) value;

      } else {

        /* We might have a null value and errors. Avoid creating a

         * HbaseObjectWritable, because the constructor fails on null. */

        if (value != null) {

          result = new HbaseObjectWritable(value);

        }

      }


      int size = BUFFER_INITIAL_SIZE;

      if (result instanceof WritableWithSize) {

        // get the size hint.

        WritableWithSize ohint = (WritableWithSize) result;

        long hint = ohint.getWritableSize() + Bytes.SIZEOF_BYTE +

          (2 * Bytes.SIZEOF_INT);

        if (hint > Integer.MAX_VALUE) {

          // oops, new problem.

          IOException ioe =

            new IOException("Result buffer size too large: " + hint);

          errorClass = ioe.getClass().getName();

          error = StringUtils.stringifyException(ioe);

        } else {

          size = (int)hint;

        }

      }


      ByteBufferOutputStream buf = new ByteBufferOutputStream(size);

      DataOutputStream out = new DataOutputStream(buf);

      try {

        // Call id.

        out.writeInt(this.id);

        // Write flag.

        byte flag = (error != null)?

          ResponseFlag.getErrorAndLengthSet():
ResponseFlag.getLengthSetOnly();

        out.writeByte(flag);

        // Place holder for length set later below after we

        // fill the buffer with data.

        out.writeInt(0xdeadbeef);

        out.writeInt(status.state);

      } catch (IOException e) {

        errorClass = e.getClass().getName();

        error = StringUtils.stringifyException(e);

      }