arrow git commit: ARROW-1547: [JAVA] Fix 8x memory over-allocation in BitVector

[we...@apache.org]

Repository: arrow
Updated Branches:
  refs/heads/master 2706b7fc5 -> d4685f420


ARROW-1547: [JAVA] Fix 8x memory over-allocation in BitVector

Problem:

Typically there are 3 ways of specifying the amount of memory needed for vectors.
CASE (1) allocateNew() – here the application doesn't really specify the size of memory or value count. Each vector type has a default value count (4096) and therefore a default size (in bytes) is used in such cases.

For example, for a 4 byte fixed-width vector, we will allocate 32KB of memory for a call to allocateNew().

CASE (2) setInitialCapacity(count) followed by allocateNew() - In this case also the application doesn't specify the value count or size in allocateNew(). However, the call to setInitialCapacity() dictates the amount of memory the subsequent call to allocateNew() will allocate.

For example, we can do setInitialCapacity(1024) and the call to allocateNew() will allocate 4KB of memory for the 4 byte fixed-width vector.

CASE (3) allocateNew(count) - The application is specific about requirements.
For nullable vectors, the above calls also allocate the memory for validity vector.

The problem is that Bit Vector uses a default memory size in bytes of 4096. In other words, we allocate a vector for 4096*8 value count.

In the default case (as explained above), the vector types have a value count of 4096 so we need only 4096 bits (512 bytes) in the bit vector and not really 4096 as the size in bytes.

This happens in CASE 1 where the application depends on the default memory allocation . In such cases, the size of buffer for bit vector is 8x than actually needed

Author: siddharth <si...@dremio.com>

Closes #1109 from siddharthteotia/ARROW-1547 and squashes the following commits:

c92164a0 [siddharth] addressed review comments
f3d12348 [siddharth] ARROW-1547: Fix 8x memory over-allocation in BitVector


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/d4685f42
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/d4685f42
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/d4685f42

Branch: refs/heads/master
Commit: d4685f420a5bb75cf440e6e687f3ae279e66b73f
Parents: 2706b7f
Author: siddharth <si...@dremio.com>
Authored: Tue Sep 19 12:13:54 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Tue Sep 19 12:13:54 2017 -0400

----------------------------------------------------------------------
 .../src/main/java/org/apache/arrow/vector/BitVector.java    | 4 ++--
 .../apache/arrow/vector/TestBufferOwnershipTransfer.java    | 9 +++++++--
 2 files changed, 9 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/d4685f42/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
index e191116..591d13c 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
@@ -44,7 +44,7 @@ public final class BitVector extends BaseDataValueVector implements FixedWidthVe
   private final Mutator mutator = new Mutator();
 
   int valueCount;
-  private int allocationSizeInBytes = INITIAL_VALUE_ALLOCATION;
+  private int allocationSizeInBytes = getSizeFromCount(INITIAL_VALUE_ALLOCATION);
   private int allocationMonitor = 0;
 
   public BitVector(String name, BufferAllocator allocator) {
@@ -176,7 +176,7 @@ public final class BitVector extends BaseDataValueVector implements FixedWidthVe
   @Override
   public void reset() {
     valueCount = 0;
-    allocationSizeInBytes = INITIAL_VALUE_ALLOCATION;
+    allocationSizeInBytes = getSizeFromCount(INITIAL_VALUE_ALLOCATION);
     allocationMonitor = 0;
     zeroVector();
     super.reset();

http://git-wip-us.apache.org/repos/asf/arrow/blob/d4685f42/java/vector/src/test/java/org/apache/arrow/vector/TestBufferOwnershipTransfer.java
----------------------------------------------------------------------
diff --git a/java/vector/src/test/java/org/apache/arrow/vector/TestBufferOwnershipTransfer.java b/java/vector/src/test/java/org/apache/arrow/vector/TestBufferOwnershipTransfer.java
index 54fc306..1a801a6 100644
--- a/java/vector/src/test/java/org/apache/arrow/vector/TestBufferOwnershipTransfer.java
+++ b/java/vector/src/test/java/org/apache/arrow/vector/TestBufferOwnershipTransfer.java
@@ -47,7 +47,9 @@ public class TestBufferOwnershipTransfer {
     v1.makeTransferPair(v2).transfer();
 
     assertEquals(0, childAllocator1.getAllocatedMemory());
-    assertEquals(5 * 4096, childAllocator2.getAllocatedMemory());
+    int expectedBitVector = 512;
+    int expectedValueVector = 4096*4;
+    assertEquals(expectedBitVector + expectedValueVector, childAllocator2.getAllocatedMemory());
   }
 
   @Test
@@ -66,7 +68,10 @@ public class TestBufferOwnershipTransfer {
     v1.makeTransferPair(v2).transfer();
 
     assertEquals(0, childAllocator1.getAllocatedMemory());
-    int expected = 8 * 4096 + 4 * 4096 + 4096;
+    int expectedValueVector = 4096*8;
+    int expectedOffsetVector = 4096*4;
+    int expectedBitVector = 512;
+    int expected = expectedBitVector + expectedOffsetVector + expectedValueVector;
     assertEquals(expected, childAllocator2.getAllocatedMemory());
   }