You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Julien Le Dem (JIRA)" <ji...@apache.org> on 2016/08/12 18:23:20 UTC
[jira] [Commented] (ARROW-260)
TestValueVector.testFixedVectorReallocation and
testVariableVectorReallocation are flaky
[ https://issues.apache.org/jira/browse/ARROW-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419253#comment-15419253 ]
Julien Le Dem commented on ARROW-260:
-------------------------------------
from jihoonson: https://github.com/apache/arrow/pull/116#issuecomment-239468610
{quote}
I'm also suffering from this error, and found the reason. The BaseValueVector.MAX_ALLOCATION_SIZE value used in TestValueVector is declared as a static variable which means its value is initialized when the BaseValueVector class is loaded. Obviously, changing BaseValueVector.MAX_ALLOCATION_SIZE by setting the "arrow.vector.max_allocation_bytes" property in TestValueVector doesn't work. As a result, this test tries to allocate a very large array of the size of Integer.MAX_VALUE, which causes OOM.
I think that this test looks fragile, and thus we need to make the allocation size small. However, reducing the allocation size causes another problem. Many tests in TestValueVector expect the OversizedAllocationException when value allocation reaches to MAX_VALUE_ALLOCATION. So, reducing the allocation size makes OversizedAllocation tests difficult.
I would like to separate all these OversizedAllocation tests, but not sure this is a right way. Please give me some help.
{quote}
> TestValueVector.testFixedVectorReallocation and testVariableVectorReallocation are flaky
> ----------------------------------------------------------------------------------------
>
> Key: ARROW-260
> URL: https://issues.apache.org/jira/browse/ARROW-260
> Project: Apache Arrow
> Issue Type: Test
> Components: Java - Vectors
> Reporter: Julien Le Dem
>
> The Travis-ci build has failled several times on these tests.
> It looks like they often throw OOME.
> stacktrace bellow:
> {noformat}
> testFixedVectorReallocation(org.apache.arrow.vector.TestValueVector) Time elapsed: 0.174 sec <<< ERROR!
> java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException> but was<org.apache.arrow.memory.OutOfMemoryException>
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
> at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
> at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
> at org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
> at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
> at org.apache.arrow.vector.UInt4Vector.allocateBytes(UInt4Vector.java:189)
> at org.apache.arrow.vector.UInt4Vector.allocateNew(UInt4Vector.java:171)
> at org.apache.arrow.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:106)
> testVariableVectorReallocation(org.apache.arrow.vector.TestValueVector) Time elapsed: 0.148 sec <<< ERROR!
> java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException> but was<org.apache.arrow.memory.OutOfMemoryException>
> at java.nio.Bits.reserveMemory(Bits.java:658)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
> at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
> at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
> at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
> at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
> at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
> at org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
> at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
> at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
> at org.apache.arrow.vector.VarCharVector.allocateNew(VarCharVector.java:364)
> at org.apache.arrow.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:163)
> Results :
> Tests in error:
> TestValueVector.testFixedVectorReallocation » Unexpected exception, expected<...
> TestValueVector.testVariableVectorReallocation » Unexpected exception, expect...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)