You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@olingo.apache.org by "Devansh Soni (Jira)" <ji...@apache.org> on 2020/12/23 20:03:00 UTC
[jira] [Updated] (OLINGO-1504) JVM crashes due to OutOfMemory
encountered: Java heap space
[ https://issues.apache.org/jira/browse/OLINGO-1504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devansh Soni updated OLINGO-1504:
---------------------------------
Attachment: JprofilerHeapWalkerGraph.png
HeapDumpLargestObjects.png
> JVM crashes due to OutOfMemory encountered: Java heap space
> ------------------------------------------------------------
>
> Key: OLINGO-1504
> URL: https://issues.apache.org/jira/browse/OLINGO-1504
> Project: Olingo
> Issue Type: Bug
> Components: odata4-client
> Affects Versions: (Java) V4 4.7.0
> Reporter: Devansh Soni
> Priority: Major
> Attachments: HeapDumpLargestObjects.png, JprofilerHeapWalkerGraph.png
>
>
> Hi
> The issue occurs for non-paginated OData feeds. The feed I had tested had 100,000 rows and 9 columns. The JVM crashes due to insufficient heap size and I can find the stack trace from the hs_err_pidPID log file.
> {code:java}
> ID Value
> 26 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> 27 j java.util.Arrays.copyOf([BI)[B+1
> 28 j java.io.ByteArrayOutputStream.grow(I)V+36
> 29 j java.io.ByteArrayOutputStream.ensureCapacity(I)V+12
> 30 j java.io.ByteArrayOutputStream.write([BII)V+38
> 31 j org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;[B)J+19
> 32 j org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;I)J+5
> 33 j org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J+5
> 34 j org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I+2
> 35 j org.apache.olingo.client.core.communication.response.AbstractODataResponse.getRawResponse()Ljava/io/InputStream;+136
> 36 j org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Lorg/apache/olingo/client/api/domain/ClientEntitySetIterator;+23
> 37 j org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Ljava/lang/Object;+1
> 38 j com.tableausoftware.odata.ODataProtocolImpl.fetchV4(Ljava/net/URI;Z)Lcom/tableausoftware/odata/ODataProtocolImpl$ODataResults;+50
> 39 j com.tableausoftware.odata.ODataResultSetV4.nextBlockImpl()Lcom/tableausoftware/data/generated/DataStream$Block;+23
> 40 j com.tableausoftware.data.ProtobufResultSet.nextBlock()Lcom/tableausoftware/data/generated/DataStream$Block;+1
> 41 j com.tableau.connect.service.QueryTask.readData()V+46
> 42 j com.tableau.connect.service.QueryTask.call()Ljava/lang/Void;+9
> 43 j com.tableau.connect.service.QueryTask.call()Ljava/lang/Object;+1
> 44 j java.util.concurrent.FutureTask.run()V+42
> 45 j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
> 46 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
> 47 j java.lang.Thread.run()V+11
> 48 v ~StubRoutines::call_stub
> {code}
>
> The issue happens because there are multiple copies of streams being created in the AbstractODataResponse.[getRawResponse|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/response/AbstractODataResponse.java#L300] method, specifically in the org.apache.commons.io.IOUtils.copy method. To me it seems like it fails to expand the internal byte buffer when it reaches capacity.
> However, I do not understand why is the response payload being copied in a ByteArrayOutputStream
> {noformat}
> org.apache.commons.io.IOUtils.copy(payload, byteArrayOutputStream);
> {noformat}
> and then again converted into a ByteArrayInputStream. This copying of streams causes creation of multiple byte buffers which fills up the heap memory.
> The ODataEntitySetIteratorResponseImpl.[getBody|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/request/retrieve/ODataEntitySetIteratorRequestImpl.java#L78] call the getRawResponse() in the constructor call of
> {noformat}
> entitySetIterator = new ClientEntitySetIterator<>(
> odataClient, getRawResponse(), ContentType.parse(getContentType()));
> }
> {noformat}
>
> However the [constructor|https://github.com/apache/olingo-odata4/blob/master/lib/client-api/src/main/java/org/apache/olingo/client/api/domain/ClientEntitySetIterator.java#L79] of ClientEntitySetIterator accepts InputStream.
> So I do not understand the reason behind conversion of the payload into ByteArrayInputStream in the AbstractODataResponse. I am trying to figure the reason why this was done versus returning the payload InputStream as-is.
> For fixing the problem, we have tried increasing the Java heap size but it is just a temporary solution since once the OData feed size increases further beyond a limit, it will fail again.
> I also got heap dump for the Java crash and was able to visualize the largest object byte[] in a jprofiler to reach the same conclusion as above.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)