[jira] [Created] (OLINGO-1504) JVM crashes due to OutOfMemory encountered: Java heap space

["Devansh Soni (Jira)" <ji...@apache.org>]

Devansh Soni created OLINGO-1504:
------------------------------------

             Summary: JVM crashes due to OutOfMemory encountered: Java heap space 
                 Key: OLINGO-1504
                 URL: https://issues.apache.org/jira/browse/OLINGO-1504
             Project: Olingo
          Issue Type: Bug
          Components: odata4-client
    Affects Versions: (Java) V4 4.7.0
            Reporter: Devansh Soni


Hi 
 The issue occurs for non-paginated OData feeds. The feed I had tested had 100,000 rows and 9 columns. The JVM crashes due to insufficient heap size and I can find the stack trace from the hs_err_pidPID log file. 
{code:java}
ID	Value
26	Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
27	j  java.util.Arrays.copyOf([BI)[B+1
28	j  java.io.ByteArrayOutputStream.grow(I)V+36
29	j  java.io.ByteArrayOutputStream.ensureCapacity(I)V+12
30	j  java.io.ByteArrayOutputStream.write([BII)V+38
31	j  org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;[B)J+19
32	j  org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;I)J+5
33	j  org.apache.commons.io.IOUtils.copyLarge(Ljava/io/InputStream;Ljava/io/OutputStream;)J+5
34	j  org.apache.commons.io.IOUtils.copy(Ljava/io/InputStream;Ljava/io/OutputStream;)I+2
35	j  org.apache.olingo.client.core.communication.response.AbstractODataResponse.getRawResponse()Ljava/io/InputStream;+136
36	j  org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Lorg/apache/olingo/client/api/domain/ClientEntitySetIterator;+23
37	j  org.apache.olingo.client.core.communication.request.retrieve.ODataEntitySetIteratorRequestImpl$ODataEntitySetIteratorResponseImpl.getBody()Ljava/lang/Object;+1
38	j  com.tableausoftware.odata.ODataProtocolImpl.fetchV4(Ljava/net/URI;Z)Lcom/tableausoftware/odata/ODataProtocolImpl$ODataResults;+50
39	j  com.tableausoftware.odata.ODataResultSetV4.nextBlockImpl()Lcom/tableausoftware/data/generated/DataStream$Block;+23
40	j  com.tableausoftware.data.ProtobufResultSet.nextBlock()Lcom/tableausoftware/data/generated/DataStream$Block;+1
41	j  com.tableau.connect.service.QueryTask.readData()V+46
42	j  com.tableau.connect.service.QueryTask.call()Ljava/lang/Void;+9
43	j  com.tableau.connect.service.QueryTask.call()Ljava/lang/Object;+1
44	j  java.util.concurrent.FutureTask.run()V+42
45	j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
46	j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
47	j  java.lang.Thread.run()V+11
48	v  ~StubRoutines::call_stub

{code}
 

The issue happens because there are multiple copies of streams being created in the AbstractODataResponse.[getRawResponse|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/response/AbstractODataResponse.java#L300] method, specifically in the org.apache.commons.io.IOUtils.copy method. To me it seems like it fails to expand the internal byte buffer when it reaches capacity. 
 However, I do not understand why is the response payload being copied in a ByteArrayOutputStream 
{noformat}
org.apache.commons.io.IOUtils.copy(payload, byteArrayOutputStream);
{noformat}
and then again converted into a ByteArrayInputStream. This copying of streams causes creation of multiple byte buffers which fills up the heap memory. 

The ODataEntitySetIteratorResponseImpl.[getBody|https://github.com/apache/olingo-odata4/blob/master/lib/client-core/src/main/java/org/apache/olingo/client/core/communication/request/retrieve/ODataEntitySetIteratorRequestImpl.java#L78] call the getRawResponse() in the constructor call of 
{noformat}
entitySetIterator = new ClientEntitySetIterator<>(
                odataClient, getRawResponse(), ContentType.parse(getContentType()));
      }
{noformat}
 
However the [constructor|https://github.com/apache/olingo-odata4/blob/master/lib/client-api/src/main/java/org/apache/olingo/client/api/domain/ClientEntitySetIterator.java#L79] of ClientEntitySetIterator accepts InputStream. 
So I do not understand the reason behind conversion of the payload into ByteArrayInputStream in the AbstractODataResponse. I am trying to figure the reason why this was done versus returning the payload InputStream as-is. 

For fixing the problem, we have tried increasing the Java heap size but it is just a temporary solution since once the OData feed size increases further beyond a limit, it will fail again.

I also got heap dump for the Java crash and was able to visualize the largest object byte[] in a jprofiler to reach the same conclusion as above. 



  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)