[GitHub] [beam] damccorm opened a new issue, #20771: "OutOfMemoryError: Direct buffer memory" in streaming dataflow at Google Cloud

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #20771:
URL: https://github.com/apache/beam/issues/20771

   Hi ALL.
   
   It seems there is some regression error in the last Beam version – 2.27.0.
   
   There is a streaming dataflow that is runned in Google Cloud.
   This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0.
   But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below:
   
   ```
   
   Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory
   org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932)
   org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972)
   org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149)
   Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175)
   java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
   org.conscrypt.ConscryptEngineSocket$SSLInputStream.<init>(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197)
   org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290)
   java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
   java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
   java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
   java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
   java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
   java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367)
   java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
   com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
   com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541)
   com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474)
   com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591)
   org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857)
   org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304)
   java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834)
   
   ```
   
   The error is rare. 99% of a time the dataflow works without this (actually any) error.
   
   This issue blocks us to make upgrade of the Beam library for the dataflow.
   
    
   
   Imported from Jira [BEAM-11825](https://issues.apache.org/jira/browse/BEAM-11825). Original Jira may contain additional context.
   Reported by: trunikov.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org