[jira] [Created] (BEAM-11825) "OutOfMemoryError: Direct buffer memory" in streaming dataflow at Google Cloud

["DMYTRO TRUNYKOV (Jira)" <ji...@apache.org>]

DMYTRO TRUNYKOV created BEAM-11825:
--------------------------------------

             Summary: "OutOfMemoryError: Direct buffer memory" in streaming dataflow at Google Cloud
                 Key: BEAM-11825
                 URL: https://issues.apache.org/jira/browse/BEAM-11825
             Project: Beam
          Issue Type: Bug
          Components: beam-community, beam-model
    Affects Versions: 2.27.0
         Environment: Google Cloud.

A list of linked to the dataflow libraries:

proto-google-cloud-spanner-v1-2.0.2.jar
grpc-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
commons-io-2.6.jar
grpc-alts-1.34.0.jar
beam-runners-core-construction-java-2.27.0.jar
beam-vendor-grpc-1_26_0-0.3.jar
proto-google-cloud-spanner-admin-instance-v1-2.0.2.jar
vavr-0.10.2.jar
jackson-dataformat-yaml-2.12.0.jar
grpc-api-1.34.0.jar
jaxb-api-2.3.0.jar
hamcrest-core-2.1.jar
grpc-netty-1.32.2.jar
kamon-bundle_2.12-2.1.4.jar
beam-sdks-java-extensions-protobuf-2.27.0.jar
grpc-netty-shaded-1.34.0.jar
jackson-databind-2.12.0.jar
api-common-1.10.1.jar
paranamer-2.7.jar
google-cloud-bigquery-1.122.2.jar
google-auth-library-credentials-0.22.0.jar
log4j-api-2.6.2.jar
google-api-services-cloudresourcemanager-v1-rev20200720-1.30.10.jar
proto-google-cloud-spanner-admin-database-v1-2.0.2.jar
netty-codec-http2-4.1.51.Final.jar
perfmark-api-0.19.0.jar
netty-handler-4.1.51.Final.jar
checker-compat-qual-2.5.5.jar
beam-vendor-sdks-java-extensions-protobuf-2.27.0.jar
beam-model-fn-execution-2.27.0.jar
junit-4.13.1.jar
google-api-services-bigquery-v2-rev20200916-1.30.10.jar
netty-tcnative-boringssl-static-2.0.33.Final.jar
resilience4j-core-1.6.1.jar
kamon-prometheus_2.12-2.1.4.jar
netty-common-4.1.51.Final.jar
google-cloud-core-1.93.9.jar
google-http-client-jackson2-1.38.0.jar
grpc-google-cloud-spanner-v1-2.0.2.jar
google-api-client-java6-1.30.10.jar
proto-google-iam-v1-1.0.3.jar
grpc-stub-1.34.0.jar
classgraph-4.8.65.jar
beam-sdks-java-fn-execution-2.27.0.jar
commons-lang3-3.5.jar
gax-1.60.1.jar
proto-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
jsr305-3.0.2.jar
grpc-google-cloud-bigtable-admin-v2-1.14.0.jar
proto-google-cloud-bigquerystorage-v1alpha2-0.105.5.jar
opencensus-api-0.24.0.jar
beam-sdks-java-extensions-google-cloud-platform-core-2.27.0.jar
opencensus-contrib-grpc-util-0.24.0.jar
protobuf-java-util-3.14.0.jar
netty-buffer-4.1.51.Final.jar
google-api-services-pubsub-v1-rev20200713-1.30.10.jar
httpclient-cache-4.5.9.jar
google-api-services-storage-v1-rev20200927-1.30.10.jar
google-auth-library-oauth2-http-0.22.0.jar
proto-google-cloud-bigquerystorage-v1-1.5.5.jar
simpleclient-0.9.0.jar
auto-value-annotations-1.7.4.jar
grpc-google-cloud-pubsublite-v1-0.7.0.jar
metrics-core-3.2.6.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
beam-sdks-java-core-2.27.0.jar
google-cloud-bigquerystorage-1.5.5.jar
error_prone_annotations-2.4.0.jar
xz-1.8.jar
protobuf-java-3.14.0.jar
slf4j-api-1.7.30.jar
grpc-auth-1.34.0.jar
jna-platform-5.5.0.jar
google-api-services-clouddebugger-v2-rev20200501-1.30.10.jar
scala-library-2.12.11.jar
datastore-v1-proto-client-1.6.3.jar
joda-time-2.10.5.jar
grpc-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
okhttp-3.14.7.jar
google-api-services-healthcare-v1beta1-rev20200713-1.30.10.jar
proto-google-common-protos-2.0.1.jar
proto-google-cloud-bigtable-admin-v2-1.14.0.jar
jackson-annotations-2.12.0.jar
conscrypt-openjdk-uber-2.5.1.jar
config-1.4.0.jar
jackson-mapper-asl-1.9.13.jar
commons-codec-1.14.jar
grpc-context-1.34.0.jar
args4j-2.33.jar
commons-compress-1.20.jar
json-20200518.jar
grpc-google-cloud-spanner-admin-instance-v1-2.0.2.jar
animal-sniffer-annotations-1.19.jar
netty-handler-proxy-4.1.51.Final.jar
logback-classic-1.2.3.jar
google-api-client-jackson2-1.30.10.jar
logstash-logback-encoder-6.2.jar
logback-core-1.2.3.jar
bigtable-client-core-1.16.0.jar
threetenbp-1.5.0.jar
snakeyaml-1.27.jar
opencensus-contrib-http-util-0.24.0.jar
google-http-client-1.38.0.jar
google-cloud-pubsub-1.110.0.jar
resilience4j-retry-1.6.1.jar
gcsio-2.1.6.jar
beam-vendor-bytebuddy-1_10_8-0.1.jar
google-cloud-bigtable-1.14.0.jar
proto-google-cloud-datastore-v1-0.88.0.jar
grpc-google-common-protos-1.18.1.jar
jackson-core-asl-1.9.13.jar
util-2.1.6.jar
checker-qual-3.7.0.jar
google-extensions-0.5.1.jar
httpclient-4.5.13.jar
netty-resolver-4.1.51.Final.jar
netty-transport-4.1.51.Final.jar
grpc-google-cloud-spanner-admin-database-v1-2.0.2.jar
proto-google-cloud-bigtable-v2-1.16.1.jar
commons-lang-2.6.jar
gson-2.8.6.jar
google-oauth-client-1.31.2.jar
grpc-protobuf-lite-1.34.0.jar
beam-model-job-management-2.27.0.jar
google-cloud-core-grpc-1.93.9.jar
beam-runners-java-fn-execution-2.27.0.jar
gax-grpc-1.60.1.jar
netty-codec-4.1.51.Final.jar
failureaccess-1.0.1.jar
beam-sdks-java-expansion-service-2.27.0.jar
guava-retrying-2.0.0.jar
snappy-java-1.1.4.jar
grpc-google-cloud-pubsub-v1-1.90.3.jar
beam-sdks-java-io-google-cloud-platform-2.27.0.jar
jna-5.5.0.jar
google-api-client-1.30.10.jar
google-cloud-core-http-1.93.9.jar
hamcrest-2.1.jar
beam-model-pipeline-2.27.0.jar
kamon-core_2.12-2.1.4.jar
grpc-grpclb-1.34.0.jar
avro-1.8.2.jar
flogger-system-backend-0.5.1.jar
flogger-0.5.1.jar
proto-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
beam-runners-direct-java-2.27.0.jar
simpleclient_pushgateway-0.9.0.jar
google-cloud-spanner-2.0.2.jar
google-http-client-protobuf-1.33.0.jar
okio-1.17.2.jar
netty-codec-socks-4.1.51.Final.jar
beam-vendor-guava-26_0-jre-0.1.jar
gax-httpjson-0.76.1.jar
beam-runners-google-cloud-dataflow-java-2.27.0.jar
netty-codec-http-4.1.51.Final.jar
simpleclient_common-0.9.0.jar
grpc-core-1.34.0.jar
google-http-client-appengine-1.36.0.jar
oshi-core-4.2.1.jar
httpcore-4.4.13.jar
javax.annotation-api-1.3.2.jar
google-api-services-iamcredentials-v1-rev20201022-1.30.10.jar
google-oauth-client-java6-1.31.2.jar
guava-30.0-jre.jar
commons-collections-3.2.2.jar
grpc-google-cloud-bigquerystorage-v1-1.5.5.jar
grpc-protobuf-1.34.0.jar
grpc-google-cloud-bigtable-v2-1.14.0.jar
jackson-core-2.12.0.jar
google-cloud-pubsublite-0.7.0.jar
commons-logging-1.2.jar
annotations-4.1.1.4.jar
google-api-services-dataflow-v1b3-rev20200713-1.30.10.jar
proto-google-cloud-pubsublite-v1-0.7.0.jar
vavr-match-0.10.2.jar
j2objc-annotations-1.3.jar
metrics-0.25.0.jar
proto-google-cloud-pubsub-v1-1.92.0.jar
            Reporter: DMYTRO TRUNYKOV
            Assignee: Aizhamal Nurmamat kyzy


Hi ALL.

It seems there is some regression error in the last Beam version – 2.27.0.

There is a streaming dataflow that is runned in Google Cloud.
This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0.
But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below:

{noformat}
Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149) Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175) java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) org.conscrypt.ConscryptEngineSocket$SSLInputStream.<init>(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197) org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) {noformat}
The error is rare. 99% of a time the dataflow works without this (actually any) error.

This issue blocks us to make upgrade of the Beam library for the dataflow.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)