You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/03/19 17:19:00 UTC
[jira] [Updated] (BEAM-11825) "OutOfMemoryError: Direct buffer
memory" in streaming dataflow at Google Cloud
[ https://issues.apache.org/jira/browse/BEAM-11825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Beam JIRA Bot updated BEAM-11825:
---------------------------------
Labels: OOM stale-assigned (was: OOM)
> "OutOfMemoryError: Direct buffer memory" in streaming dataflow at Google Cloud
> ------------------------------------------------------------------------------
>
> Key: BEAM-11825
> URL: https://issues.apache.org/jira/browse/BEAM-11825
> Project: Beam
> Issue Type: Bug
> Components: beam-community, beam-model
> Affects Versions: 2.27.0
> Environment: Google Cloud.
> A list of linked to the dataflow libraries:
> proto-google-cloud-spanner-v1-2.0.2.jar
> grpc-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
> commons-io-2.6.jar
> grpc-alts-1.34.0.jar
> beam-runners-core-construction-java-2.27.0.jar
> beam-vendor-grpc-1_26_0-0.3.jar
> proto-google-cloud-spanner-admin-instance-v1-2.0.2.jar
> vavr-0.10.2.jar
> jackson-dataformat-yaml-2.12.0.jar
> grpc-api-1.34.0.jar
> jaxb-api-2.3.0.jar
> hamcrest-core-2.1.jar
> grpc-netty-1.32.2.jar
> kamon-bundle_2.12-2.1.4.jar
> beam-sdks-java-extensions-protobuf-2.27.0.jar
> grpc-netty-shaded-1.34.0.jar
> jackson-databind-2.12.0.jar
> api-common-1.10.1.jar
> paranamer-2.7.jar
> google-cloud-bigquery-1.122.2.jar
> google-auth-library-credentials-0.22.0.jar
> log4j-api-2.6.2.jar
> google-api-services-cloudresourcemanager-v1-rev20200720-1.30.10.jar
> proto-google-cloud-spanner-admin-database-v1-2.0.2.jar
> netty-codec-http2-4.1.51.Final.jar
> perfmark-api-0.19.0.jar
> netty-handler-4.1.51.Final.jar
> checker-compat-qual-2.5.5.jar
> beam-vendor-sdks-java-extensions-protobuf-2.27.0.jar
> beam-model-fn-execution-2.27.0.jar
> junit-4.13.1.jar
> google-api-services-bigquery-v2-rev20200916-1.30.10.jar
> netty-tcnative-boringssl-static-2.0.33.Final.jar
> resilience4j-core-1.6.1.jar
> kamon-prometheus_2.12-2.1.4.jar
> netty-common-4.1.51.Final.jar
> google-cloud-core-1.93.9.jar
> google-http-client-jackson2-1.38.0.jar
> grpc-google-cloud-spanner-v1-2.0.2.jar
> google-api-client-java6-1.30.10.jar
> proto-google-iam-v1-1.0.3.jar
> grpc-stub-1.34.0.jar
> classgraph-4.8.65.jar
> beam-sdks-java-fn-execution-2.27.0.jar
> commons-lang3-3.5.jar
> gax-1.60.1.jar
> proto-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
> jsr305-3.0.2.jar
> grpc-google-cloud-bigtable-admin-v2-1.14.0.jar
> proto-google-cloud-bigquerystorage-v1alpha2-0.105.5.jar
> opencensus-api-0.24.0.jar
> beam-sdks-java-extensions-google-cloud-platform-core-2.27.0.jar
> opencensus-contrib-grpc-util-0.24.0.jar
> protobuf-java-util-3.14.0.jar
> netty-buffer-4.1.51.Final.jar
> google-api-services-pubsub-v1-rev20200713-1.30.10.jar
> httpclient-cache-4.5.9.jar
> google-api-services-storage-v1-rev20200927-1.30.10.jar
> google-auth-library-oauth2-http-0.22.0.jar
> proto-google-cloud-bigquerystorage-v1-1.5.5.jar
> simpleclient-0.9.0.jar
> auto-value-annotations-1.7.4.jar
> grpc-google-cloud-pubsublite-v1-0.7.0.jar
> metrics-core-3.2.6.jar
> listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> beam-sdks-java-core-2.27.0.jar
> google-cloud-bigquerystorage-1.5.5.jar
> error_prone_annotations-2.4.0.jar
> xz-1.8.jar
> protobuf-java-3.14.0.jar
> slf4j-api-1.7.30.jar
> grpc-auth-1.34.0.jar
> jna-platform-5.5.0.jar
> google-api-services-clouddebugger-v2-rev20200501-1.30.10.jar
> scala-library-2.12.11.jar
> datastore-v1-proto-client-1.6.3.jar
> joda-time-2.10.5.jar
> grpc-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
> okhttp-3.14.7.jar
> google-api-services-healthcare-v1beta1-rev20200713-1.30.10.jar
> proto-google-common-protos-2.0.1.jar
> proto-google-cloud-bigtable-admin-v2-1.14.0.jar
> jackson-annotations-2.12.0.jar
> conscrypt-openjdk-uber-2.5.1.jar
> config-1.4.0.jar
> jackson-mapper-asl-1.9.13.jar
> commons-codec-1.14.jar
> grpc-context-1.34.0.jar
> args4j-2.33.jar
> commons-compress-1.20.jar
> json-20200518.jar
> grpc-google-cloud-spanner-admin-instance-v1-2.0.2.jar
> animal-sniffer-annotations-1.19.jar
> netty-handler-proxy-4.1.51.Final.jar
> logback-classic-1.2.3.jar
> google-api-client-jackson2-1.30.10.jar
> logstash-logback-encoder-6.2.jar
> logback-core-1.2.3.jar
> bigtable-client-core-1.16.0.jar
> threetenbp-1.5.0.jar
> snakeyaml-1.27.jar
> opencensus-contrib-http-util-0.24.0.jar
> google-http-client-1.38.0.jar
> google-cloud-pubsub-1.110.0.jar
> resilience4j-retry-1.6.1.jar
> gcsio-2.1.6.jar
> beam-vendor-bytebuddy-1_10_8-0.1.jar
> google-cloud-bigtable-1.14.0.jar
> proto-google-cloud-datastore-v1-0.88.0.jar
> grpc-google-common-protos-1.18.1.jar
> jackson-core-asl-1.9.13.jar
> util-2.1.6.jar
> checker-qual-3.7.0.jar
> google-extensions-0.5.1.jar
> httpclient-4.5.13.jar
> netty-resolver-4.1.51.Final.jar
> netty-transport-4.1.51.Final.jar
> grpc-google-cloud-spanner-admin-database-v1-2.0.2.jar
> proto-google-cloud-bigtable-v2-1.16.1.jar
> commons-lang-2.6.jar
> gson-2.8.6.jar
> google-oauth-client-1.31.2.jar
> grpc-protobuf-lite-1.34.0.jar
> beam-model-job-management-2.27.0.jar
> google-cloud-core-grpc-1.93.9.jar
> beam-runners-java-fn-execution-2.27.0.jar
> gax-grpc-1.60.1.jar
> netty-codec-4.1.51.Final.jar
> failureaccess-1.0.1.jar
> beam-sdks-java-expansion-service-2.27.0.jar
> guava-retrying-2.0.0.jar
> snappy-java-1.1.4.jar
> grpc-google-cloud-pubsub-v1-1.90.3.jar
> beam-sdks-java-io-google-cloud-platform-2.27.0.jar
> jna-5.5.0.jar
> google-api-client-1.30.10.jar
> google-cloud-core-http-1.93.9.jar
> hamcrest-2.1.jar
> beam-model-pipeline-2.27.0.jar
> kamon-core_2.12-2.1.4.jar
> grpc-grpclb-1.34.0.jar
> avro-1.8.2.jar
> flogger-system-backend-0.5.1.jar
> flogger-0.5.1.jar
> proto-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
> beam-runners-direct-java-2.27.0.jar
> simpleclient_pushgateway-0.9.0.jar
> google-cloud-spanner-2.0.2.jar
> google-http-client-protobuf-1.33.0.jar
> okio-1.17.2.jar
> netty-codec-socks-4.1.51.Final.jar
> beam-vendor-guava-26_0-jre-0.1.jar
> gax-httpjson-0.76.1.jar
> beam-runners-google-cloud-dataflow-java-2.27.0.jar
> netty-codec-http-4.1.51.Final.jar
> simpleclient_common-0.9.0.jar
> grpc-core-1.34.0.jar
> google-http-client-appengine-1.36.0.jar
> oshi-core-4.2.1.jar
> httpcore-4.4.13.jar
> javax.annotation-api-1.3.2.jar
> google-api-services-iamcredentials-v1-rev20201022-1.30.10.jar
> google-oauth-client-java6-1.31.2.jar
> guava-30.0-jre.jar
> commons-collections-3.2.2.jar
> grpc-google-cloud-bigquerystorage-v1-1.5.5.jar
> grpc-protobuf-1.34.0.jar
> grpc-google-cloud-bigtable-v2-1.14.0.jar
> jackson-core-2.12.0.jar
> google-cloud-pubsublite-0.7.0.jar
> commons-logging-1.2.jar
> annotations-4.1.1.4.jar
> google-api-services-dataflow-v1b3-rev20200713-1.30.10.jar
> proto-google-cloud-pubsublite-v1-0.7.0.jar
> vavr-match-0.10.2.jar
> j2objc-annotations-1.3.jar
> metrics-0.25.0.jar
> proto-google-cloud-pubsub-v1-1.92.0.jar
> Reporter: DMYTRO TRUNYKOV
> Assignee: Aizhamal Nurmamat kyzy
> Priority: P3
> Labels: OOM, stale-assigned
>
> Hi ALL.
> It seems there is some regression error in the last Beam version – 2.27.0.
> There is a streaming dataflow that is runned in Google Cloud.
> This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0.
> But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below:
> {noformat}
> Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149) Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175) java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) org.conscrypt.ConscryptEngineSocket$SSLInputStream.<init>(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197) org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) {noformat}
> The error is rare. 99% of a time the dataflow works without this (actually any) error.
> This issue blocks us to make upgrade of the Beam library for the dataflow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)