You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/03/19 17:19:00 UTC
[jira] [Commented] (BEAM-11825) "OutOfMemoryError: Direct buffer
memory" in streaming dataflow at Google Cloud
[ https://issues.apache.org/jira/browse/BEAM-11825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17305049#comment-17305049 ]
Beam JIRA Bot commented on BEAM-11825:
--------------------------------------
This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned.
> "OutOfMemoryError: Direct buffer memory" in streaming dataflow at Google Cloud
> ------------------------------------------------------------------------------
>
> Key: BEAM-11825
> URL: https://issues.apache.org/jira/browse/BEAM-11825
> Project: Beam
> Issue Type: Bug
> Components: beam-community, beam-model
> Affects Versions: 2.27.0
> Environment: Google Cloud.
> A list of linked to the dataflow libraries:
> proto-google-cloud-spanner-v1-2.0.2.jar
> grpc-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
> commons-io-2.6.jar
> grpc-alts-1.34.0.jar
> beam-runners-core-construction-java-2.27.0.jar
> beam-vendor-grpc-1_26_0-0.3.jar
> proto-google-cloud-spanner-admin-instance-v1-2.0.2.jar
> vavr-0.10.2.jar
> jackson-dataformat-yaml-2.12.0.jar
> grpc-api-1.34.0.jar
> jaxb-api-2.3.0.jar
> hamcrest-core-2.1.jar
> grpc-netty-1.32.2.jar
> kamon-bundle_2.12-2.1.4.jar
> beam-sdks-java-extensions-protobuf-2.27.0.jar
> grpc-netty-shaded-1.34.0.jar
> jackson-databind-2.12.0.jar
> api-common-1.10.1.jar
> paranamer-2.7.jar
> google-cloud-bigquery-1.122.2.jar
> google-auth-library-credentials-0.22.0.jar
> log4j-api-2.6.2.jar
> google-api-services-cloudresourcemanager-v1-rev20200720-1.30.10.jar
> proto-google-cloud-spanner-admin-database-v1-2.0.2.jar
> netty-codec-http2-4.1.51.Final.jar
> perfmark-api-0.19.0.jar
> netty-handler-4.1.51.Final.jar
> checker-compat-qual-2.5.5.jar
> beam-vendor-sdks-java-extensions-protobuf-2.27.0.jar
> beam-model-fn-execution-2.27.0.jar
> junit-4.13.1.jar
> google-api-services-bigquery-v2-rev20200916-1.30.10.jar
> netty-tcnative-boringssl-static-2.0.33.Final.jar
> resilience4j-core-1.6.1.jar
> kamon-prometheus_2.12-2.1.4.jar
> netty-common-4.1.51.Final.jar
> google-cloud-core-1.93.9.jar
> google-http-client-jackson2-1.38.0.jar
> grpc-google-cloud-spanner-v1-2.0.2.jar
> google-api-client-java6-1.30.10.jar
> proto-google-iam-v1-1.0.3.jar
> grpc-stub-1.34.0.jar
> classgraph-4.8.65.jar
> beam-sdks-java-fn-execution-2.27.0.jar
> commons-lang3-3.5.jar
> gax-1.60.1.jar
> proto-google-cloud-bigquerystorage-v1beta1-0.105.5.jar
> jsr305-3.0.2.jar
> grpc-google-cloud-bigtable-admin-v2-1.14.0.jar
> proto-google-cloud-bigquerystorage-v1alpha2-0.105.5.jar
> opencensus-api-0.24.0.jar
> beam-sdks-java-extensions-google-cloud-platform-core-2.27.0.jar
> opencensus-contrib-grpc-util-0.24.0.jar
> protobuf-java-util-3.14.0.jar
> netty-buffer-4.1.51.Final.jar
> google-api-services-pubsub-v1-rev20200713-1.30.10.jar
> httpclient-cache-4.5.9.jar
> google-api-services-storage-v1-rev20200927-1.30.10.jar
> google-auth-library-oauth2-http-0.22.0.jar
> proto-google-cloud-bigquerystorage-v1-1.5.5.jar
> simpleclient-0.9.0.jar
> auto-value-annotations-1.7.4.jar
> grpc-google-cloud-pubsublite-v1-0.7.0.jar
> metrics-core-3.2.6.jar
> listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> beam-sdks-java-core-2.27.0.jar
> google-cloud-bigquerystorage-1.5.5.jar
> error_prone_annotations-2.4.0.jar
> xz-1.8.jar
> protobuf-java-3.14.0.jar
> slf4j-api-1.7.30.jar
> grpc-auth-1.34.0.jar
> jna-platform-5.5.0.jar
> google-api-services-clouddebugger-v2-rev20200501-1.30.10.jar
> scala-library-2.12.11.jar
> datastore-v1-proto-client-1.6.3.jar
> joda-time-2.10.5.jar
> grpc-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
> okhttp-3.14.7.jar
> google-api-services-healthcare-v1beta1-rev20200713-1.30.10.jar
> proto-google-common-protos-2.0.1.jar
> proto-google-cloud-bigtable-admin-v2-1.14.0.jar
> jackson-annotations-2.12.0.jar
> conscrypt-openjdk-uber-2.5.1.jar
> config-1.4.0.jar
> jackson-mapper-asl-1.9.13.jar
> commons-codec-1.14.jar
> grpc-context-1.34.0.jar
> args4j-2.33.jar
> commons-compress-1.20.jar
> json-20200518.jar
> grpc-google-cloud-spanner-admin-instance-v1-2.0.2.jar
> animal-sniffer-annotations-1.19.jar
> netty-handler-proxy-4.1.51.Final.jar
> logback-classic-1.2.3.jar
> google-api-client-jackson2-1.30.10.jar
> logstash-logback-encoder-6.2.jar
> logback-core-1.2.3.jar
> bigtable-client-core-1.16.0.jar
> threetenbp-1.5.0.jar
> snakeyaml-1.27.jar
> opencensus-contrib-http-util-0.24.0.jar
> google-http-client-1.38.0.jar
> google-cloud-pubsub-1.110.0.jar
> resilience4j-retry-1.6.1.jar
> gcsio-2.1.6.jar
> beam-vendor-bytebuddy-1_10_8-0.1.jar
> google-cloud-bigtable-1.14.0.jar
> proto-google-cloud-datastore-v1-0.88.0.jar
> grpc-google-common-protos-1.18.1.jar
> jackson-core-asl-1.9.13.jar
> util-2.1.6.jar
> checker-qual-3.7.0.jar
> google-extensions-0.5.1.jar
> httpclient-4.5.13.jar
> netty-resolver-4.1.51.Final.jar
> netty-transport-4.1.51.Final.jar
> grpc-google-cloud-spanner-admin-database-v1-2.0.2.jar
> proto-google-cloud-bigtable-v2-1.16.1.jar
> commons-lang-2.6.jar
> gson-2.8.6.jar
> google-oauth-client-1.31.2.jar
> grpc-protobuf-lite-1.34.0.jar
> beam-model-job-management-2.27.0.jar
> google-cloud-core-grpc-1.93.9.jar
> beam-runners-java-fn-execution-2.27.0.jar
> gax-grpc-1.60.1.jar
> netty-codec-4.1.51.Final.jar
> failureaccess-1.0.1.jar
> beam-sdks-java-expansion-service-2.27.0.jar
> guava-retrying-2.0.0.jar
> snappy-java-1.1.4.jar
> grpc-google-cloud-pubsub-v1-1.90.3.jar
> beam-sdks-java-io-google-cloud-platform-2.27.0.jar
> jna-5.5.0.jar
> google-api-client-1.30.10.jar
> google-cloud-core-http-1.93.9.jar
> hamcrest-2.1.jar
> beam-model-pipeline-2.27.0.jar
> kamon-core_2.12-2.1.4.jar
> grpc-grpclb-1.34.0.jar
> avro-1.8.2.jar
> flogger-system-backend-0.5.1.jar
> flogger-0.5.1.jar
> proto-google-cloud-bigquerystorage-v1beta2-0.105.5.jar
> beam-runners-direct-java-2.27.0.jar
> simpleclient_pushgateway-0.9.0.jar
> google-cloud-spanner-2.0.2.jar
> google-http-client-protobuf-1.33.0.jar
> okio-1.17.2.jar
> netty-codec-socks-4.1.51.Final.jar
> beam-vendor-guava-26_0-jre-0.1.jar
> gax-httpjson-0.76.1.jar
> beam-runners-google-cloud-dataflow-java-2.27.0.jar
> netty-codec-http-4.1.51.Final.jar
> simpleclient_common-0.9.0.jar
> grpc-core-1.34.0.jar
> google-http-client-appengine-1.36.0.jar
> oshi-core-4.2.1.jar
> httpcore-4.4.13.jar
> javax.annotation-api-1.3.2.jar
> google-api-services-iamcredentials-v1-rev20201022-1.30.10.jar
> google-oauth-client-java6-1.31.2.jar
> guava-30.0-jre.jar
> commons-collections-3.2.2.jar
> grpc-google-cloud-bigquerystorage-v1-1.5.5.jar
> grpc-protobuf-1.34.0.jar
> grpc-google-cloud-bigtable-v2-1.14.0.jar
> jackson-core-2.12.0.jar
> google-cloud-pubsublite-0.7.0.jar
> commons-logging-1.2.jar
> annotations-4.1.1.4.jar
> google-api-services-dataflow-v1b3-rev20200713-1.30.10.jar
> proto-google-cloud-pubsublite-v1-0.7.0.jar
> vavr-match-0.10.2.jar
> j2objc-annotations-1.3.jar
> metrics-0.25.0.jar
> proto-google-cloud-pubsub-v1-1.92.0.jar
> Reporter: DMYTRO TRUNYKOV
> Assignee: Aizhamal Nurmamat kyzy
> Priority: P3
> Labels: OOM, stale-assigned
>
> Hi ALL.
> It seems there is some regression error in the last Beam version – 2.27.0.
> There is a streaming dataflow that is runned in Google Cloud.
> This dataflow is stable and has no any issues when built against Apache Beam version 2.23.0.
> But if we build this dataflow against Apache Beam version 2.27.0 we catch an exception like below:
> {noformat}
> Error message from worker: java.lang.RuntimeException: java.lang.OutOfMemoryError: Direct buffer memory org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149) Caused by: java.lang.OutOfMemoryError: Direct buffer memory java.base/java.nio.Bits.reserveMemory(Bits.java:175) java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) org.conscrypt.ConscryptEngineSocket$SSLInputStream.<init>(ConscryptEngineSocket.java:722) org.conscrypt.ConscryptEngineSocket.startHandshake(ConscryptEngineSocket.java:197) org.conscrypt.ConscryptEngineSocket.waitForHandshake(ConscryptEngineSocket.java:547) org.conscrypt.ConscryptEngineSocket.getOutputStream(ConscryptEngineSocket.java:290) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:476) java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265) java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187) java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081) java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1367) java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1342) java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:113) com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:541) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:474) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304) java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:834) {noformat}
> The error is rare. 99% of a time the dataflow works without this (actually any) error.
> This issue blocks us to make upgrade of the Beam library for the dataflow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)