[jira] [Comment Edited] (BEAM-6923) OOM errors in jobServer when using GCS artifactDir

["Ankur Goenka (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933761#comment-16933761 ] 

Ankur Goenka edited comment on BEAM-6923 at 9/19/19 9:06 PM:
-------------------------------------------------------------

Sorry, for getting it to it so late. 

I am trying to reproduce the problem and getting stuck with some other issue with environment when running command in https://jira.apache.org/jira/browse/BEAM-6923?focusedCommentId=16901955&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16901955

 
{code:java}
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Exception in thread "main" java.lang.UnsupportedOperationException: Not yet implemented. at org.apache.beam.runners.reference.JobServicePipelineResult.metrics(JobServicePipelineResult.java:124) at org.apache.beam.sdk.testutils.metrics.MetricsReader.getDistributions(MetricsReader.java:139) at org.apache.beam.sdk.testutils.metrics.MetricsReader.getEndTimeMetric(MetricsReader.java:122) at org.apache.beam.sdk.loadtests.LoadTest.readMetrics(LoadTest.java:124) at org.apache.beam.sdk.loadtests.LoadTest.run(LoadTest.java:102) at org.apache.beam.sdk.loadtests.ParDoLoadTest.run(ParDoLoadTest.java:53) at org.apache.beam.sdk.loadtests.ParDoLoadTest.main(ParDoLoadTest.java:103)
{code}
 


was (Author: angoenka):
Sorry, for getting it to it so late. 

I am trying to reproduce the problem and getting stuck with some other issue with environment when running command in https://jira.apache.org/jira/browse/BEAM-6923?focusedCommentId=16901955&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16901955

 

Will post my finding.

> OOM errors in jobServer when using GCS artifactDir
> --------------------------------------------------
>
>                 Key: BEAM-6923
>                 URL: https://issues.apache.org/jira/browse/BEAM-6923
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-harness
>            Reporter: Lukasz Gajowy
>            Assignee: Ankur Goenka
>            Priority: Major
>         Attachments: Instance counts.png, Paths to GC root.png, Telemetries.png, heapdump size-sorted.png
>
>
> When starting jobServer with artifactDir pointing to a GCS bucket: 
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
> and running a Java portable pipeline with the following, portability related pipeline options: 
> {code:java}
> --runner=PortableRunner --jobEndpoint=localhost:8099 --defaultEnvironmentType=DOCKER --defaultEnvironmentConfig=gcr.io/<my-freshly-built-sdk-harness-image>/java:latest'{code}
>  
> I'm facing a series of OOM errors, like this: 
> {code:java}
> Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: Java heap space
> at com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
> at com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
> at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
> at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
> at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
> at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>  
> This does not happen when I'm using a local filesystem for the artifact staging location. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)