You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Bjorn De Bakker (Jira)" <ji...@apache.org> on 2022/04/13 15:23:00 UTC
[jira] [Comment Edited] (BEAM-14184) DirectStreamObserver does not respect channel isReady
[ https://issues.apache.org/jira/browse/BEAM-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521723#comment-17521723 ]
Bjorn De Bakker edited comment on BEAM-14184 at 4/13/22 3:22 PM:
-----------------------------------------------------------------
Thanks Luke. I just tested it, but unfortunately the same error came up. The container img we're using runs Apache Beam version 2.30.0 (apache/beam_python3.7_sdk:2.30.0), so I'm not sure if that's the problem.
This is how I run the pipeline:
{code:java}
args:
- "-m"
- "src.batch_inference_pipeline.pipeline"
- "--padding_x=0"
- "--padding_y=0"
- "--runner=FlinkRunner"
- "--flink_master=beam-flink-cluster-jobmanager:8081"
- "--flink_submit_uber_jar"
- "--environment_type=EXTERNAL"
- "--environment_config=localhost:50000"
- "--experiments=beam_fn_api_buffered_stream" {code}
Maybe I made a mistake here.
was (Author: JIRAUSER288009):
Thanks Luke. I just tested it, but unfortunately the same error came up. The container img we're using runs Apache Beam version 3.7, so I'm not sure if that's the problem.
This is how I run the pipeline:
{code:java}
args:
- "-m"
- "src.batch_inference_pipeline.pipeline"
- "--padding_x=0"
- "--padding_y=0"
- "--runner=FlinkRunner"
- "--flink_master=beam-flink-cluster-jobmanager:8081"
- "--flink_submit_uber_jar"
- "--environment_type=EXTERNAL"
- "--environment_config=localhost:50000"
- "--experiments=beam_fn_api_buffered_stream" {code}
Maybe I made a mistake here.
> DirectStreamObserver does not respect channel isReady
> -----------------------------------------------------
>
> Key: BEAM-14184
> URL: https://issues.apache.org/jira/browse/BEAM-14184
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
>
> Leads to OOMs like:
> {noformat}
> Output channel stalled for 1023s, outbound thread CHAIN MapPartition (MapPartition at [1]PerformInference) -> FlatMap (FlatMap at ExtractOutput[0]) -> Map (Key Extractor) -> GroupCombine (GroupCombine at GroupCombine: PerformInferenceAndCombineResults_dep_049/GroupPredictionsByImage) -> Map (Key Extractor) (1/1). See: https://issues.apache.org/jira/browse/BEAM-4280 for the history for this issue.
> Feb 18, 2022 11:51:05 AM org.apache.beam.vendor.grpc.v1p36p0.io.grpc.netty.NettyServerTransport notifyTerminated
> INFO: Transport failed
> org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 2097152 byte(s) of direct memory (used: 1205862679, max: 1207959552)
> at org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:754)
> at org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:709)
> at org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:645)
> at org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:621)
> at org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:204)
> at org.apache.beam.vendor.grpc.v1p36p0.io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:188)
> {noformat}
> See more context in https://lists.apache.org/thread/llmxodbmczhn10c98prs8wmd5hy4nvff
--
This message was sent by Atlassian Jira
(v8.20.1#820001)