You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "aprochko (via GitHub)" <gi...@apache.org> on 2026/07/16 16:59:50 UTC
[I] [Bug]: job launch problem 2.75.0 [beam]
aprochko opened a new issue, #39353:
URL: https://github.com/apache/beam/issues/39353
### What happened?
Hi,
we have several dataflow jobs which are stucked after the start after updating to 2.75.0
The job is able to start, but is not able to process the records after specific step (it might be "Flatten.pCollections()", but not sure)
There are no logs, only increased amount of messages likes
INFO 2026-07-16T16:35:59.254244700Z [213.238s][info][gc,phases ] GC(120) Post Compact 2.382ms
INFO 2026-07-16T16:35:59.254234010Z [213.235s][info][gc,phases,start] GC(120) Post Compact
INFO 2026-07-16T16:35:59.254230210Z [213.235s][info][gc,phases ] GC(120) Compaction Phase 248.772ms
INFO 2026-07-16T16:35:59.254226800Z [212.986s][info][gc,phases,start] GC(120) Compaction Phase
INFO 2026-07-16T16:35:59.254223760Z [212.986s][info][gc,phases ] GC(120) Adjust Pointers 518.781ms
INFO 2026-07-16T16:35:59.254220850Z [212.467s][info][gc,phases,start] GC(120) Adjust Pointers
INFO 2026-07-16T16:35:59.254216840Z [212.467s][info][gc,phases ] GC(120) Forward 71.019ms
INFO 2026-07-16T16:35:59.254213710Z [212.396s][info][gc,phases,start] GC(120) Forward
INFO 2026-07-16T16:35:59.254191910Z [212.396s][info][gc,phases ] GC(120) Summary Phase 0.024ms
INFO 2026-07-16T16:35:59.254188350Z [212.396s][info][gc,phases,start] GC(120) Summary Phase
INFO 2026-07-16T16:35:59.254183520Z [212.396s][info][gc,phases ] GC(120) Marking Phase 945.528ms
INFO 2026-07-16T16:35:59.254179840Z [211.451s][info][gc,phases,start] GC(120) Marking Phase
INFO 2026-07-16T16:35:59.254177050Z [211.451s][info][gc,start ] GC(120) Pause Full (Allocation Failure)
INFO 2026-07-16T16:35:59.254174130Z [211.231s][info][gc,cpu ] GC(119) User=3.41s Sys=0.01s Real=1.75s
INFO 2026-07-16T16:35:59.254171130Z [211.231s][info][gc ] GC(119) Pause Full (Allocation Failure) 1777M->1512M(2121M) 1750.276ms
INFO 2026-07-16T16:35:59.254168630Z [211.231s][info][gc,metaspace ] GC(119) Metaspace: 108557K(109504K)->108557K(109504K) NonClass: 98747K(99200K)->98747K(99200K) Class: 9810K(10304K)->9810K(10304K)
INFO 2026-07-16T16:35:59.254165700Z [211.231s][info][gc,heap ] GC(119) ParOldGen: 1549115K(1629184K)->1549047K(1629184K)
INFO 2026-07-16T16:35:59.254162980Z [211.231s][info][gc,heap ] GC(119) PSYoungGen: 271360K(542720K)->0K(542720K) Eden: 271360K(271360K)->0K(271360K) From: 0K(271360K)->0K(271360K)
INFO 2026-07-16T16:35:59.254160200Z [211.231s][info][gc,phases ] GC(119) Post Compact 2.306ms
INFO 2026-07-16T16:35:59.254156890Z [211.228s][info][gc,phases,start] GC(119) Post Compact
INFO 2026-07-16T16:35:59.254153680Z [211.228s][info][gc,phases ] GC(119) Compaction Phase 236.062ms
INFO 2026-07-16T16:35:59.254147580Z [210.992s][info][gc,phases,start] GC(119) Compaction Phase
INFO 2026-07-16T16:35:59.251831950Z [210.992s][info][gc,phases ] GC(119) Adjust Pointers 519.856ms
INFO 2026-07-16T16:35:59.251829300Z [210.472s][info][gc,phases,start] GC(119) Adjust Pointers
INFO 2026-07-16T16:35:59.251826350Z [210.472s][info][gc,phases ] GC(119) Forward 63.422ms
Does anyone else have such an issues? We have them in 3 / 10 jobs.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: job launch problem 2.75.0 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #39353:
URL: https://github.com/apache/beam/issues/39353#issuecomment-5036437761
also which Beam version were you upgrade from? If it's Dataflow runner please open a customer support ticket so engineers can inspect the job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: job launch problem 2.75.0 [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #39353:
URL: https://github.com/apache/beam/issues/39353#issuecomment-5035775529
Could you provide more information of what your pipeline looks like?
What is the source and sink? What kind of steps/transforms you use in the middle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: job launch problem 2.75.0 [beam]
Posted by "aprochko (via GitHub)" <gi...@apache.org>.
aprochko commented on issue #39353:
URL: https://github.com/apache/beam/issues/39353#issuecomment-5044549402
Hi @shunping @Abacn
thank you for the response
1) From 2.74.0
2) Typical pipeline
KafkaIO -> some aggregation functions with state/timers and BigTable/Bigquery clients (not IO-ed) -> Flatten Pcollections -> Aggregation with state/timers -> BigTableIO + io.requestresponse
I will create support ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: job launch problem 2.75.0 [beam]
Posted by "aprochko (via GitHub)" <gi...@apache.org>.
aprochko commented on issue #39353:
URL: https://github.com/apache/beam/issues/39353#issuecomment-5668994596
## Follow-up: traced this to a Beam POM change between 2.74.0 → 2.75.0
We dug into this a bit more on our side. Our pipeline: KafkaIO → aggregation functions with state/timers, calling BigTable/BigQuery clients directly (not via IO transforms) → Flatten PCollections → aggregation with state/timers → BigTableIO + io.requestresponse.
**What changed between 2.74.0 and 2.75.0**
Comparing `beam-sdks-java-io-google-cloud-platform` POMs on Maven Central:
In 2.74.0, the `google-cloud-bigtable` dependency is explicitly pinned:
```xml
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-bigtable</artifactId>
<version>2.73.1</version>
...
```
In 2.75.0, the same dependency has **no explicit version** anymore. It now resolves transitively through:
```
libraries-bom:26.83.0 → google-cloud-bom:0.264.0 → gapic-libraries-bom:1.86.2 → google-cloud-bigtable-bom:2.78.1
```
(confirmed directly in `gapic-libraries-bom-1.86.2.pom`). For Beam 2.76.0, the same chain resolves to bigtable 2.80.0.
**Why this matters**
Starting with `google-cloud-bigtable` 2.76.0, DirectPath is enabled by default (bypasses the Cloud Front End, direct gRPC), and 2.77.0 introduced a new session-based ("VRpc") client stack. The versions Beam now pulls in transitively (2.78.1 for 2.75.0, 2.80.0 for 2.76.0) both sit inside this new stack, and the client's own changelog shows a sequence of concurrency fixes across 2.78.0–2.81.0 (session pool leaks, deadlocks, and in 2.81.0 specifically "Bound SessionPoolImpl lock to prevent pod-wide wedge") — which lines up closely with the "job gets stuck, 100% CPU, no progress" symptom.
**Short stack trace we saw** (appeared once, after ~2 hours of the job running):
```
com.google.api.gax.rpc.InternalException: io.grpc.StatusRuntimeException: INTERNAL:
Abrupt GOAWAY closed sent stream. HTTP/2 error code: INTERNAL_ERROR,
debug data: Cancelling all calls
at com.google.cloud.bigtable.data.v2.stub.BigtableChannelPool...
```
Is there anything that can realistically be improved on Beam's side here — re-pinning explicitly, or at least surfacing this as a known-risky transitive upgrade?
For context on our side, we use both `BigtableIO` (for writes) and our own custom repository classes (for reads). We initially had the read-side client created lazily and never explicitly closed, and suspected that pattern as a contributing factor, so we changed it to create the `BigtableDataClient` explicitly in each DoFn's `@Setup` and close it in `@Teardown`. That didn't resolve it — we're still seeing the same freeze on 2.83.0 with the client properly scoped and closed, which points more toward the client/DirectPath negotiation itself rather than our lifecycle management around it.
We also run in a fairly locked-down private network (restrictive firewall rules), and suspect DirectPath's connection negotiation may not be degrading cleanly in that environment — which could explain why we see a silent freeze instead of a clean connection error. We're not sure how much of this is on us vs. the client/Beam interaction, but I guess it's important to share it there :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: job launch problem 2.75.0 [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #39353:
URL: https://github.com/apache/beam/issues/39353#issuecomment-5035821934
cc'ed @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org