You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@pekko.apache.org by "He-Pin (via GitHub)" <gi...@apache.org> on 2026/04/14 08:02:06 UTC

[I] Rewritten more to graphstage [pekko]

He-Pin opened a new issue, #2860:
URL: https://github.com/apache/pekko/issues/2860

   I think we should rewrite more stream operators to graphstages, there are some are not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-6482139963

   Current-main recheck for #2860 (Pekko `a55a008fdf`, 2026-10-29). This supersedes parts of the older remaining-work list.
   
   The TLS GraphStage implementation and the default-engine migration are separate milestones. The new engine is implemented and has substantial regression coverage, but `pekko.stream.materializer.tls.engine` still defaults to `legacy-actor`. This review found a reproducible buffer-allocation/growth regression; the focused fix is #3728.
   
   **Already completed**
   
   - #2873 removed the dead internal Unzip implementation.
   - #2874 migrated fanout publisher execution to `FanoutPublisherBridgeStage`; #2910 fixed abrupt-shutdown signaling.
   - #3027 removed the dead `ActorProcessor`, `ActorProcessorImpl`, and `ExposedPublisherReceive` types. `ActorProcessor.scala` still contains buffering helpers used by legacy TLS.
   - #3176 removed `FanoutPublisherSink`. Current `Sink.asPublisher(true)` directly composes `Flow[T].async` with the GraphStage bridge. This is no longer a future rewrite.
   - #2878 added the opt-in TLS GraphStage. #3324 subsequently fixed deferred-output handling on inbound close, and #3323 fixed transport abort on session-verifier rejection.
   
   **Review against the original TLSActor state machine**
   
   The comparison uses the actual `TransferState` algebra, not just matching phase names: `A || B` completes when both complete, while `A && B` completes when either completes ([Transfer.scala:114](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/impl/Transfer.scala#L114)).
   
   | Original phase | GraphStage counterpart | Contract to preserve |
   | --- | --- | --- |
   | `bidirectional` | `BidirectionalPhase` | Inbound first, then outbound; engine NEED_WRAP can progress without application data. |
   | `flushingOutbound` | `FlushingOutboundPhase` | Flush protocol output subject to transport demand; handle SSLException without stranding shutdown. |
   | `awaitingClose` | `AwaitingClosePhase` | Continue consuming peer close records, discarding application output. |
   | `outboundClosed` | `OutboundClosedPhase` | Continue inbound processing and required handshake/close writes. |
   | `inboundClosed` | `InboundClosedPhase` | Preserve ignore-cancel behavior while processing the remaining outbound side. |
   | `completedPhase` / `pumpFinished` | `CompletedPhase` / `StageCompletionPendingFlag` | Cancel inputs, complete outputs, and finish already-pending output before stopping. |
   
   The original states are at [TLSActor.scala:233](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/impl/io/TLSActor.scala#L233), and the explicit port predicates are at [TlsGraphStage.scala:186](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/impl/io/TlsGraphStage.scala#L186). The main new proof obligations are batching, delayed flushes, output slots, and pooled buffers. #3324 is an example of why copying the old phases alone was insufficient.
   
   I also checked an early cipher-output cancellation hypothesis with both implementations. With two `Source.maybe` inputs, a server engine waiting for input, `Sink.cancelled` on cipher output, and `Sink.ignore` on plaintext output, the legacy and GraphStage paths behaved alike in all four async/no-async combinations. That experiment does not establish a migration regression, so it is not being presented as one or used to justify a redundant spec.
   
   Explicitly selecting GraphStage also exposed a closing-policy assumption in the existing public `TcpSpec` TLS example: its server uses `takeWhile` and expects `hello`, but default `IgnoreComplete` permits the resulting plaintext-output cancellation to discard the last pending byte (`hell` was observed on both unmodified main and the buffer-fix branch). This is allowed by the documented TLS closing contract, not evidence that the state machine must drain arbitrary plaintext on eager cancellation. [SslTlsOptions.scala:85](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/SslTlsOptions.scala#L85) describes the server use case for `IgnoreCancel`, and the shared `CancellingRHS` scenario already uses it. #3727 corrects that test's server policy while retaining its full-response and termination assertions; production close semantics stay unchanged.
   
   **Confirmed gap: TLS allocation depends on the TCP pool size**
   
   [TlsGraphStage.scala:144](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/impl/io/TlsGraphStage.scala#L144) acquires three transport buffers from the TCP pool. [acquireTransportBuffer:682](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/impl/io/TlsGraphStage.scala#L682) throws if a fixed-size pooled buffer is smaller than `SSLSession.getPacketBufferSize`. Legacy TLS allocates its own buffers independently of TCP.
   
   On JDK 21 with `-Dpekko.io.tcp.direct-buffer-size=8192`:
   
   - `TlsSpec`: **113 passed**.
   - `TlsGraphStageSpec`, filtered to `SingleBytes`: **8 failed**, covering TLS 1.2/1.3 and in-memory/TCP communication patterns.
   - All eight fail at startup: `TLS packet buffer requires 16709 bytes but pekko.io.tcp.direct-buffer-size is 8192`.
   
   The differential run used the compiled ScalaTest classes. An equivalent focused sbt reproduction is:
   
   ```sh
   rtk sbt -Dpekko.io.tcp.direct-buffer-size=8192 \
     "stream-tests / Test / testOnly org.apache.pekko.stream.io.TlsSpec -- -z SingleBytes" \
     "stream-tests / Test / testOnly org.apache.pekko.stream.io.TlsGraphStageSpec -- -z SingleBytes"
   ```
   
   There is also a confirmed runtime failure even when the configured buffer fits the initial packet size. [growTransportOutBuffer:578](https://github.com/apache/pekko/blob/a55a008fdf/stream/src/main/scala/org/apache/pekko/stream/impl/io/TlsGraphStage.scala#L578) asks the same fixed-size pool for twice the current capacity, which cannot succeed.
   
   A second differential probe used real JDK 21 TLS 1.2 engines, completed the handshake, and round-tripped an initial 2,048-byte payload with a 17,000-byte pool. Holding the batching timer then allowed a 1,024-byte wrap to retain 1,053 ciphertext bytes. The next 2,048-byte write encountered only 15,947 free bytes, below the provider's 16,709-byte requirement, so `wrap` returned `BUFFER_OVERFLOW`. GraphStage attempted to obtain 34,000 bytes from the 17,000-byte pool and failed; legacy TLS delivered all 5,120 application bytes. The wrapper only observed real engine results; it did not fabricate the overflow.
   
   The focused fix preserves pooled direct buffers when they fit and allocates larger heap buffers when needed, including output growth. Heap buffers are never returned to the fixed-size TCP pool. Three necessary directional cases cover TLS 1.2/1.3 with an 8 KiB pool, pool ownership after stage termination, and overflow while earlier ciphertext remains buffered. All three failed before the fix; the existing deferred-close cases continued to pass. The deterministic overflow case reuses the existing scripted engine/scheduler rather than adding a broad duplicate suite.
   
   The default pool uses 128 KiB buffers, so the three held buffers cost approximately **384 KiB of direct memory per live TLS stage**, before application buffers. #3019 proposed heap/on-demand buffers but was closed without merging. That work remains relevant; measure native memory as well as heap allocation and throughput.
   
   **Downstream status**
   
   Current pekko-http main (`127528261`) has no source references to `TLSActor`, `TlsModule`, `TlsUtils`, or `TlsGraphStage`; HTTP and HTTP/2 use the public `TLS(...)` API. The earlier direct-dependency warning should not be repeated as an unverified current blocker.
   
   pekko-http already runs both TLS engines across JDK 17/21/25 and Scala 2.13/3. All **12 jobs** in the [2026-10-28 nightly](https://github.com/apache/pekko-http/actions/runs/40615900881) succeeded. The JDK 21/Scala 2.13/graph-stage log used Pekko `2.0.0-M4+190-2c035294-SNAPSHOT`: this is existing integration evidence, not a downstream rerun against today's exact HEAD. Published downstream versions and other consumers still need checking before class removal.
   
   **Suggested 2.0.0 scope**
   
   | Priority | Work | Boundary / acceptance criteria |
   | --- | --- | --- |
   | P0 | TLS buffer compatibility and targeted lifecycle coverage, then a separate default-switch PR | Preserve both engines' contracts; cross-JDK TLS, HTTP/HTTP2 and Artery validation; native-memory and latency evidence. Keep `legacy-actor` as a transition fallback. |
   | P1 | Small `Sink.cancelled` / `CancelSink` GraphStage conversion | Preserve cancellation without requesting an element, subscription ordering, `NotUsed`, and attributes. Run `CancelledSinkSubscriberTest` plus focused sink/flow tests. Verify materialization benefit rather than assuming it. |
   | P1 | Remove unused `FanIn` and `FanOut` actor classes, subject to dependency/MiMa checks | Only the unused classes. Their companion objects still supply `InputBunch` / `OutputBunch` and protocol messages used by TLSActor; do not delete the whole files while fallback TLS remains. |
   | P2 | Separate investigation of `Flow.fromProcessor(Mat)` / `ProcessorModulePhase` | Preserve one factory call per materialization, the original materialized value, Reactive Streams sequencing/cancellation, and avoid unnecessary additional async boundaries. Prototype and benchmark first. |
   | Later | `PublisherSource`, `SubscriberSource`, `SubscriberSink`, and single-subscriber `PublisherSink` | These are Reactive Streams boundary adapters, not ordinary actor operators. `PublisherSink` deliberately uses `VirtualPublisher` to delay subscription. A GraphStage wrapper alone may add overhead or alter contracts. |
   
   Useful TLS follow-ups before changing the default:
   
   - Extend batching, fragmented-record, slow-consumer and timer/cancellation edge cases to TLS 1.3; the dedicated edge-case fixture currently fixes TLS 1.2, although the shared suite covers both protocols.
   - Keep Artery TLS coverage with GraphStage explicitly selected in the cutover gate; the six cases run here passed on JDK 21, rather than constituting a cross-JDK Artery matrix.
   - Retain public-API selection coverage in isolated JVMs/CI after #3727. Selection is currently a classloader-global `val` using `ConfigFactory.load()`, not each materializer's config; the selection unit tests exercise precedence helpers. The local TCP and Artery runs here explicitly selected GraphStage.
   - Improve benchmark scope: [warmRoundTrip:148](https://github.com/apache/pekko/blob/a55a008fdf/bench-jmh/src/main/scala/org/apache/pekko/stream/io/TlsBenchmark.scala#L148) creates a fresh client/server pair for every 1,000-record invocation, despite the class comment saying engines are reused. Add an actual long-lived-connection fixture, TLS 1.3, latency, and live-connection memory measurements. Historical throughput numbers alone do not establish those properties.
   
   After a transition release and downstream validation, retire `TlsModule`, its materializer island/phase, and `TLSActor`, then prune the unused buffering/Pump/Transfer chain. First move the still-used `TlsUtils.applySessionParameters` out of `TLSActor.scala`: the GraphStage calls it too.
   
   Do not blanket-delete `ActorSubscriberMessage` or `ActorPublisher.scala`: current GraphStage internals use the former; the latter owns shutdown sentinels used by the GraphInterpreter, fanout bridge, subscriber management, and hubs. Actor-ref sources, stream refs, TCP and file/stream I/O adapters are already GraphStages. Interpreter/supervisor actors are runtime infrastructure, outside this operator migration.
   
   **Validation**
   
   - JDK 21, current Pekko HEAD: `TlsGraphStageDeferredCloseSpec` **3 passed**, then `TlsSpec`, `TlsGraphStageSpec`, `TlsGraphStageEdgeCasesSpec`, `TlsGraphStageIsolatedSpec`, `TlsEngineSelectionSpec`, and `TLSUtilsSpec` **254 passed**: **257 total, 0 failed/ignored/pending** under default buffer configuration.
   - Small-buffer differential run: **113 legacy passes / 8 GraphStage failures**, as described above.
   - Legacy/GraphStage early-cancellation differential experiment: matching behavior; not counted as a migration defect.
   - Fix branch on JDK 21: **260 TLS tests passed**, plus **8/8** small-pool shared communication cases and **6/6** Artery TLS tests with GraphStage explicitly selected.
   - JDK 17 and 25: all **6/6** focused small-buffer/deferred-close cases passed on each JDK.
   - Real 17,000-byte overflow probe after the fix: both engines delivered **5,120 bytes**, including successful GraphStage recovery from the observed overflow.
   - Corrected public TCP TLS case: both engines pass on JDK 21; a separate unchanged-main probe with the corrected server policy passed TLS 1.2/1.3 **10 times each**.
   - Native scalafmt, header generation, cross-Scala header checks and `git diff --check` passed. `sortImports` hit a scalameta linkage error; unrelated edits were removed.
   - Full local PR validation/MiMa are delegated to remote CI as requested; an in-progress local Paradox build was stopped before completion.
   - The focused PRs are initially drafts while final review and remote CI proceed; no default-engine change or legacy-class removal is included.
   
   The fix PR addresses this demonstrated regression only. Switching the default, changing the normal buffer-memory policy, and retiring legacy classes should remain separate changes with their own acceptance evidence.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4275843532

   ## PR-3 Progress Update
   
   PR **#2877** has been opened as a draft: _feat(stream): add GraphStage-backed TLS path behind feature switch_
   
   ### What is in PR-3
   
   A new `TlsGraphStage` (pure `BidiGraphStage` implementation of TLS) is added behind a runtime feature switch:
   - Config: `pekko.stream.materializer.tls.use-legacy-actor = off`
   - JVM flag: `-Dpekko.stream.materializer.tls.use-legacy-actor=false`
   - **Default: legacy `TLSActor` path unchanged**
   
   ### Test status
   | Suite | Status |
   |-------|--------|
   | `TlsSpec` (legacy path) | ✅ 111/111 |
   | `TlsGraphStageSpec` (GraphStage path) | ✅ 107 pass, 4 pending* |
   | `MaterializerStateSpec` | ✅ 6/6 |
   
   *Four "reliably cancel subscriptions" tests are marked `pending` with detailed Scaladoc. Root cause: async boundary ordering — demand (1 inter-actor hop) consistently wins over failure (2 hops) in the TLS actor mailbox. The async boundary itself is intentional (SSLEngine thread-safety + per-connection actor isolation). See class Scaladoc in `TlsGraphStageSpec` for full explanation.
   
   ### PRs so far
   - PR-1: #2873 (remove dead internal `impl.FanOut.Unzip`)
   - PR-2: #2874 (migrate `Sink.asPublisher(fanout=true)` to GraphStage)
   - **PR-3: #2877 (TLS GraphStage path — this PR)**
   
   ### Coming up
   - **PR-4**: TLS cutover and legacy `TLSActor` / `TlsModule` retirement (after PR-3 is reviewed/merged)
   - **PR-5**: Final cleanup of unused `FanIn.scala` / `FanOut.scala` substrate
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "pjfanning (via GitHub)" <gi...@apache.org>.
pjfanning commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4276169128

   TLSActor is used in pekko-http. So we need to retain that class in its current form for a period of a few weeks. So we could add its replacement but keep the old code for a few weeks to keep our builds working. We can update pekko-http but let's not break it and force a panicked update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4274476482

   Based on the current `apache/pekko` codebase, I re-validated the scope of #2860 and would propose splitting the work into a sequence of small PRs.
   
   ## Scope correction
   
   `Unzip` is no longer a public GraphStage migration target: the public DSL `scaladsl.Unzip` is already GraphStage-based. What remains is an internal legacy `impl.FanOut.Unzip`, which appears unused and is a good first cleanup candidate.
   
   ## Proposed PR order
   
   1. **PR-1: remove internal legacy `impl.FanOut.Unzip`**
      - small, low-risk cleanup
      - if needed, add directional regression coverage around the current public `Unzip` behavior
   2. **PR-2: migrate the `Sink.asPublisher(fanout = true)` path**
      - covers `FanoutPublisherSink` / `FanoutProcessorImpl` / required replacement plumbing
      - behavior-sensitive and TCK-sensitive, so this should stay separate from PR-1
   3. **PR-3: add a GraphStage-backed TLS path behind a safe switch**
      - keep the existing TLS actor/island path as fallback initially
   4. **PR-4: TLS cutover and retirement of old TLS actor/island infrastructure**
   5. **PR-5: final cleanup of now-unused legacy substrate**
      - only after grep + compile + MiMa prove there are no remaining users
   
   ## Notes on dependencies
   
   - Each change should be a **separate PR**.
   - If a PR depends on an earlier one, the PR description should explicitly list the required **merge order**.
   - TLS should not be the first migration PR; it is the highest-risk area because of custom materialization/island behavior, SSLEngine state handling, and compatibility concerns.
   
   ## Validation focus
   
   - PR-1: `GraphUnzipSpec`, stream compile, MiMa
   - PR-2: `FanoutProcessorSpec`, `TimeoutsSpec`, fanout-related TCK, MiMa
   - PR-3 / PR-4: `TlsSpec`, `TcpSpec`, `MaterializerStateSpec`, remote `TlsTcpSpec`, relevant artery TLS coverage, MiMa
   
   I have not started code changes yet in this thread; this comment is only to align the execution plan before taking the first slice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4321398733

   ## Updated roadmap after the current GraphStage work
   
   I re-checked the current state of #2860 and the existing comments. The remaining work should stay narrow and staged.
   
   ### Current state
   
   - **PR #2873** is merged: removed the dead internal `impl.FanOut.Unzip`.
   - **PR #2874** is merged: `Sink.asPublisher(fanout = true)` now routes through the GraphStage bridge.
   - **PR #2877** is superseded by **PR #2878**.
   - **PR #2878** is the current TLS GraphStage PR:
     - keeps the legacy actor path as the default;
     - adds the GraphStage TLS engine as an opt-in path;
     - keeps the TLSActor-compatible state machine semantics;
     - adds TLSv1.3 coverage for both paths;
     - includes JMH numbers and cross-JDK validation notes.
   
   ### Recommended next steps
   
   1. **Finish PR #2878 first**
      - Keep the scope stable.
      - Only address CI, review feedback, documentation wording, and correctness issues.
      - Do not mix this PR with deletion of the old TLS infrastructure.
   
   2. **Do not remove `TLSActor` immediately**
      - `TLSActor` is still used by pekko-http, as noted above.
      - We should keep the class and old route for a short transition period so downstream builds do not break unexpectedly.
   
   3. **Update pekko-http next**
      - Find and remove the direct `TLSActor` dependency in pekko-http.
      - Move it to public/stable TLS APIs or to the new GraphStage route where appropriate.
      - This is the real prerequisite for deleting `TLSActor` from Pekko Streams.
   
   4. **Follow-up PR: switch the default**
      - After #2878 is merged and pekko-http is no longer blocked, open a separate PR to make `graph-stage` the default TLS engine.
      - Keep `legacy-actor` as a fallback for at least one transition release.
      - Include cross-JDK TLS regression results and JMH data in that PR.
   
   5. **Final cleanup PR**
      - Once the default has been switched and downstream usage has moved away:
        - remove `TlsModule`;
        - remove `TlsModuleIslandTag` / `TlsModulePhase`;
        - remove `TLSActor`;
        - then remove now-unused `FanIn.scala`, `FanOut.scala`, and the old `Pump` / `TransferState` / `TransferPhase` substrate if grep, compile, and MiMa agree.
      - `ActorSubscriberMessage` should not be removed unless separately proven unused, because current GraphStage internals still use it.
   
   ### Low-priority / separate investigation
   
   `SourceModule`, `SinkModule`, and `ProcessorModulePhase` are Reactive Streams boundary plumbing rather than normal operators. They can be investigated later, but they should not be bundled with the TLS migration. The risk/benefit ratio is much worse than the already completed fanout and TLS work.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4792956839

   Review update after re-checking current `main` and the existing issue comments.
   
   I would refine the remaining roadmap as follows:
   
   1. Treat `Sink.asPublisher(fanout = true)` as a small follow-up investigation, not as a redo of the already merged runtime migration.
      - #2874 moved the fanout publisher runtime to `FanoutPublisherBridgeStage`.
      - Current `Sink.asPublisher(true)` still constructs `FanoutPublisherSink`, which is a `SinkModule` wrapper.
      - `FanoutPublisherSink.create` then materializes `Source.asSubscriber.toMat(Sink.fromGraph(new FanoutPublisherBridgeStage))` internally.
      - So the runtime fanout behavior is GraphStage-backed, but the construction/materialization path still goes through the old `SinkModule` plus `Source.asSubscriber` boundary.
   
   2. A focused PR could explore making `Sink.asPublisher(fanout = true)` return the bridge stage directly.
      - Goal: preserve the existing materialized `Publisher`, fanout semantics, input-buffer behavior, subscription timeout behavior, active/late subscriber behavior, and abrupt-shutdown signaling.
      - Possible cleanup if successful: remove `FanoutPublisherSink` if it has no remaining users.
      - Keep `Sink.asPublisher(fanout = false)` out of scope, since that still has different single-subscriber `VirtualPublisher` behavior.
   
   3. Keep TLS cleanup separate.
      - #2878 has added the TLS GraphStage engine, but the legacy actor path is still the default in config.
      - `TLSActor` should stay until downstream direct users such as pekko-http are migrated.
      - After downstream usage is gone, switch the default to `graph-stage` with a transition fallback, then remove `TlsModule`, `TlsModuleIslandTag`, `TlsModulePhase`, `TLSActor`, and only then prune now-unused `FanIn` / `FanOut` / `Transfer` substrate if grep, compile, tests, and MiMa agree.
   
   4. Leave `SourceModule`, `SinkModule`, and `ProcessorModulePhase` as a later Reactive Streams boundary cleanup track.
      - They are boundary plumbing, not normal operators.
      - They should not be bundled with TLS deletion.
      - The `fanout = true` wrapper exploration above is a good narrow first step toward shrinking that area.
   
   Suggested validation for the `fanout = true` follow-up:
   
   - `stream / Test / testOnly org.apache.pekko.stream.impl.FanoutPublisherBehaviorSpec`
   - `stream / Test / testOnly org.apache.pekko.stream.scaladsl.PublisherSinkSpec`
   - `stream / Test / testOnly org.apache.pekko.stream.scaladsl.SinkSpec`
   - `stream / Test / testOnly org.apache.pekko.stream.scaladsl.FlowSpec`
   - `stream / Test / testOnly org.apache.pekko.stream.impl.TimeoutsSpec`
   - relevant `stream-tests-tck` fanout publisher coverage
   - `stream / Test / compile`
   - `stream / mimaReportBinaryIssues`
   
   This keeps the next slice small: first remove the remaining `fanout = true` SinkModule wrapper if behavior stays identical, then continue the larger TLS and Reactive Streams boundary cleanup tracks separately.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-6492009852

   ### Motivation
   This corrects the TcpSpec conclusions in [the comment created on 2026-10-28 at 22:20:27Z](https://github.com/apache/pekko/issues/2860#issuecomment-6482139963), without rewriting that comment.
   
   ### Modification
   - The reproduced `hell`/`hello` failure does not show that TLS had accepted the final `o` before observing plaintext-output cancellation. In every failing diagnostic run, the server-side event order was `accept(h), accept(e), accept(l), accept(l), cancel, accept(o)`.
   - The statement that #3727 "restores the original four-argument test" was incorrect. `TcpSpec.scala`, its four-argument call, closing policy, and `hello` assertion are unchanged in the PR. The referenced `31f6681c00` is superseded by `205155160a`.
   - The current PR description also needs a qualification: only `h/e/l/l` had been accepted **before cancellation**. A late `onPush(o)` was observed afterwards on both revisions; saying that `o` never entered the operator would be inaccurate.
   
   ### Result
   #3727 fixes the separate loss of plaintext already accepted at cancellation; it does not eliminate the original TcpSpec failure. The observed echo failure is a cancellation-versus-delivery race, outside that drain guarantee. These logs establish the event order at TLS, not the exact upstream buffer responsible. The follow-up is #3729.
   
   The drain also applies when plaintext-output cancellation is observed before peer transport-input completion. If transport-input completion is observed first, the existing inbound-close contract discards accepted plaintext, matching the legacy actor engine.
   
   ### Tests
   On 2026-10-29, using OpenJDK 25.0.2, macOS arm64, Scala 2.13.18, and the unchanged TLS 1.3 TcpSpec case:
   
   | Revision | Engine | Uninstrumented runs | Failures |
   | --- | --- | ---: | ---: |
   | `a55a008fdf` (`origin/main`) | legacy-actor | 20 | 0 (0%) |
   | `a55a008fdf` | graph-stage | 20 | 8 (40%) |
   | `205155160a` (#3727) | legacy-actor | 20 | 0 (0%) |
   | `205155160a` | graph-stage | 20 | 13 (65%) |
   
   All failures returned `hell`, expected `hello`. Each repeat ran the original ScalaTest case in a fresh JVM using sbt-exported test classpath and JVM options; `stream-tests/Test/fork = true` was confirmed, and GraphStage was explicitly selected by system property.
   
   A separate diagnostic batch ran 20 times per engine per revision. GraphStage failed 11/20 (55%) on main and 12/20 (60%) on the PR, always with the event order above. Successful GraphStage runs accepted `hello` before cancellation. Both legacy batches passed 20/20 and logged `hello` at receipt and `ChoppingBlock.chopInto`. Diagnostic changes were local only and removed.
   
   These are shared-workstation observations, not comparable failure-probability estimates: the one-minute load ranged from 5.76–6.72 during the main baseline and 19.40–34.44 during the PR baseline; logging can also affect scheduling. Earlier individual passes do not establish that this intermittent case was fixed.
   
   ### References
   Refs #2860, #3727, and #3729. Buffer allocation/growth remains separate in #3728.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4274746256

   Progress update:
   
   Two review-complete draft PR slices are now open for issue #2860:
   
   1. PR #2873 - chore(stream): remove internal FanOut Unzip
      - Removes the dead internal `org.apache.pekko.stream.impl.Unzip` implementation from `FanOut.scala`
      - Adds the matching `2.0.x` MiMa excludes
      - Validation: `stream/test:compile`, `stream/mimaReportBinaryIssues`, `GraphUnzipSpec`
   
   2. PR #2874 - feat(stream): replace fanout publisher runtime with GraphStage bridge
      - Replaces the actor-backed `FanoutProcessorImpl` runtime behind `Sink.asPublisher(fanout = true)` with `FanoutPublisherBridgeStage`
      - Deletes `FanoutProcessor.scala`
      - Replaces the old implementation-bound spec with `FanoutPublisherBehaviorSpec`
      - Adds the matching `2.0.x` MiMa excludes
      - Validation: `stream/test:compile`, `stream/mimaReportBinaryIssues`, `FanoutPublisherBehaviorSpec`, `TimeoutsSpec`, `FlowSpec`, `SinkSpec`, `SinkTest`, `FanoutPublisherTest`
   
   Both slices were independently cross-reviewed locally (code/RS semantics/safety/test adequacy) before opening the draft PRs.
   
   Next up: continue with the TLS scope from #2860. The current plan is to treat TLS as its own focused migration area and keep the performance-sensitive actor/island constraints explicit during the design and review work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4276471086

   I think.we may need keep it in 2.20.0 and remove it in 2.1.0?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org


Re: [I] Rewritten more to graphstage [pekko]

Posted by "He-Pin (via GitHub)" <gi...@apache.org>.
He-Pin commented on issue #2860:
URL: https://github.com/apache/pekko/issues/2860#issuecomment-4274435588

   ## Analysis: Legacy Stream Components That Can Be Refactored to GraphStage
   
   After analyzing the codebase, here is a structured inventory of components still using the legacy `Pump`/`InputBunch`/`OutputBunch`/`TransferPhase` infrastructure that could be refactored to modern `GraphStage` equivalents.
   
   ---
   
   ### Component Analysis Table
   
   | # | Component | File | Current Pattern | Legacy Dependencies | Shape | Refactor Priority | Complexity | Fusion Benefit |
   |---|-----------|------|-----------------|---------------------|-------|-------------------|------------|----------------|
   | 1 | **TLSActor** | `impl/io/TLSActor.scala` | `Actor + Pump` | `InputBunch` (2), `OutputBunch` (2), `TransferPhase`, custom `TransferState` | `BidiShape` | **High** | Very High | Low |
   | 2 | **Unzip (Legacy)** | `impl/FanOut.scala:328` | `FanOut + Pump` | `OutputBunch`, `BatchingInputBuffer`, `TransferPhase` | `FanOutShape2` | **High** | Low | High |
   | 3 | **FanoutProcessorImpl** | `impl/FanoutProcessor.scala:135` | `ActorProcessorImpl + Pump` | `BatchingInputBuffer`, `FanoutOutputs`, `TransferPhase` | `FlowShape` | **High** | Medium | Medium |
   | 4 | **FanoutPublisherSink** | `impl/Sinks.scala:114` | `SinkModule` → creates `FanoutProcessorImpl` | Depends on #3 | `SinkShape` | Medium | Medium | Medium |
   | 5 | **FanIn (Base Class)** | `impl/FanIn.scala` | `Actor + Pump` | `InputBunch`, `SimpleOutputs` | Variable | Low | Medium | High |
   | 6 | **FanOut (Base Class)** | `impl/FanOut.scala:273` | `Actor + Pump` | `OutputBunch`, `BatchingInputBuffer` | Variable | Low | Medium | High |
   | 7 | **ActorProcessorImpl** | `impl/ActorProcessor.scala:272` | `Actor + Pump` | `BatchingInputBuffer`, `SimpleOutputs` | `FlowShape` | Low | Medium | Low |
   | 8 | **TlsModule** | `impl/io/TlsModule.scala` | `AtomicModule` + custom Island | `TlsModuleIslandTag`, spawns `TLSActor` | `BidiShape` | **Done (reference)** | High | Low |
   
   ---
   
   ### Detailed Breakdown
   
   #### 1. TLSActor (Highest Priority)
   The most complex legacy component. Handles bidirectional SSL/TLS encryption/decryption with a sophisticated state machine using `TransferPhase` patterns (`bidirectional`, `flushingOutbound`, `awaitingClose`, `outboundClosed`, `inboundClosed`). Uses custom `TransferState` implementations like `ChoppingBlock` for byte chunking.
   
   **Key complexity**: SSL handshake state management, half-closed connection handling, buffer chopping/put-back logic, and the `maxTLSIterations` loop guard for unwrapping.
   
   #### 2. Unzip (Legacy) (High Priority, Low Complexity)
   Simple fan-out that splits `Tuple2`/`Pair` elements into two output streams. Already has a modern `GraphStage` counterpart in `UnzipWith2`. The legacy version can be directly replaced.
   
   #### 3. FanoutProcessorImpl (High Priority)
   Reactive Streams processor supporting multiple downstream subscribers. Includes `SubscriptionTimeout` handling. Used by `BroadcastHub` and similar fan-out patterns.
   
   ---
   
   ### Legacy Infrastructure Dependency Map
   
   ```
   Pump (trait)
   ├── FanIn (Actor) ─── InputBunch ─── BatchingInputBuffer
   │   └── (no direct subclasses in use)
   │
   ├── FanOut (Actor) ─── OutputBunch ─── FanoutOutputs
   │   └── Unzip ← Component #2
   │
   ├── ActorProcessorImpl (Actor) ─── SimpleOutputs
   │   └── FanoutProcessorImpl ← Component #3
   │       └── FanoutPublisherSink ← Component #4
   │
   └── TLSActor (Actor) ─── InputBunch + OutputBunch ← Component #1
   ```
   
   ---
   
   ### Advantages of Refactoring to GraphStage
   
   1. **Eliminate Legacy Infrastructure**: Each refactored component removes dependency on the `Pump`/`InputBunch`/`OutputBunch`/`TransferPhase` system, which is the old Akka Streams architecture predating GraphStage. This reduces the overall codebase maintenance burden.
   
   2. **Materializer Simplification**: Components with special `Island` tags (`TlsModule`, `SourceModule`, `SinkModule`, `ProcessorModule`) require separate materialization phases. Refactoring to GraphStage means fewer islands, simpler traversal, and less special-case handling in `PhasedFusingActorMaterializer`.
   
   3. **Uniform Architecture**: GraphStage is the modern, well-understood abstraction. Having all operators use the same model makes the codebase more consistent, easier to understand, and simpler for new contributors.
   
   4. **Better Testability**: GraphStage logic (`GraphStageLogic`) is more testable in isolation compared to actor-based implementations that require actor system setup and message passing.
   
   5. **Future-Proofing**: The legacy Pump infrastructure is technical debt from early Akka Streams. Removing it incrementally (one component at a time, opt-in via config like the TlsModule approach) allows safe migration without breaking changes.
   
   6. **Fusion Opportunities**: Some legacy components (like Unzip) could potentially be fused with adjacent stages when converted to GraphStage, reducing the number of async boundaries and improving throughput. Note: TLSActor likely won't benefit from fusion (must run as its own island due to SSLEngine state management), but still benefits from uniform architecture.
   
   7. **Reduced Actor Overhead**: Each legacy component spawns a dedicated actor. GraphStage components can be fused into a single `ActorGraphInterpreter` actor, reducing actor count, memory overhead, and context switching.
   
   ---
   
   ### Recommended Order
   
   | Phase | Components | Rationale |
   |-------|------------|-----------|
   | **Phase 1** | Unzip (#2) | Lowest complexity, highest confidence gain. Proves the migration pattern. |
   | **Phase 2** | TLSActor + TlsModule (#1) | Highest impact. Follow the opt-in pattern (system property → config) described in the original proposal. |
   | **Phase 3** | FanoutProcessorImpl + FanoutPublisherSink (#3, #4) | Medium complexity, enables removal of `ProcessorModuleIslandTag`. |
   | **Phase 4** | FanIn/FanOut base classes (#5, #6) | Clean up after #2 is done. Evaluate if they have remaining users. |
   | **Phase 5** | ActorProcessorImpl (#7) | deepest integration with Reactive Streams adapter, handle last. |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org