You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "junaiddshaukat (via GitHub)" <gi...@apache.org> on 2026/06/08 09:55:28 UTC

[PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

junaiddshaukat opened a new pull request, #38843:
URL: https://github.com/apache/beam/pull/38843

   ## Summary
   Direct follow-up to #38764, per Slack with @je-ik:
   
   1. Implement `Redistribute.arbitrarily()` as a runner-native passthrough so the fuser terminates stages at the Redistribute boundary, enabling chained-stage E2E tests without dragging GBK in.
   2. Drop the `byte[]` type bound on `ExecutableStageProcessor` / `KStreamsPayload` at the processor edge so non-byte[] outputs (e.g. `Integer` from `MapElements`) flow stage-to-stage without the cast lying about runtime type.
   3. Add a chained `Impulse -> MapElements<Integer> -> Redistribute -> ParDo` E2E test under the EMBEDDED Java SDK harness, asserting the Integer survives the runner round-trip via `SharedTestCollector`.
   
   ## Wiring details
   - `knownUrns()` exposed; `TrivialNativeTransformExpander.forKnownUrns(...)` runs before `GreedyPipelineFuser.fuse(...)` so the Redistribute composite's sub-transforms (which include GBK, not yet supported) are stripped before fusion.
   - `@AutoService(NativeTransforms.IsNativeTransform.class)` registers Redistribute URN so `QueryablePipeline` treats the stripped composite as a primitive producer of its outputs. Mirrors Flink's `IsFlinkNativeTransform`.
   
   ## Out of scope
   - `Redistribute.byKey()` URN — rehashing belongs with the GroupByKey sub-issue.
   - WatermarkManager — separate sub-issue.
   
   ## Validation
   `./gradlew :runners:kafka-streams:check` green locally, 15 tests pass (incl. new `ChainedExecutableStageTest`).
   
   Closes #38840
   Refs #18479
   cc @je-ik


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on code in PR #38843:
URL: https://github.com/apache/beam/pull/38843#discussion_r3372797256


##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/ExecutableStageProcessor.java:
##########
@@ -157,26 +161,25 @@ private void closeBundleAndFlush(Record<byte[], KStreamsPayload<byte[]>> record)
     } finally {
       currentBundle = null;
     }
-    ProcessorContext<byte[], KStreamsPayload<byte[]>> ctx = checkInitialized(context);
+    ProcessorContext<byte[], KStreamsPayload<?>> ctx = checkInitialized(context);
     // The harness has finished the bundle (close() returned) so no further enqueues happen.
     // ConcurrentLinkedQueue's weakly-consistent iterator is therefore safe to drain via forEach.
     pendingOutputs.forEach(
         output ->
             ctx.forward(
-                new Record<byte[], KStreamsPayload<byte[]>>(
+                new Record<byte[], KStreamsPayload<?>>(
                     record.key(), KStreamsPayload.data(output), record.timestamp())));
     pendingOutputs.clear();

Review Comment:
   :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on code in PR #38843:
URL: https://github.com/apache/beam/pull/38843#discussion_r3380519907


##########
runners/kafka-streams/src/test/java/org/apache/beam/runners/kafka/streams/translation/ChainedExecutableStageTest.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.kafka.streams.translation;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.kafka.streams.KafkaStreamsPipelineOptions;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.CrashingRunner;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Impulse;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.util.construction.Environments;
+import org.apache.beam.sdk.util.construction.PipelineOptionsTranslation;
+import org.apache.beam.sdk.util.construction.PipelineTranslation;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.junit.Test;
+
+/**
+ * End-to-end test that exercises non-{@code byte[]} value types flowing across an {@code
+ * ExecutableStage} boundary.
+ *
+ * <p>Builds a pipeline {@code Impulse -> MapElements<byte[], Integer> -> Redistribute.arbitrarily()
+ * -> ParDo<Integer, Void>(record into SharedTestCollector)}. The Redistribute boundary terminates
+ * the upstream {@code ExecutableStage} (because the runner registers Redistribute as a
+ * runner-native transform in {@link KafkaStreamsPipelineTranslator#knownUrns()}), so the
+ * Integer-emitting stage and the recording stage live in separate stages and the Integer payload
+ * actually flows stage-to-stage through the runner — not just through user code inside a single
+ * fused stage. That is what proves the {@code byte[]}-erasure cast we used to do was a latent bug,
+ * and that the type-agnostic {@code KStreamsPayload<?>} edge handles it correctly.
+ */
+public class ChainedExecutableStageTest {
+
+  private static final String JOB_ID = "ks-chained-stage-test";
+  private static final String APPLICATION_ID = "ks-chained-stage-test";
+  private static final int EXPECTED_VALUE = 42;
+
+  /**
+   * Records the integer payload it sees so the test can verify it survived the harness round-trip
+   * across the Redistribute boundary.
+   */
+  private static class RecordingFn extends DoFn<Integer, Void> {
+    private final SharedTestCollector<Integer> collector;
+
+    RecordingFn(SharedTestCollector<Integer> collector) {
+      this.collector = collector;
+    }
+
+    @ProcessElement
+    public void processElement(@Element Integer input) {
+      collector.record(input);
+    }
+  }
+
+  @Test
+  public void chainedStagesPropagateIntegerValueAcrossRedistributeBoundary() throws Exception {
+    try (SharedTestCollector<Integer> collector = SharedTestCollector.create()) {
+      Pipeline pipeline = Pipeline.create(pipelineOptions());
+      pipeline
+          .apply("impulse", Impulse.create())
+          .apply(
+              "to-integer",
+              MapElements.into(TypeDescriptors.integers()).via((byte[] ignored) -> EXPECTED_VALUE))
+          .setTypeDescriptor(TypeDescriptor.of(Integer.class))
+          .apply("redistribute", Redistribute.<Integer>arbitrarily())
+          .apply("record", ParDo.of(new RecordingFn(collector)));
+
+      RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline);

Review Comment:
   Can we verify the Pipeline has two ExecutableStages here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #38843:
URL: https://github.com/apache/beam/pull/38843#discussion_r3372267621


##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/RedistributeTranslator.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.kafka.streams.translation;
+
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Iterables;
+
+/**
+ * Runner-native translator for {@code beam:transform:redistribute_arbitrarily:v1}.
+ *
+ * <p>The default Beam expansion of {@code Redistribute.arbitrarily()} goes through {@code
+ * GroupByKey} for materialization. The Kafka Streams runner does not need to do any actual
+ * redistribution in the single-instance topology — Kafka Streams is already handling per-task
+ * processing — so we provide a passthrough: the output PCollection is mapped to the same processor
+ * that already produces the input PCollection. No topology node is added.
+ *
+ * <p>The translator is registered in {@link KafkaStreamsPipelineTranslator#knownUrns()} so {@link
+ * org.apache.beam.sdk.util.construction.graph.TrivialNativeTransformExpander} strips the
+ * sub-transforms of {@code Redistribute} before the fuser runs. That keeps the Redistribute
+ * boundary intact and lets the fuser split adjacent stages correctly.
+ *
+ * <p>The keyed variant ({@code redistribute_by_key:v1}) is intentionally not handled here: it
+ * implies a rehash/partition step that the runner can implement on top of GroupByKey when that
+ * lands, but cannot reasonably no-op the way the arbitrary variant can.
+ */
+class RedistributeTranslator implements PTransformTranslator {
+
+  @Override
+  public void translate(
+      String transformId, RunnerApi.Pipeline pipeline, KafkaStreamsTranslationContext context) {
+    RunnerApi.PTransform transform = pipeline.getComponents().getTransformsOrThrow(transformId);
+    String inputPCollectionId = Iterables.getOnlyElement(transform.getInputsMap().values());
+    String parentProcessor = context.getProcessorNameForPCollection(inputPCollectionId);
+    String outputPCollectionId = Iterables.getOnlyElement(transform.getOutputsMap().values());

Review Comment:
   ![medium](https://www.gstatic.com/codereviewagent/medium-priority.svg)
   
   Add a defensive null check for `parentProcessor`. If the input PCollection does not have a registered producer processor (e.g., due to an unexpected pipeline structure or translation order issue), this will fail fast with a clear error message instead of causing a downstream `NullPointerException`.
   
   ```java
       String parentProcessor = context.getProcessorNameForPCollection(inputPCollectionId);
       if (parentProcessor == null) {
         throw new IllegalStateException(
             String.format(
                 "No producer processor found for input PCollection %s of transform %s",
                 inputPCollectionId, transformId));
       }
       String outputPCollectionId = Iterables.getOnlyElement(transform.getOutputsMap().values());
   ```



##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/ExecutableStageProcessor.java:
##########
@@ -157,26 +161,25 @@ private void closeBundleAndFlush(Record<byte[], KStreamsPayload<byte[]>> record)
     } finally {
       currentBundle = null;
     }
-    ProcessorContext<byte[], KStreamsPayload<byte[]>> ctx = checkInitialized(context);
+    ProcessorContext<byte[], KStreamsPayload<?>> ctx = checkInitialized(context);
     // The harness has finished the bundle (close() returned) so no further enqueues happen.
     // ConcurrentLinkedQueue's weakly-consistent iterator is therefore safe to drain via forEach.
     pendingOutputs.forEach(
         output ->
             ctx.forward(
-                new Record<byte[], KStreamsPayload<byte[]>>(
+                new Record<byte[], KStreamsPayload<?>>(
                     record.key(), KStreamsPayload.data(output), record.timestamp())));
     pendingOutputs.clear();

Review Comment:
   ![medium](https://www.gstatic.com/codereviewagent/medium-priority.svg)
   
   Draining the queue using `poll()` in a loop is more idiomatic and robust than using `forEach` followed by `clear()`. It avoids relying on the weakly-consistent iterator of `ConcurrentLinkedQueue` and ensures elements are safely removed as they are processed, preventing any potential race conditions or memory retention issues if the queue is accessed concurrently.
   
   ```suggestion
       // The harness has finished the bundle (close() returned) so no further enqueues happen.
       WindowedValue<?> output;
       while ((output = pendingOutputs.poll()) != null) {
         ctx.forward(
             new Record<>(
                 record.key(), KStreamsPayload.data(output), record.timestamp()));
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38843:
URL: https://github.com/apache/beam/pull/38843#issuecomment-4660241024

   Assigning reviewers:
   
   R: @kennknowles added as fallback since no labels match configuration
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik merged PR #38843:
URL: https://github.com/apache/beam/pull/38843


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "junaiddshaukat (via GitHub)" <gi...@apache.org>.
junaiddshaukat commented on code in PR #38843:
URL: https://github.com/apache/beam/pull/38843#discussion_r3379309095


##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/ExecutableStageProcessor.java:
##########
@@ -157,26 +161,25 @@ private void closeBundleAndFlush(Record<byte[], KStreamsPayload<byte[]>> record)
     } finally {
       currentBundle = null;
     }
-    ProcessorContext<byte[], KStreamsPayload<byte[]>> ctx = checkInitialized(context);
+    ProcessorContext<byte[], KStreamsPayload<?>> ctx = checkInitialized(context);
     // The harness has finished the bundle (close() returned) so no further enqueues happen.
     // ConcurrentLinkedQueue's weakly-consistent iterator is therefore safe to drain via forEach.
     pendingOutputs.forEach(
         output ->
             ctx.forward(
-                new Record<byte[], KStreamsPayload<byte[]>>(
+                new Record<byte[], KStreamsPayload<?>>(
                     record.key(), KStreamsPayload.data(output), record.timestamp())));
     pendingOutputs.clear();

Review Comment:
   switched to poll() loop in the latest commit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "junaiddshaukat (via GitHub)" <gi...@apache.org>.
junaiddshaukat commented on code in PR #38843:
URL: https://github.com/apache/beam/pull/38843#discussion_r3381682442


##########
runners/kafka-streams/src/test/java/org/apache/beam/runners/kafka/streams/translation/ChainedExecutableStageTest.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.kafka.streams.translation;
+
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.kafka.streams.KafkaStreamsPipelineOptions;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.options.PortablePipelineOptions;
+import org.apache.beam.sdk.testing.CrashingRunner;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.Impulse;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.Redistribute;
+import org.apache.beam.sdk.util.construction.Environments;
+import org.apache.beam.sdk.util.construction.PipelineOptionsTranslation;
+import org.apache.beam.sdk.util.construction.PipelineTranslation;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.beam.sdk.values.TypeDescriptors;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.TopologyTestDriver;
+import org.junit.Test;
+
+/**
+ * End-to-end test that exercises non-{@code byte[]} value types flowing across an {@code
+ * ExecutableStage} boundary.
+ *
+ * <p>Builds a pipeline {@code Impulse -> MapElements<byte[], Integer> -> Redistribute.arbitrarily()
+ * -> ParDo<Integer, Void>(record into SharedTestCollector)}. The Redistribute boundary terminates
+ * the upstream {@code ExecutableStage} (because the runner registers Redistribute as a
+ * runner-native transform in {@link KafkaStreamsPipelineTranslator#knownUrns()}), so the
+ * Integer-emitting stage and the recording stage live in separate stages and the Integer payload
+ * actually flows stage-to-stage through the runner — not just through user code inside a single
+ * fused stage. That is what proves the {@code byte[]}-erasure cast we used to do was a latent bug,
+ * and that the type-agnostic {@code KStreamsPayload<?>} edge handles it correctly.
+ */
+public class ChainedExecutableStageTest {
+
+  private static final String JOB_ID = "ks-chained-stage-test";
+  private static final String APPLICATION_ID = "ks-chained-stage-test";
+  private static final int EXPECTED_VALUE = 42;
+
+  /**
+   * Records the integer payload it sees so the test can verify it survived the harness round-trip
+   * across the Redistribute boundary.
+   */
+  private static class RecordingFn extends DoFn<Integer, Void> {
+    private final SharedTestCollector<Integer> collector;
+
+    RecordingFn(SharedTestCollector<Integer> collector) {
+      this.collector = collector;
+    }
+
+    @ProcessElement
+    public void processElement(@Element Integer input) {
+      collector.record(input);
+    }
+  }
+
+  @Test
+  public void chainedStagesPropagateIntegerValueAcrossRedistributeBoundary() throws Exception {
+    try (SharedTestCollector<Integer> collector = SharedTestCollector.create()) {
+      Pipeline pipeline = Pipeline.create(pipelineOptions());
+      pipeline
+          .apply("impulse", Impulse.create())
+          .apply(
+              "to-integer",
+              MapElements.into(TypeDescriptors.integers()).via((byte[] ignored) -> EXPECTED_VALUE))
+          .setTypeDescriptor(TypeDescriptor.of(Integer.class))
+          .apply("redistribute", Redistribute.<Integer>arbitrarily())
+          .apply("record", ParDo.of(new RecordingFn(collector)));
+
+      RunnerApi.Pipeline pipelineProto = PipelineTranslation.toProto(pipeline);

Review Comment:
   Done added an assertion right after `prepareForTranslation` that counts ExecutableStage URN transforms in the prepared pipeline proto and requires exactly two. Verified locally that the two stages are the upstream MapElements stage and the downstream RecordingFn ParDo stage, separated by the runner-native Redistribute boundary. If fusion ever collapsed them the test fails before we even hit the topology driver. Thanks!
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner — Redistribute translator + ExecutableStage type-agnostic edge [beam]

Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #38843:
URL: https://github.com/apache/beam/pull/38843#issuecomment-4647616851

   ## Summary of Changes
   
   Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
   
   This pull request improves the Kafka Streams runner by enabling support for arbitrary data types across stage boundaries and introducing a native implementation for Redistribute.arbitrarily(). These changes allow for more complex pipeline structures and resolve previous limitations regarding type-specific casting, ensuring robust data flow between fused stages.
   
   ### Highlights
   
   * **Redistribute.arbitrarily() Implementation**: Implemented a runner-native passthrough for Redistribute.arbitrarily() to allow the fuser to terminate stages at the boundary, facilitating chained-stage E2E testing.
   * **Type-Agnostic Processor Edge**: Removed the byte[] type bound on ExecutableStageProcessor and KStreamsPayload, allowing non-byte[] outputs to flow between stages without incorrect runtime type casting.
   * **New E2E Test**: Added a chained E2E test (Impulse -> MapElements -> Redistribute -> ParDo) to verify that non-byte[] data types correctly survive the runner round-trip.
   
   <details>
   <summary><b>New Features</b></summary>
   
   🧠 You can now enable **Memory** (public preview) to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
   </details>
   
   <details>
   <summary><b>Using Gemini Code Assist</b></summary>
   <br>
   
   The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
   
   <b>Invoking Gemini</b>
   
   You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
   
   Feature | Command | Description
   --- | --- | ---
   Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
   Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
   Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
   Help | `/gemini help` | Displays a list of available commands.
   
   <b>Customization</b>
   
   To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
   
   <b>Limitations & Feedback</b>
   
   Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
   </details>
   
   
   [^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org