You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "junaiddshaukat (via GitHub)" <gi...@apache.org> on 2026/06/20 11:41:46 UTC

[I] [GSoC 2026] Kafka Streams runner — KStreamsPayload Serde (GroupByKey prerequisite) [beam]

junaiddshaukat opened a new issue, #39042:
URL: https://github.com/apache/beam/issues/39042

   Tracking issue: #18479
   Follows: #38987 (WatermarkManager part 2, merged)
   Unblocks: GroupByKey
   
   ## Summary
   A Kafka Serde for KStreamsPayload so the envelope can cross a topic boundary. Until now KStreamsPayload only flowed in-JVM via ProcessorContext#forward, so no serialization was needed; GroupByKey introduces the first real topic (the key-based repartition topic, per the plan agreed with @je-ik), which needs the payload serialized. Splitting this out as its own small PR ahead of GBK.
   
   ## Scope
   - KStreamsPayloadSerde<T>: parameterized by the Coder<WindowedValue<T>> for the
     data variant (different topics carry different element types; the watermark
     variant is coder-independent). Wire format is a one-byte discriminator + body:
     data = [0x00][windowedValueCoder-encoded value]; watermark =
     [0x01][long millis][int sourcePartition][int totalSourcePartitions].
   - Unit tests: round-trip of data, watermark, and terminal MAX watermark
     payloads, plus an unknown-tag failure.
   
   ## Out of scope
   - Wiring the serde into the GBK repartition / watermark fan-out — that's the
     GroupByKey PR.
   
   ## Notes
   - Assumes non-null payloads: the topics it's used on (repartition, watermark
     fan-out) are not log-compacted, so no tombstones occur.
   
   ## Testing
   ./gradlew :runners:kafka-streams:check green; 4 unit tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GSoC 2026] Kafka Streams runner — KStreamsPayload Serde (GroupByKey prerequisite) [beam]

Posted by "junaiddshaukat (via GitHub)" <gi...@apache.org>.
junaiddshaukat commented on issue #39042:
URL: https://github.com/apache/beam/issues/39042#issuecomment-4757604049

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GSoC 2026] Kafka Streams runner — KStreamsPayload Serde (GroupByKey prerequisite) [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik closed issue #39042: [GSoC 2026] Kafka Streams runner — KStreamsPayload Serde (GroupByKey prerequisite)
URL: https://github.com/apache/beam/issues/39042


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [GSoC 2026] Kafka Streams runner — KStreamsPayload Serde (GroupByKey prerequisite) [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on issue #39042:
URL: https://github.com/apache/beam/issues/39042#issuecomment-4815792735

   Closed via #39051 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org