You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "david-simplehash (via GitHub)" <gi...@apache.org> on 2023/06/06 17:28:23 UTC
[GitHub] [beam] david-simplehash opened a new issue, #27033: [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK
david-simplehash opened a new issue, #27033:
URL: https://github.com/apache/beam/issues/27033
### What would you like to happen?
Support for adding record headers in `WriteToKafka` in python SDK.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK [beam]
Posted by "grusc (via GitHub)" <gi...@apache.org>.
grusc commented on issue #27033:
URL: https://github.com/apache/beam/issues/27033#issuecomment-3204663683
I'm willing to take this issue, however I need a little guidance on the approach.
From my naive perspective it could work to register WriteRecords for external invocation. But I'm not really sure how I get the PCollection on the Python side setup in a way that the transform on the Java side receives a PCollection<ProducerRecord> as input.
@liferoad maybe you can share some quick inside if that is even feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #27033:
URL: https://github.com/apache/beam/issues/27033#issuecomment-3206242773
If you haven't worked on the cross-lang before, better check https://beam.apache.org/documentation/sdks/java-multi-language-pipelines/ first and understand how it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #27033:
URL: https://github.com/apache/beam/issues/27033#issuecomment-3137561000
cc @an2x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK [beam]
Posted by "jboudouar-onclusive (via GitHub)" <gi...@apache.org>.
jboudouar-onclusive commented on issue #27033:
URL: https://github.com/apache/beam/issues/27033#issuecomment-3135237723
Any updates on this? Headers are supported in ReadFromKafka but still not in WriteToKafka. This is an unusual inconsistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK [beam]
Posted by "PDGGK (via GitHub)" <gi...@apache.org>.
PDGGK commented on issue #27033:
URL: https://github.com/apache/beam/issues/27033#issuecomment-3822498617
Hi, I'd like to work on this issue.
I've analyzed the current implementation:
- Python `WriteToKafka` is an `ExternalTransform` wrapper (kafka.py:270)
- Uses URN `beam:transform:org.apache.beam:kafka_write:v1`
- Java side maps KV → ProducerRecord without headers support
- Headers need per-record support via Row schema input
My plan:
1. Extend Java `KafkaIO.Write` to accept Row input with `{key, value, headers[]}` schema
2. Update Python wrapper to support Row-based input
3. Add cross-language integration tests
Estimated: ~300-500 lines across Python + Java + tests.
I'm an existing contributor with merged PRs (#37298).
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK [beam]
Posted by "johnjcasey (via GitHub)" <gi...@apache.org>.
johnjcasey closed issue #27033: [Feature Request]: Add support to write record headers in `WriteToKafka` in python SDK
URL: https://github.com/apache/beam/issues/27033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org