You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Ashfaqbs (via GitHub)" <gi...@apache.org> on 2025/05/18 04:59:50 UTC
[PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Ashfaqbs opened a new pull request, #34992:
URL: https://github.com/apache/beam/pull/34992
This Apache Beam YAML pipeline demonstrates a basic Kafka-to-Kafka message mirroring use case. It reads raw byte messages from a source Kafka topic (i-topic) and writes them directly to a target Kafka topic (o-topic) without any transformation or decoding.
The pipeline is defined using Beam’s YAML DSL and leverages the ReadFromKafka and WriteToKafka transforms. It is configured to use the RAW data format, making it suitable for scenarios where the message structure is opaque or processing is deferred downstream.
This example serves as a minimal reference for validating Kafka I/O configurations and can be extended for more complex streaming dataflows. It is compatible with runners that support Kafka I/O, including Direct Runner and Flink Runner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-2888796112
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #34992: Add Kafka to Kafka RAW example pipeline configuration
URL: https://github.com/apache/beam/pull/34992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3220036575
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @liferoad for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3258142961
Reminder, please take a look at this pr: @liferoad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3324614772
Hello! To get this review moving, I recommend iterating on the changes until the tests are all green. Hopefully you can also reproduce the test failures locally, to make it easy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3397281497
Reminder, please take a look at this pr: @liferoad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3314935494
Reminder, please take a look at this pr: @shunping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3220285591
please fix the failed workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3233962749
Hi @Ashfaqbs,
1. I understand that failure is expected in this code situation, but we can't leave it like that and will need to address it in either of two ways - a) do what Robert suggested [here](https://github.com/apache/beam/pull/34992#issuecomment-2913225173) with not testing this particular example or b) update this example to match something [like](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/io/kafka.yaml). My preference is the latter option (you can use Charles's [PR](https://github.com/apache/beam/pull/35151/files) for a go by), so that all the examples that we have are tested.
2. We should also move your example to this [location](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/yaml/examples/transforms/io).
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3179765513
Assigning reviewers:
R: @claudevdm for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3220286184
remind me after tests pass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "Ashfaqbs (via GitHub)" <gi...@apache.org>.
Ashfaqbs commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3016387879
assign @robertwb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-2913225173
Looks like this is failing because we are trying to test it (and it has no output asserts nor the requisite kafka topics set up). You could explicitly skip this particular example at https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/testing/examples_test.py#L310 (though it might be good to at least ensure that it parses).
The RAT failure is because explicit license headers are required on all files, see for example https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/wordcount_minimal.yaml that you could copy over. (A brief description of this pipeline would be good as well.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3403476684
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3270693460
Unless there is something fundamentally wrong with the existing example, I would probably recommend that we keep the existing as is and then add another example that maybe you call it kafkaToKafka.yaml which essentially would be what you have now. Long as the examples work, more the better. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3206070453
Reminder, please take a look at this pr: @claudevdm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3233528749
> @liferoad , The Python PreCommit test failures are unrelated and expected — they attempt to run YAMLs without assertable output
Got it. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "Ashfaqbs (via GitHub)" <gi...@apache.org>.
Ashfaqbs commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3016390762
Hi Robert
As discussed via email, this PR contributes a minimal Kafka-to-Kafka example in RAW byte format using Beam's YAML DSL. It's a simple pass-through pipeline that reads from an input Kafka topic and writes to an output topic without any transformation or decoding. This should serve as a helpful reference for users validating Kafka I/O integration with Beam YAML.
✅ Apache license header and description are included
✅ File is renamed to `kafka-to-kafka_example.yaml` to skip automated test execution (as the pipeline has no assertions and requires Kafka infra)
✅ All non-functional checks (RAT, Lint, Formatters, Docker, etc.) have passed
❌ The Python PreCommit test failures are unrelated and expected — they attempt to run YAMLs without assertable output
Let me know if any further changes are needed. Thanks again for your time and review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "Ashfaqbs (via GitHub)" <gi...@apache.org>.
Ashfaqbs commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3287417982
Hi @derrickaw ,
I’ve reverted the original example to its initial state and created a new file: kafkaToKafka.yaml, which includes my Beam Kafka-to-Kafka mirroring example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3270440457
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @shunping for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3681969147
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "Ashfaqbs (via GitHub)" <gi...@apache.org>.
Ashfaqbs commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3232652722
@liferoad , The Python PreCommit test failures are unrelated and expected — they attempt to run YAMLs without assertable output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3179761331
assign set of reviewers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3371352380
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @liferoad for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3356039988
Reminder, please take a look at this pr: @shunping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Kafka to Kafka RAW example pipeline configuration [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34992:
URL: https://github.com/apache/beam/pull/34992#issuecomment-3650910708
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org