You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 21:55:08 UTC
[GitHub] [beam] damccorm opened a new issue, #21225: Support for Confluent Schema Registry
damccorm opened a new issue, #21225:
URL: https://github.com/apache/beam/issues/21225
I would like to use the Python components [`apache_beam.io.kafka.``WriteToKafka`|https://beam.apache.org/releases/pydoc/2.32.0/apache_beam.io.kafka.html#apache_beam.io.kafka.WriteToKafka] and [`apache_beam.io.kafka.ReadFromKafka`|https://beam.apache.org/releases/pydoc/2.32.0/apache_beam.io.kafka.html#apache_beam.io.kafka.ReadFromKafka] while accessing and updating [Schema Registry](https://docs.confluent.io/platform/current/schema-registry/serdes-develop/serdes-protobuf.html) for sending and reading `protobuf` messages.
In Java it is possible to achieve [this](https://beam.apache.org/releases/javadoc/2.22.0/org/apache/beam/sdk/io/kafka/KafkaIO.html):
```
PCollection<KafkaRecord<Long, GenericRecord>> input = pipeline
.apply(KafkaIO.<Long, GenericRecord>read()
.withBootstrapServers("broker_1:9092,broker_2:9092")
.withTopic("my_topic")
.withKeyDeserializer(LongDeserializer.class)
// Use Confluent Schema Registry, specify schema registry URL and value subject
.withValueDeserializer(
ConfluentSchemaRegistryDeserializerProvider.of("http://localhost:8081", "my_topic-value"))
...
```
Imported from Jira [BEAM-12871](https://issues.apache.org/jira/browse/BEAM-12871). Original Jira may contain additional context.
Reported by: denesb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lydian commented on issue #21225: Support for Confluent Schema Registry
Posted by "lydian (via GitHub)" <gi...@apache.org>.
lydian commented on issue #21225:
URL: https://github.com/apache/beam/issues/21225#issuecomment-1554194367
I would also need this feature as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] faelchristofano commented on issue #21225: Support for Confluent Schema Registry
Posted by GitBox <gi...@apache.org>.
faelchristofano commented on issue #21225:
URL: https://github.com/apache/beam/issues/21225#issuecomment-1370047935
This feature will be extremely useful. Pythons is one of the most popular languages to handler data and it will help to diffuse more the Apache Beam to consume data from kafka and ensure that the producer/consumer won't broke in case changes in the schema.
I have a use case that this feature will help me a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org