You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/19 14:59:25 UTC
[GitHub] [pulsar] cbornet opened a new issue, #17719: PIP-208: HTTP Sink
cbornet opened a new issue, #17719:
URL: https://github.com/apache/pulsar/issues/17719
### Motivation
Currently, when you want to consume from Pulsar topics in applications written in languages that don't have a Pulsar driver supported, you need to run some type of proxy like the WebSocket Proxy or Pulsar Beam. In production this needs additional effort to deploy, scale, load balance, monitor, and so on...
Pulsar IO is a framework that deals with all these operational subjects and can be leveraged to provide a way to push messages to external systems using HTTP, a protocol supported by every existing language and OS.
### Goal
This proposal defines an HTTP Sink that sends the messages to a configured URL.
It takes inspiration from [Pulsar Beam](https://github.com/kafkaesque-io/pulsar-beam) and the [Confluent HTTP Sink connector](https://docs.confluent.io/kafka-connectors/http/current/overview.html).
### API Changes
_No response_
### Implementation
A `pulsar-io-http` module will be added to `pulsar-io`.
On building the project `pulsar-io-http-{version}.nar` will be built and added to the `pulsar-all` distribution.
The name of the Sink will be `http`.
The HTTP Sink pushes records to any HTTP server with the record value in the body of a POST method.
The body of the HTTP request is the JSON representation of the record value.
Some headers are added to the HTTP request:
* `PulsarTopic`: the topic of the record
* `PulsarKey`: the key of the record
* `PulsarEventTime`: the event time of the record
* `PulsarPublishTime`: the publish time of the record
* `PulsarMessageId`: the ID of the message contained in the record
* `PulsarProperties-*`: each record property is passed with the property name prefixed by `PulsarProperties-`
### Alternatives
Creating a separated project for this Sink is rejected since:
* this Sink is very useful for developers to test the Pulsar IO framework, transform functions, and to make demos.
* the code has a very small footprint with no external dependencies.
* it should be visible at the same level as other sinks
### Anything else?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #17719: PIP-208: HTTP Sink
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #17719:
URL: https://github.com/apache/pulsar/issues/17719#issuecomment-1284820704
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on issue #17719: PIP-208: HTTP Sink
Posted by GitBox <gi...@apache.org>.
cbornet commented on issue #17719:
URL: https://github.com/apache/pulsar/issues/17719#issuecomment-1325115206
Closed by #17581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet closed issue #17719: PIP-208: HTTP Sink
Posted by GitBox <gi...@apache.org>.
cbornet closed issue #17719: PIP-208: HTTP Sink
URL: https://github.com/apache/pulsar/issues/17719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org