[GitHub] [pulsar] jianyun8023 opened a new issue #7939: S3 Sink Connector

[GitBox <gi...@apache.org>]

jianyun8023 opened a new issue #7939:
URL: https://github.com/apache/pulsar/issues/7939


   **Describe the solution you'd like**
   
   Implement a Pulsar connector to copy Pulsar messages to S3.
   
   Features
   
   - Exactly Once Delivery: We should use a deterministic partitioner to write messages to S3 buckets.
   - Support pluggable data format with or without a schema: We should support writing data to S3 in Avro and JSON format. Also, support writing data as Parquet format.
   - Support pluggable partitioner: We should support partitioning based on Pulsar partitions. Besides that, we should also support field partitioning and time-based partitioning in days or house.
   - Support Non-AWS Object Storage (optional)
   - Schema Evolution


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] jianyun8023 closed issue #7939: S3 Sink Connector

[GitBox <gi...@apache.org>]

jianyun8023 closed issue #7939:
URL: https://github.com/apache/pulsar/issues/7939


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org