[GitHub] [incubator-pinot] npawar commented on pull request #7026: Add Apache Pulsar low level and high level connector

[GitBox <gi...@apache.org>]

npawar commented on pull request #7026:
URL: https://github.com/apache/incubator-pinot/pull/7026#issuecomment-856391679


   > The PR addresses #5837. The code contains both low level and high level pinot pulsar connectors. Basic testing has been done to ensure the connectors work as expected.
   > 
   > Pending items -
   > 
   > * Verify the connector when partitions are changed for a topic
   > * Add consumer for non-partitioned topics
   > * Unit and Integration Tests
   > 
   > The new properties for pulsar are
   > 
   > * `start_type` : This is the initial position from which to consume data. It can be set to either `earliest`, `latest` or messageId in the format `ledgerNum:entryNum:partitionNum`
   > * `streamType` : should be set to `pulsar`
   > * `bootstrap.servers` : comma-seperated list of pulsar brokers
   > * `stream.pulsar.consumer.factory.class.name` : should be set to `org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory`
   > 
   > ### Sample table config:
   > ```json
   > {
   >   "tableName": "pulsartTest",
   >   "tableType": "REALTIME",
   >   "segmentsConfig": {
   >     "timeColumnName": "DaysSinceEpoch",
   >     "schemaName": "airlineStats",
   >     "replicasPerPartition": "1"
   >   },
   >   "tenants": {},
   >   "tableIndexConfig": {
   >     "loadMode": "MMAP",
   >     "streamConfigs": {
   >       "streamType": "pulsar",
   >       "bootstrap.servers" : "pulsar://localhost:6650,pulsar://localhost:6651",
   >       "start_position" : "earliest", 
   >       "stream.pulsar.consumer.type": "lowlevel",
   >       "stream.pulsar.topic.name": "pulsar-test",
   >       "stream.pulsar.fetch.timeout.millis": "10000",
   >       "stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.json. StreamJSONMessageDecoder",
   >       "stream.pulsar.consumer.factory.class.name": "org.apache.pinot.plugin.stream.pulsar.PulsarConsumerFactory",
   >       "realtime.segment.flush.threshold.size": "10000",
   >       "realtime.segment.flush.threshold.time": "1h"
   >     }
   >   },
   >   "metadata": {
   >     "customConfigs": {}
   >   }
   > }
   > ```
   
   do you mind also adding an example config for highlevel consumer?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org