[GitHub] [incubator-pinot] alexandervivas commented on issue #6433: Consumption from external kafka instances in third party services

[GitBox <gi...@apache.org>]

alexandervivas commented on issue #6433:
URL: https://github.com/apache/incubator-pinot/issues/6433#issuecomment-759484544


   Hey @fx19880617 and @npawar, thanks for commenting.
   
   By the time we started to test pinot for our analytics modules in pinot docs there wasn't any way to configure a LLC since for that we needed the kafka's zookeeper address and we don't have access to it in the third party platform so we decided to go for the one that could read the entire stream.
   
   So far our table config have been looked like this:
   
   `{
     "tableName":"dpt_video_event_captured_v2",
     "tableType":"REALTIME",
     "segmentsConfig":{
       "timeColumnName":"EVENT_TIME",
       "timeType":"SECONDS",
       "retentionTimeUnit":"DAYS",
       "retentionTimeValue":"3650",
       "segmentAssignmentStrategy":"BalanceNumSegmentAssignmentStrategy",
       "schemaName":"dpt_video_event_captured_v2",
       "replication":"1",
       "replicasPerPartition":"1"
     },
     "tenants":{
   
     },
     "tableIndexConfig":{
       "loadMode":"MMAP",
       "streamConfigs":{
         "streamType":"kafka",
         "stream.kafka.consumer.type":"simple",
         "stream.kafka.topic.name":"dpt_video_event-captured_v2",
         "stream.kafka.decoder.class.name":"org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
         "stream.kafka.consumer.factory.class.name":"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
         "stream.kafka.broker.list":"$KAFKA_BROKER_URL",
         "stream.kafka.hlc.bootstrap.server":"$KAFKA_BROKER_URL",
         "schema.registry.url":"$SCHEMA_REGISTRY_URL",
         "security.protocol":"SSL",
         "ssl.truststore.location":"/opt/pinot/deployment/kafka/truststore.jks",
         "ssl.truststore.password":"$KAFKA_BROKER_PASSWORD",
         "ssl.keystore.location":"/opt/pinot/deployment/kafka/keystore.p12",
         "ssl.keystore.password":"$KAFKA_BROKER_PASSWORD",
         "ssl.keystore.type":"PKCS12",
         "ssl.key.password":"$KAFKA_BROKER_PASSWORD",
         "stream.kafka.decoder.prop.schema.registry.rest.url":"$SCHEMA_REGISTRY_URL",
         "stream.kafka.decoder.prop.schema.registry.ssl.truststore.location":"/opt/pinot/deployment/sr/truststore.jks",
         "stream.kafka.decoder.prop.schema.registry.ssl.truststore.password":"$SCHEMA_REGISTRY_PASSWORD",
         "stream.kafka.decoder.prop.schema.registry.ssl.truststore.type":"JKS",
         "stream.kafka.decoder.prop.schema.registry.ssl.keystore.location":"/opt/pinot/deployment/sr/keystore.p12",
         "stream.kafka.decoder.prop.schema.registry.ssl.keystore.password":"$SCHEMA_REGISTRY_PASSWORD",
         "stream.kafka.decoder.prop.schema.registry.ssl.keystore.type":"PKCS12",
         "stream.kafka.decoder.prop.schema.registry.ssl.key.password":"$SCHEMA_REGISTRY_PASSWORD",
         "stream.kafka.decoder.prop.schema.registry.ssl.protocol":"SSL",
         "stream.kafka.decoder.prop.schema.registry.basic.auth.credentials.source":"USER_INFO",
         "stream.kafka.decoder.prop.schema.registry.basic.auth.user.info":"$SCHEMA_REGISTRY_USERNAME:$SCHEMA_REGISTRY_PASSWORD",
         "realtime.segment.flush.threshold.time":"30s",
         "realtime.segment.flush.threshold.size":"0",
         "realtime.segment.flush.desired.size":"500M",
         "stream.kafka.consumer.prop.auto.offset.reset":"smallest",
         "stream.kafka.hlc.group.id":"dtp-mls-pinot-tables-$NAMESPACE"
       },
       "invertedIndexColumns":[ "EXTRAPARAM15" ]
     },
     "metadata":{
       "customConfigs":{
   
       }
     }
   }`
   
   We know very little about how to properly configure it and we're trying to tune it by using your auto tune tool, but we don't actually know for sure, for example, if all those config params are actually necessary (besides the ones used to connect to kafka via basic auth with ssl certificates).
   
   Now I see the docs changed and now it is possible to configure it using an LLC but then, if the group id is not needed, or used in the path consumption, how do we properly Identify pinot's consumers when we go into the third party platform?
   
   If for some reason we want to play a bit with that topic for other kind of analytics then how do we make pinot consume it without getting in the way of the first one?
   
   Any suggestions are greatly appreciated.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org