Apache Pinot Daily Email Digest (2022-03-15)

[Pinot Slack Email Digest <ap...@gmail.com>]

### _#general_

  
 **@naga.b:** Hi All , Currently I am working in Pinot POC to solve OLAP based
needs in our org.Our architect team worried about using zookeeper/helix as
metadata storage . Lets say , if want to run Pinot for 5 year in production,
at certain point of time ,zookeeper will run into disk space full issue as it
store all segments metadata in it. right ? . [Druid is using sql database
(mysql/postgresdb) to store this kind of metadata] Please suggest right
solution to solve this problem. also curious to know about how other big giant
(Uber ,Linkedin , etc ) using pinot in their production setup.  
**@g.kishore:** Helix/zookeeper has been used in production for 10+ years.
managing thousands of nodes. what Pinot stores in ZK is minimal and has never
been an issue.  
**@mayanks:** Yes, this is true for multiple orgs across different companies
^^ @naga.b  
 **@ryantle1028:** Hi ALL, I'm in POC Pinot. Pinot support real time table
with kafka SASL_SSL (authen kerbelos + cert ) ?  
**@mayanks:** @slack1  

### _#troubleshooting_

  
 **@ryantle1028:** hi anyone can show me example config realtime table with
kafka sasl_ssl.  
**@xiangfu0:** Kafka consumer configs are transparently passed  
**@ryantle1028:** this is my table code { "tableName":
"user_stream_kkts_test", "tableType": "REALTIME", "tenants": { "broker":
"DefaultTenant", "server": "DefaultTenant", "tagOverrideConfig": {} },
"segmentsConfig": { "schemaName": "user_stream_kkts", "timeColumnName":
"timestampInEpoch", "replication": "1", "replicasPerPartition": "1",
"retentionTimeUnit": null, "retentionTimeValue": null, "completionConfig":
null, "crypterClassName": null, "peerSegmentDownloadScheme": null },
"tableIndexConfig": { "loadMode": "MMAP", "invertedIndexColumns": [],
"createInvertedIndexDuringSegmentGeneration": false, "rangeIndexColumns": [],
"sortedColumn": [], "bloomFilterColumns": [], "bloomFilterConfigs": null,
"noDictionaryColumns": [], "onHeapDictionaryColumns": [],
"varLengthDictionaryColumns": [], "enableDefaultStarTree": false,
"starTreeIndexConfigs": null, "enableDynamicStarTreeCreation": false,
"segmentPartitionConfig": null, "columnMinMaxValueGeneratorMode": null,
"aggregateMetrics": false, "nullHandlingEnabled": false, "streamConfigs": {
"streamType": "kafka", "stream.kafka.topic.name": "user_stream",
"stream.kafka.broker.list": "kkts01:9094,kkts02:9094,kkts03:9094",
"stream.kafka.consumer.type": "lowlevel",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest",
"stream.kafka.consumer.factory.class.name":
"org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.decoder.class.name":
"org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.segment.size": "100M",
"stream.kafka.consumer.prop.group.id": "rdp_lookup", "security.protocol":
"SASL_SSL", "sasl.mechanism": "GSSAPI", "sasl.kerberos.service.name":
"bigfoot", "ssl.truststore.location": "/data/apache-
pinot/keytab/kfuat_rdp/kkts.client.truststore.jks", "ssl.truststore.password":
"godzilla007", "sasl.jaas.config":
"com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true
useKeyTab=true storeKey=true keyTab=/data/apache-
pinot/keytab/kfuat_rdp/kfuat_rdp.client.keytab
principal=kfuat_rdp@KAFKA.SECURE;" } }, "metadata": {}, "ingestionConfig": {
"filterConfig": null, "transformConfigs": null }, "quota": { "storage": null,
"maxQueriesPerSecond": null }, "task": null, "routing": {
"segmentPrunerTypes": null, "instanceSelectorType": null }, "query": {
"timeoutMs": null }, "fieldConfigList": null, "upsertConfig": null,
"tierConfigs": null }  
**@ryantle1028:** ```{ "_code": 500, "_error":
"org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"
}```  
**@ryantle1028:** @xiangfu0 please help  
**@xiangfu0:** does this config work with Kafka console consumer ?  
**@xiangfu0:** If you can consume with Kafka console consumer config file,
then you should be able to use same configs to ingest from Pinot  

### _#flink-pinot-connector_

  
 **@naga.b:** @naga.b has joined the channel  
\--------------------------------------------------------------------- To
unsubscribe, e-mail: dev-unsubscribe@pinot.apache.org For additional commands,
e-mail: dev-help@pinot.apache.org