[GitHub] [incubator-pinot] kriti-sc edited a comment on pull request #7081: [#6823] Obfuscation of sensitive config

[GitBox <gi...@apache.org>]

kriti-sc edited a comment on pull request #7081:
URL: https://github.com/apache/incubator-pinot/pull/7081#issuecomment-874337369


   @npawar @xiangfu0 
   
   Working on obfuscating credentials (aws access & secret key for kinesis; truststore passwords, etc for kafka) - https://github.com/kriti-sc/incubator-pinot/pull/2/files [WIP]
   
   Ingestion configs for both offline and realtime tables will be passed in the `ingestionConfig` key, like below? 
   ```
   ...
   },
     "ingestionConfig": {
       "batchIngestionConfig": {
         "segmentIngestionType": "APPEND",
         "segmentIngestionFrequency": "DAILY",
         "batchConfigMaps": [
           {
             "inputDirURI": "s3://my.s3.bucket/batch/airlineStats/rawdata/",
             "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
             "input.fs.prop.region": "us-west-2",
             "includeFileNamePattern": "glob:**/*.avro",
             "excludeFileNamePattern": "glob:**/*.tmp",
             "inputFormat": "avro",
             "outputDirURI": "s3://my.s3.bucket/batch/airlineStats/segments",
             "push.mode": "metadata",
             "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
             "input.fs.prop.region": "us-west-2",
             "input.fs.prop.accessKey": "ASIAV5GTANORQHDSIFPO",
             "input.fs.prop.secretKey": "ASIAV5GTANORQHDSIFPO"
           }
         ],
         "segmentNameSpec": {},
         "pushSpec": {}
       }
     },...
   ```
   
   And passing configs for realtime tables in the `tableIndexConfig` has been deprecated, like below?
   ```
   ....,
   "tableIndexConfig": {
       "loadMode": "MMAP",
       "invertedIndexColumns": [
         "organization",
         "repo"
       ],
       "streamConfigs": {
         "streamType": "kinesis",
         "stream.kinesis.topic.name": "delftstream",
         "region": "us-east-1",
         "accessKey": "ASIAV5GTANOR5V3USMDL",
         "secretKey": "rW+0UXnwrgSADyvfkXzDloc6V1Kfux5vId+2Kbst",
         "shardIteratorType": "AFTER_SEQUENCE_NUMBER",
         "stream.kinesis.consumer.type": "highlevel",
         "stream.kinesis.fetch.timeout.millis": "30000",
         "stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
         "stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
         "realtime.segment.flush.threshold.size": "1000000",
         "realtime.segment.flush.threshold.time": "6h"
       }
     },...
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org