You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ImAbhishekTomar (via GitHub)" <gi...@apache.org> on 2023/02/03 12:05:26 UTC

[GitHub] [pinot] ImAbhishekTomar opened a new issue, #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

ImAbhishekTomar opened a new issue, #10225:
URL: https://github.com/apache/pinot/issues/10225

   Tying to create a table.
   
   > 2023/02/03 11:52:20.696 INFO [AddTableCommand] [main] {"code":500,"error":"org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"}
   
   ```
   $bin/pinot-admin.sh AddTable -schemaFile /opt/pinot/volume_app/dirrtl/dirrtl-schema.json  -tableConfigFile /opt/pinot/volume_app/dirrtl/dirrtl-table-realtime.json -exec# 
   
   [0.001s][warning][gc] -Xloggc is deprecated. Will use -Xlog:gc:gc-pinot-controller.log instead.
   SLF4J: Class path contains multiple SLF4J bindings.
   SLF4J: Found binding in [jar:file:/opt/pinot/lib/pinot-all-0.12.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-file-system/pinot-s3/pinot-s3-0.12.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-yammer/pinot-yammer-0.12.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-metrics/pinot-dropwizard/pinot-dropwizard-0.12.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-environment/pinot-azure/pinot-azure-0.12.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/opt/pinot/plugins/pinot-input-format/pinot-parquet/pinot-parquet-0.12.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
   WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
   WARNING: An illegal reflective access operation has occurred
   WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-0.12.0-jar-with-dependencies.jar) to method java.lang.Object.finalize()
   WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
   WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
   WARNING: All illegal access operations will be denied in a future release
   2023/02/03 11:52:19.942 INFO [AddTableCommand] [main] Executing command: AddTable -tableConfigFile /opt/pinot/volume_app/dirrtl/dirrtl-table-realtime.json -offlineTableConfigFile null -realtimeTableConfigFilenull -schemaFile /opt/pinot/volume_app/dirrtl/dirrtl-schema.json -controllerProtocol http -controllerHost 172.19.0.3 -controllerPort 9000 -user null -password [hidden] -exec
   2023/02/03 11:52:20.696 INFO [AddTableCommand] [main] {"code":500,"error":"org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"}
   
   ```
   
   **Table JSON**
   
   ```
   {
     "tableName": "schema-dirrtl",
     "tableType": "REALTIME",
     "tenants": {
       "broker": "DefaultTenant",
       "server": "DefaultTenant",
       "tagOverrideConfig": {}
     },
     "segmentsConfig": {
       "schemaName": "schema-dirrtl",
       "timeColumnName": "UPDT_TMSTM",
       "replication": "1",
       "replicasPerPartition": "1",
       "retentionTimeUnit": null,
       "retentionTimeValue": null,
       "completionConfig": null,
       "crypterClassName": null,
       "peerSegmentDownloadScheme": null
     },
     "tableIndexConfig": {
       "loadMode": "MMAP",
       "invertedIndexColumns": [],
       "createInvertedIndexDuringSegmentGeneration": false,
       "rangeIndexColumns": [],
       "sortedColumn": [],
       "bloomFilterColumns": [],
       "bloomFilterConfigs": null,
       "noDictionaryColumns": [],
       "onHeapDictionaryColumns": [],
       "varLengthDictionaryColumns": [],
       "enableDefaultStarTree": false,
       "starTreeIndexConfigs": null,
       "enableDynamicStarTreeCreation": false,
       "segmentPartitionConfig": null,
       "columnMinMaxValueGeneratorMode": null,
       "aggregateMetrics": false,
       "nullHandlingEnabled": false,
       "streamConfigs": {
         "streamType": "kafka",
         "stream.kafka.topic.name": "mytopic",
         "stream.kafka.broker.list": "kafka-2-client.prod1.gke.kaas-prod-us.gcp.extscloud.com:16000",
         "stream.kafka.consumer.type": "lowlevel",
         "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
         "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
         "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaAvroMessageDecoder",
         "realtime.segment.flush.threshold.rows": "0",
         "realtime.segment.flush.threshold.time": "24h",
         "realtime.segment.flush.threshold.segment.size": "100M",
         "security.protocol": "SSL",
         "sasl.mechanism": "PLAN",
         "ssl.keystore.location": "/opt/pinot/volume_app/dirrtl/keystore.jks",
         "ssl.keystore.password": "**********",
         "ssl.truststore.location": "/opt/pinot/volume_app/dirrtl/truststore.jks",
         "ssl.truststore.password": "**********",
         "ssl.key.password": "****************",
         "ssl.client.auth": "required",
         "stream.kafka.decoder.prop.schema.registry.rest.url": "http://kaas-test-schema-registry-a.company.com/ "
       }
     },
     "metadata": {},
     "ingestionConfig": {
       "filterConfig": null,
       "transformConfigs": []
     },
     "quota": {
       "storage": null,
       "maxQueriesPerSecond": null
     },
     "task": null,
     "routing": {
       "segmentPrunerTypes": null,
       "instanceSelectorType": null
     },
     "query": {
       "timeoutMs": null
     },
     "fieldConfigList": null,
     "upsertConfig": null,
     "tierConfigs": null
   }
   
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1418610755

   here are the controller logs. @navina 
   
   ```
   Caused by: java.nio.file.NoSuchFileException: /opt/pinot/volume_app/BEF0244T/truststore.jks
   at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
   at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
   at java.nio.file.Files.newByteChannel(Files.java:371) ~[?:?]
   at java.nio.file.Files.newByteChannel(Files.java:422) ~[?:?]
   at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420) ~[?:?]
   at java.nio.file.Files.newInputStream(Files.java:156) ~[?:?]
   at org.apache.pinot.shaded.org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:370) ~[pinot-kafka-2.0-0.12.0-shaded.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
   ```
   
   
   <img width="1562" alt="Screenshot 2023-02-06 at 12 35 51 PM" src="https://user-images.githubusercontent.com/6933841/216905977-fdfb1961-acb7-4b74-b9e3-2ef6210a2a0e.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer [pinot]

Posted by "araghavendra16 (via GitHub)" <gi...@apache.org>.
araghavendra16 commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1750726670

   I successfully installed Pinot manually and running all components including Zookeeper and Kafka. When I try to add a Pinot Table, I'm getting "code" : 500, "error": "org.apache.pinot.shaded.org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"} I'm trying to add these tables https://github.com/mneedham/pinot-wiki/tree/main/config Please guide me what's wrong. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer [pinot]

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1934202266

   After one year, I restarted this proof of concept and got stuck at the same point. Can someone help me with this? If possible, can we talk on a call?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1416285841

   @ImAbhishekTomar What version of Pinot are you using? Can you please post your question in the Slack community so we can help you out faster?  https://join.slack.com/t/apache-pinot/shared_invite/zt-1olhndcgi-mnKFUmgj72sLHwc8bbiUlw 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1424652956

   Do you see stack trace for the failure of StreamMessageDecoder creation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1420348237

   **Server Error**
   Caused by: java.lang.RuntimeException: Caught exception while creating StreamMessageDecoder from stream config: StreamConfig{_type='kafka', _topicName='mytoppic-abc-01', _consumerTypes=[LOWLEVEL], _consumerFactoryClassName='org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory', _offsetCriteria='OffsetCriteria{_offsetType=LARGEST, _offsetString='largest'}', _connectionTimeoutMillis=30000, _fetchTimeoutMillis=5000, _idleTimeoutMillis=180000, _flushThresholdRows=0, _flushThresholdTimeMillis=86400000, _flushSegmentDesiredSizeBytes=104857600, _flushAutotuneInitialRows=100000, _decoderClass='org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder', _decoderProperties={}, _groupId='null', _topicConsumptionRateLimit=-1.0, _tableNameWithType='db2BEF0244T_REALTIME}


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1423723610

   https://gist.github.com/ImAbhishekTomar/727952185b2241430500263c04209be9


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [I] Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1989720505

   @zhtaoxiang Can you ask someone to take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1420518089

   pinot-server        |  at org.apache.pinot.shaded.org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299) ~[pinot-kafka-2.0-0.12.0-shaded.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]
   
   pinot-server        | Caused by: java.nio.file.NoSuchFileException: /opt/pinot/app/keystore.jks
   pinot-server        |  at org.apache.pinot.shaded.org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299) ~[pinot-kafka-2.0-0.12.0-shaded.jar:0.12.0-118f5e065cb258c171d97a586183759fbc61e2bf]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1420522317

   ![Screenshot 2023-02-07 at 3 43 28 PM](https://user-images.githubusercontent.com/6933841/217216423-07a1ab52-dd5b-4788-b89b-7425cf3d02ed.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1418713027

   I have updated table name as you suggested. here is the controller logs.
   
   ![Screenshot 2023-02-06 at 2 13 41 PM](https://user-images.githubusercontent.com/6933841/216925220-6fcc2387-21cd-4a14-9f8d-c2a50616bab9.png)
   ![Screenshot 2023-02-06 at 2 14 51 PM](https://user-images.githubusercontent.com/6933841/216925462-09ac796b-62c6-40d4-ad5f-ee86ace77ec3.png)
   
   ```
   2023/02/06 08:42:22.869 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(73e53422_DEFAULT)] Event 73e53422_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__7__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.869 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(73e53422_DEFAULT)] Event 73e53422_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__8__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.869 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(73e53422_DEFAULT)] Event 73e53422_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__9__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.898 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__0__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__1__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__2__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__3__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__4__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__5__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__6__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__7__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__8__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   
   2023/02/06 08:42:22.899 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-PinotCluster-(bd7a73e7_DEFAULT)] Event bd7a73e7_DEFAULT : Unable to find a next state for resource: db2BEF0244T_REALTIME partition: db2BEF0244T__9__0__20230206T0842Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1423725302

   @Efrat19 already check but still same issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1420446401

   <img width="1546" alt="Screenshot 2023-02-07 at 2 48 58 PM" src="https://user-images.githubusercontent.com/6933841/217203166-0130bb1a-5338-4606-bc3b-14b2a8e2b7b3.png">


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] navina commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "navina (via GitHub)" <gi...@apache.org>.
navina commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1425045846

   @Jackie-Jiang this is the stacktrace @ImAbhishekTomar shared:
   https://gist.github.com/ImAbhishekTomar/727952185b2241430500263c04209be9 
   
   the path to the secret files is correct. but not sure why it is not able to access. 
   
   @ImAbhishekTomar : do the server and controller share the same mount point in docker in your setup? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1420354589

   this is my latest table configration.
   
   {
     "REALTIME": {
       "tableName": "db2BEF0244T_REALTIME",
       "tableType": "REALTIME",
       "segmentsConfig": {
         "timeType": "MILLISECONDS",
         "schemaName": "transcript",
         "replicasPerPartition": "1",
         "minimizeDataMovement": false,
         "timeColumnName": "UPDT_TMSTM"
       },
       "tenants": {
         "broker": "DefaultTenant",
         "server": "DefaultTenant"
       },
       "tableIndexConfig": {
         "rangeIndexVersion": 2,
         "autoGeneratedInvertedIndex": false,
         "createInvertedIndexDuringSegmentGeneration": false,
         "loadMode": "MMAP",
         "streamConfigs": {
           "streamType": "kafka",
           "stream.kafka.consumer.type": "lowlevel",
           "stream.kafka.topic.name": "mytopic-001",
           "stream.kafka.broker.list": "kafka.gcp.extscloud.com:16560",
           "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
           "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
           "realtime.segment.flush.threshold.rows": "0",
           "realtime.segment.flush.threshold.time": "24h",
           "realtime.segment.flush.threshold.segment.size": "100M",
           "security.protocol": "SSL",
           "sasl.mechanism": "PLAN",
           "ssl.keystore.location": "/opt/pinot/volume_app/BEF0244T/keystore.jks",
           "ssl.keystore.password": "*****",
           "ssl.truststore.location": "/opt/pinot/volume_app/BEF0244T/truststore.jks",
           "ssl.truststore.password": "****",
           "ssl.key.password": "*****",
           "ssl.client.auth": "***",
           "schema.registry.url": "https://registry.extscloud.com/"
         },
         "enableDefaultStarTree": false,
         "enableDynamicStarTreeCreation": false,
         "aggregateMetrics": false,
         "nullHandlingEnabled": false,
         "optimizeDictionary": false,
         "optimizeDictionaryForMetrics": false,
         "noDictionarySizeRatioThreshold": 0
       },
       "metadata": {
         "customConfigs": {}
       },
       "isDimTable": false
     }
   }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1416289439

   @navina im using this version apachepinot/pinot:0.12.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] ImAbhishekTomar commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "ImAbhishekTomar (via GitHub)" <gi...@apache.org>.
ImAbhishekTomar commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1420449468

   updated config
   
   {
     "tableName": "db2BEF0244T",
     "tableType": "REALTIME",
     "segmentsConfig": {
       "timeType": "MILLISECONDS",
       "schemaName": "transcript",
       "replicasPerPartition": "1",
       "minimizeDataMovement": false,
       "timeColumnName": "UPDT_TMSTM"
     },
     "tenants": {
       "broker": "DefaultTenant",
       "server": "DefaultTenant"
     },
     "tableIndexConfig": {
       "rangeIndexVersion": 2,
       "autoGeneratedInvertedIndex": false,
       "createInvertedIndexDuringSegmentGeneration": false,
       "loadMode": "MMAP",
       "streamConfigs": {
         "streamType": "kafka",
         "stream.kafka.consumer.type": "lowlevel",
         "stream.kafka.topic.name": "topic-name",
         "stream.kafka.broker.list": "gcp.extscloud.com:16560",
         "stream.kafka.consumer.factory.class.name": "org.apache.pinot.core.realtime.impl.kafka2.KafkaConsumerFactory",
         "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.SimpleAvroMessageDecoder",
         "realtime.segment.flush.threshold.rows": "0",
         "realtime.segment.flush.threshold.time": "24h",
         "realtime.segment.flush.threshold.segment.size": "100M",
         "security.protocol": "SSL",
         "sasl.mechanism": "PLAN",
         "ssl.keystore.location": "/opt/pinot/volume_app/BEF0244T/keystore.jks",
         "ssl.keystore.password": "**",
         "ssl.truststore.location": "/opt/pinot/volume_app/BEF0244T/truststore.jks",
         "ssl.truststore.password": "**",
         "ssl.key.password": "**",
         "ssl.client.auth": "required",
         "schema.registry.url": "https://azure-test-schema-registry.extscloud.com/",
         "schema.registry.rest.url": "https://azure-test-schema-registry.extscloud.com/",
         "stream.kafka.decoder.prop.schema": "{\n   \"type\":\"record\",\n   \"name\":\"BEF0245T\",\n   \"namespace\":\"com.optum.cdc.umr\",\n   \"fields\":[\n      {\n         \"name\":\"BEF0245T_DATA\",\n         \"type\":{\n            \"type\":\"record\",\n            \"name\":\"BEF0245T_DATA\",\n            \"fields\":[\n               {\n                  \"name\":\"ASSGN_SUBSCR_NBR\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"bytes\",\n                        \"precision\":8,\n                        \"scale\":0,\n                        \"logicalType\":\"decimal\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"INDIV_SEQ_NBR\",\n                  \"type\":[\n                     \"null\",\n                     \"int\"\n                  ],\n                  \"default\":null\n               },\n   
             {\n                  \"name\":\"SRC_CD\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"LN_INS_CD\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"EFF_CHG_DT\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"int\",\n                        \"logicalType\":\"date\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"POL_NBR\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \
 "default\":null\n               },\n               {\n                  \"name\":\"GRP_STAT_CL_CD\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"POL_COV_APP_CD\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"EFF_INS_DT\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"int\",\n                        \"logicalType\":\"date\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"EFF_STTS_CD\",\n                  \"type\":[\n                     \"null\",\n                  
    \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"EXPIR_STTS_CD\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"EXPIR_INS_DT\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"int\",\n                        \"logicalType\":\"date\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"SCHED_CLASS_CODE\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"HEALTH_ORGN_ID_CD\",\n               
    \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"TRANS_EFF_INS_DATE\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"int\",\n                        \"logicalType\":\"date\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"CNSMR_CNCT_IND\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"CLM_LOB_CD\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n
                   \"name\":\"INFO_DVD_IND\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"COV_ADVSR_IND\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"IHR_IND\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"HRA_IND\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"PERS_CARE_NOTE_IND\",\n          
         \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"CARE_TRGT_BNFT_IND\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"EFF_SCHED_BENEFIT_DT\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"int\",\n                        \"logicalType\":\"date\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"CREAT_USER_ID\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n    
            {\n                  \"name\":\"CREAT_TMSTM\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"long\",\n                        \"logicalType\":\"timestamp-micros\"\n                     }\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"UPDT_USER_ID\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"UPDT_TMSTM\",\n                  \"type\":[\n                     \"null\",\n                     {\n                        \"type\":\"long\",\n                        \"logicalType\":\"timestamp-micros\"\n                     }\n                  ],\n                  \"default\":null\n               }\n            ]\n         }\n      },\n      {\n         
 \"name\":\"beforeData\",\n         \"type\":[\n            \"null\",\n            \"BEF0245T_DATA\"\n         ],\n         \"default\":null\n      },\n      {\n         \"name\":\"headers\",\n         \"type\":{\n            \"type\":\"record\",\n            \"name\":\"Headers\",\n            \"fields\":[\n               {\n                  \"name\":\"operation\",\n                  \"type\":{\n                     \"type\":\"enum\",\n                     \"name\":\"operation\",\n                     \"symbols\":[\n                        \"INSERT\",\n                        \"UPDATE\",\n                        \"DELETE\",\n                        \"REFRESH\"\n                     ]\n                  }\n               },\n               {\n                  \"name\":\"changeSequence\",\n                  \"type\":\"string\"\n               },\n               {\n                  \"name\":\"timestamp\",\n                  \"type\":\"string\"\n               },\n               {\n  
                 \"name\":\"streamPosition\",\n                  \"type\":\"string\"\n               },\n               {\n                  \"name\":\"transactionId\",\n                  \"type\":\"string\"\n               },\n               {\n                  \"name\":\"changeMask\",\n                  \"type\":[\n                     \"null\",\n                     \"bytes\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"columnMask\",\n                  \"type\":[\n                     \"null\",\n                     \"bytes\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"externalSchemaId\",\n                  \"type\":[\n                     \"null\",\n                     \"string\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"transactionEventC
 ounter\",\n                  \"type\":[\n                     \"null\",\n                     \"long\"\n                  ],\n                  \"default\":null\n               },\n               {\n                  \"name\":\"transactionLastEvent\",\n                  \"type\":[\n                     \"null\",\n                     \"boolean\"\n                  ],\n                  \"default\":null\n               }\n            ]\n         }\n      }\n   ]\n}"
       },
       "enableDefaultStarTree": false,
       "enableDynamicStarTreeCreation": false,
       "aggregateMetrics": false,
       "nullHandlingEnabled": false,
       "optimizeDictionary": false,
       "optimizeDictionaryForMetrics": false,
       "noDictionarySizeRatioThreshold": 0
     },
     "metadata": {
       "customConfigs": {}
     },
     "isDimTable": false
   }
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Efrat19 commented on issue #10225: Realtime Ingestion Using Kafka with SSL | Error: Failed to construct kafka consumer

Posted by "Efrat19 (via GitHub)" <gi...@apache.org>.
Efrat19 commented on issue #10225:
URL: https://github.com/apache/pinot/issues/10225#issuecomment-1422956364

   Have you tried fixing your table config`"sasl.mechanism": "PLAN",` to `"sasl.mechanism": "PLAIN",`?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org