You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "chrishanKH (via GitHub)" <gi...@apache.org> on 2023/04/20 16:15:14 UTC

[GitHub] [pinot] chrishanKH opened a new issue, #10654: PinotFS for scheme: s3 has not been initialized

chrishanKH opened a new issue, #10654:
URL: https://github.com/apache/pinot/issues/10654

   I installed Pinot 0.12.0 on my AWS EKS cluster via `helm` (command below) and attempting to use S3 for deep storage. I ran an ingestion job via Spark, which completes successfully with the outputs in the right locations in S3, but Pinot reports `BAD` segments. 
   
   Upon clicking on a segment's debug details through the Pinot UI, it shows that `externalView` is `ERROR`. I couldn't find too much information on `externalView` in the Pinot docs:
   ```
   {
     "segmentName": "$tableName_OFFLINE_100",
     "serverState": {
       "Server_pinot-server-0.pinot-server-headless.$namespace.svc.cluster.local_8098": {
         "idealState": "ONLINE",
         "externalView": "ERROR",
         "segmentSize": "0 bytes",
         "consumerInfo": null,
         "errorInfo": null
       }
     }
   }
   ```
   
   Upon inspecting the logs with `kubectl logs pinot-server-0 -n $myNamespace` I noticed the error message `java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized`. Full exception here:
   
   ```
   Caught exception while fetching segment from: s3://$path_to_segments/$tableName_OFFLINE_296.tar.gz to: /var/pinot/server/data/index/$tabletName_OFFLINE/tmp/tmp-$tableName_OFFLINE_296-518442be-5f6b-47ec-a672-e70ab6bd5bd2/$tableName_OFFLINE_296.tar.gz
   java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized
   	at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.spi.filesystem.PinotFSFactory.create(PinotFSFactory.java:78) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher.fetchSegmentToLocalWithoutRetry(PinotFSSegmentFetcher.java:31) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.lambda$fetchSegmentToLocal$0(BaseSegmentFetcher.java:75) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:58) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:73) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:157) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:151) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:198) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:186) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:565) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:528) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:512) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:472) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:482) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[?:?]
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
   	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   	at java.lang.Thread.run(Thread.java:829) [?:?]
   Attempts exceeded when downloading segment: $tableName_OFFLINE_296 for table: $tableName from: s3://$pathToSegments/segments/$tableName_OFFLINE_296.tar.gz to: /var/pinot/server/data/index/$tableName_OFFLINE/tmp/tmp-$tableName_OFFLINE_296-518442be-5f6b-47ec-a672-e70ab6bd5bd2/$tableName_v2_OFFLINE_296.tar.gz
   Caught exception in state transition from OFFLINE -> ONLINE for resource: $tableName_OFFLINE, partition: $tableName_OFFLINE_296
   org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
   	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:73) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:157) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:151) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:198) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:186) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:565) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:528) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:512) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:472) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:482) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[?:?]
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
   	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   	at java.lang.Thread.run(Thread.java:829) [?:?]
   Exception while executing a state transition task $tableName_OFFLINE_296
   java.lang.reflect.InvocationTargetException: null
   	at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) ~[?:?]
   	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
   	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:350) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:278) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
   	at java.lang.Thread.run(Thread.java:829) [?:?]
   Caused by: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
   	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:73) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocalInternal(SegmentFetcherFactory.java:157) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:151) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocalInternal(SegmentFetcherFactory.java:198) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:186) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadAndDecrypt(BaseTableDataManager.java:565) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegmentFromDeepStore(BaseTableDataManager.java:528) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.downloadSegment(BaseTableDataManager.java:512) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.core.data.manager.BaseTableDataManager.addOrReplaceSegment(BaseTableDataManager.java:472) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addOrReplaceSegment(HelixInstanceDataManager.java:482) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:166) ~[pinot-all-0.13.0-SNAPSHOT-jar-with-dependencies.jar:0.13.0-SNAPSHOT-f2afe21b9962a0d676a7887edc999bcbee6c441d]
   	... 11 more
   Default rollback method invoked on error. Error Code: ERROR
   Message execution failed. msgId: ddba0761-acd8-41d3-b00e-401b958b045c, errorMsg: java.lang.reflect.InvocationTargetException
   Delete message ddba0761-acd8-41d3-b00e-401b958b045c from zk!
   message finished: ddba0761-acd8-41d3-b00e-401b958b045c, took 2428
   ```
    
   I read the documentation [here](https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot) and included the config via the `helm install command`:
   
   ```
   helm upgrade pinot pinot/pinot -n $namespace \
   --set cluster.name=pinot \
   --set server.replicaCount=2 \
   --set zookeeper.persistence.size=500G \
   --set input.fs.className=org.apache.pinot.plugin.filesystem.S3PinotFS \
   --set input.fs.prop.accessKey=$aws_access_key \
   --set input.fs.prop.secretKey=$aws_secret_key \
   --set input.fs.prop.region=$aws_region \
   --set pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS \
   --set pinot.controller.storage.factory.s3.region=$aws_region \
   --set controller.data.dir=$s3_path \
   --set controller.local.temp.dir=/tmp/pinot-dev-output \
   --set controller.enable.split.commit=true \
   --set pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher \
   --set pinot.server.instance.enable.split.commit=true \
   --set pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS \
   --set pinot.server.storage.factory.s3.region=$aws_region \
   --set pinot.server.storage.factory.s3.httpclient.maxConnections=50 \
   --set pinot.server.storage.factory.s3.httpclient.socketTimeout=30s \
   --set pinot.server.storage.factory.s3.httpclient.connectionTimeout=2s \
   --set pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s \
   --set pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=10s \
   --set pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher \
   --set pinot.minion.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS \
   --set pinot.minion.storage.factory.s3.region=$aws_region \
   --set pinot.minion.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher \
   --set pinot.controller.segment.fetcher.protocols="file\,http\,s3" \
   --set pinot.minion.segment.fetcher.protocols="file\,http\,s3" \
   --set pinot.server.segment.fetcher.protocols="file\,http\,s3"
   ```
   
   Any guidance? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] chrishanKH commented on issue #10654: PinotFS for scheme: s3 has not been initialized

Posted by "chrishanKH (via GitHub)" <gi...@apache.org>.
chrishanKH commented on issue #10654:
URL: https://github.com/apache/pinot/issues/10654#issuecomment-1521750495

   The fix is to pass the extra configs as a YAML file and include it in `extra.configs` so they are populated in [this](https://github.com/apache/pinot/blob/b0284994393c09d79917e529592b8ac10258e9ea/kubernetes/helm/pinot/templates/controller/configmap.yaml#L34) block. An example, working config looks like:
   
   ```
   controller:
     enable:
       split:
         commit: true
     data:
       dir: s3://data-pinot-deep-storage
   
     extra:
       configs: |-
         pinot.set.instance.id.to.hostname=true
         controller.task.scheduler.enabled=true
         controller.local.temp.dir=/tmp/pinot-tmp-data/
         pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
         pinot.controller.storage.factory.s3.region=ap-south-1
         pinot.controller.segment.fetcher.protocols=file,http,s3
         pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
   
   server:
     extra:
       configs: |-
         pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
         pinot.server.storage.factory.s3.region=ap-south-1
         pinot.server.segment.fetcher.protocols=file,http,s3
         pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
         pinot.server.instance.enable.split.commit=true
         pinot.server.storage.factory.s3.httpclient.maxConnections=50
         pinot.server.storage.factory.s3.httpclient.socketTimeout=30s
         pinot.server.storage.factory.s3.httpclient.connectionTimeout=2s
         pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s
         pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=10s
   
   minion:
     extra:
       configs: |-
         pinot.minion.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
         pinot.minion.storage.factory.s3.region=ap-south-1
         pinot.minion.segment.fetcher.protocols=file,http,s3
         pinot.minion.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
   ```
   Put this into a file called values-s3-deepstore.yaml and then pass it to help install:
   
   `helm install -f values.yaml -f values-s3-deepstore.yaml pinot pinot/pinot -n namespace'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] chrishanKH commented on issue #10654: PinotFS for scheme: s3 has not been initialized

Posted by "chrishanKH (via GitHub)" <gi...@apache.org>.
chrishanKH commented on issue #10654:
URL: https://github.com/apache/pinot/issues/10654#issuecomment-1520290411

   I believe the [config map](https://github.com/apache/pinot/blob/master/kubernetes/helm/pinot/templates/controller/configmap.yaml) should be updated to include the S3 related configs so they are written to `pinot-controller.conf`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] chrishanKH closed issue #10654: PinotFS for scheme: s3 has not been initialized

Posted by "chrishanKH (via GitHub)" <gi...@apache.org>.
chrishanKH closed issue #10654: PinotFS for scheme: s3 has not been initialized
URL: https://github.com/apache/pinot/issues/10654


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org