You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "piby180 (via GitHub)" <gi...@apache.org> on 2025/07/16 12:17:47 UTC
[I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
piby180 opened a new issue, #16356:
URL: https://github.com/apache/pinot/issues/16356
Hi all,
I am trying to enable cursor support for our Pinot cluster. From the docs [here](https://docs.pinot.apache.org/users/user-guide-query/query-using-cursors), I can see we can either store the query results in local storage or S3.
For pinot helm chart, we do not have any option to deploy persistent volume for broker pods, so we use node's ephermal storage for broker pods.
It could be that the local storage is not sufficient in size so I wanted to use S3. However it is not working with S3. It is storing the results at
`/tmp/broker/responseStore/data` and `/tmp/broker/responseStore/temp`
My broker configuration is
```
pinot.query.runner.port=8442
pinot.broker.enable.query.cancellation=true
pinot.broker.adaptive.server.selector.enable.stats.collection=true
pinot.broker.adaptive.server.selector.type=HYBRID
# Turn on resource usage tracking in statistics framework.
# Configuration has to be set in broker and server config files.
pinot.broker.instance.enableThreadAllocatedBytesMeasurement=true
pinot.server.instance.enableThreadAllocatedBytesMeasurement=true
pinot.broker.instance.enableThreadCpuTimeMeasurement=true
pinot.server.instance.enableThreadCpuTimeMeasurement=true
pinot.query.scheduler.accounting.factory.name=org.apache.pinot.core.accounting.PerQueryCPUMemAccountantFactory
pinot.query.scheduler.accounting.enable.thread.memory.sampling=true
# Set in broker and server
pinot.query.scheduler.accounting.oom.enable.killing.query=true
pinot.query.scheduler.accounting.query.killed.metric.enabled=true
pinot.query.scheduler.accounting.enable.thread.cpu.sampling=true
pinot.query.scheduler.accounting.oom.enable.killing.query=true
pinot.query.scheduler.accounting.publishing.jvm.heap.usage=true
# Cursor Response store
pinot.broker.cursor.result.store.type=file
pinot.broker.storage.factory.s3.region=${AWS_S3_REGION}
pinot.broker.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.broker.cursor.result.store.file.temp.dir=s3://mybucket/cursors/temp
pinot.broker.cursor.result.store.file.data.dir=s3://mybucket/cursors/data
pinot.broker.cursor.response.store.expiration=1h
pinot.broker.cursor.fetch.rows=1000
controller.cluster.response.store.cleaner.frequencyPeriod=1h
controller.cluster.response.store.cleaner.initialDelay=60
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on issue #16356:
URL: https://github.com/apache/pinot/issues/16356#issuecomment-3105612823
Doc fix is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #16356:
URL: https://github.com/apache/pinot/issues/16356#issuecomment-3097495437
cc @vrajat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat closed issue #16356: Cursor responseStore is getting stored in /tmp instead of S3
URL: https://github.com/apache/pinot/issues/16356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
Posted by "piby180 (via GitHub)" <gi...@apache.org>.
piby180 commented on issue #16356:
URL: https://github.com/apache/pinot/issues/16356#issuecomment-3083039477
Turned out there are typos in the documentation
<img width="893" height="376" alt="Image" src="https://github.com/user-attachments/assets/7b56320f-97cd-4326-8020-a27ca8951846" />
Using response instead of result worked!
```
pinot.broker.cursor.response.store.type=file
pinot.broker.storage.factory.s3.region=${AWS_S3_REGION}
pinot.broker.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.broker.cursor.response.store.file.temp.dir=s3://mybucket/cursors/temp
pinot.broker.cursor.response.store.file.data.dir=s3://mybucket/cursors/data
pinot.broker.cursor.response.store.expiration=1h
pinot.broker.cursor.fetch.rows=1000
controller.cluster.response.store.cleaner.frequencyPeriod=1h
controller.cluster.response.store.cleaner.initialDelay=60
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
Posted by "piby180 (via GitHub)" <gi...@apache.org>.
piby180 commented on issue #16356:
URL: https://github.com/apache/pinot/issues/16356#issuecomment-3078553627
I tried to use local storage but it is not working as well.
```
pinot.broker.cursor.result.store.file.temp.dir=/opt/pinot/data/cursors/temp
pinot.broker.cursor.result.store.file.data.dir=/opt/pinot/data/cursors/data
```
It is always using default values
`/tmp/broker/responseStore/data` and `/tmp/broker/responseStore/temp`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Cursor responseStore is getting stored in /tmp instead of S3 [pinot]
Posted by "vrajat (via GitHub)" <gi...@apache.org>.
vrajat commented on issue #16356:
URL: https://github.com/apache/pinot/issues/16356#issuecomment-3101153297
The example is wrong. Thanks for pointing it out. I've fixed the docs in this PR: https://app.gitbook.com/o/-LtRX9NwSr7Ga7zA4piL/s/-LtH6nl58DdnZnelPdTc-887967055/~/changes/2206/users/user-guide-query/query-using-cursors
Also the temp dir should be local file system. `PinotFS` needs a local path to stage the files before transferring it to S3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org