You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Vasu7052 (via GitHub)" <gi...@apache.org> on 2025/01/08 12:08:27 UTC
[I] Query: How to get the size occupied by indexes [pinot]
Vasu7052 opened a new issue, #14779:
URL: https://github.com/apache/pinot/issues/14779
Hi Team,
We have an events table in our Apache Pinot Cluster and around 1M rows present in it.
We have applied several indexes on this events table so we wanted to know how much size on disk/memory all the indexes are using so that we can tweak them accordingly.
```
# Pinot Server Data Directory
pinot.server.instance.dataDir=/mnt/data/apache-pinot-data/
# Pinot Server Temporary Segment Tar Directory
pinot.server.instance.segmentTarDir=/mnt/data/apache-pinot-data/segmentTar
```
Apache Pinot version: 1.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query: How to get the size occupied by indexes [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14779:
URL: https://github.com/apache/pinot/issues/14779#issuecomment-4384247724
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query: How to get the size occupied by indexes [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14779:
URL: https://github.com/apache/pinot/issues/14779#issuecomment-4627181267
This issue has been closed because it has been inactive for 180 days. If you believe this issue is still relevant, please feel free to reopen it or create a new issue. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query: How to get the size occupied by indexes [pinot]
Posted by "Vasu7052 (via GitHub)" <gi...@apache.org>.
Vasu7052 commented on issue #14779:
URL: https://github.com/apache/pinot/issues/14779#issuecomment-2579347185
Here are the APIs that i tried:
1.
```
curl -X 'GET' \
'http://localhost:9000/segments/events_v1/metadata?type=OFFLINE' \
-H 'accept: application/json'
output:
{
"events_v1_OFFLINE_1733600231086_1737667483443_0": {
"segmentName": "events_v1_OFFLINE_1733600231086_1737667483443_0",
"schemaName": null,
"crc": 992603588,
"creationTimeMillis": 1736333325493,
"creationTimeReadable": "2025-01-08T10:48:45:493 UTC",
"timeColumn": "time",
"timeUnit": "MILLISECONDS",
"timeGranularitySec": 0,
"startTimeMillis": 1733600231086,
"startTimeReadable": "2024-12-07T19:37:11.086Z",
"endTimeMillis": 1737667483443,
"endTimeReadable": "2025-01-23T21:24:43.443Z",
"segmentVersion": "v3",
"creatorName": null,
"totalDocs": 500109,
"custom": {
"input.data.file.uri": "file:/mnt/data/...output.csv"
},
"startOffset": null,
"endOffset": null,
"columns": [],
"indexes": {},
"star-tree-index": null
}
}
```
2.
```
curl -X 'GET' \
'http://localhost:9000/segments/events_v1/events_v1_OFFLINE_1733600231086_1737667483443_0/metadata' \
-H 'accept: application/json'
{
"segment.start.time": "1733600231086",
"segment.time.unit": "MILLISECONDS",
"segment.size.in.bytes": "20886469",
"segment.end.time": "1737667483443",
"segment.total.docs": "500109",
"segment.creation.time": "1736333325493",
"segment.push.time": "1736333329509",
"segment.end.time.raw": "1737667483443",
"segment.start.time.raw": "1733600231086",
"segment.index.version": "v3",
"custom.map": "{\"input.data.file.uri\":\"file:/mnt/data/.../output.csv\"}",
"segment.crc": "992603588",
"segment.download.url": "http://controller:9000/segments/events_v1/events_v1_OFFLINE_1733600231086_1737667483443_0"
}
```
None of these APIs are giving index information:
Here's the first few lines of index_map file
```
Platform.dictionary.startOffset = 0
Platform.dictionary.size = 29
Platform.forward_index.startOffset = 29
Platform.forward_index.size = 125036
account_id.dictionary.startOffset = 125065
account_id.dictionary.size = 12
account_id.forward_index.startOffset = 125077
account_id.forward_index.size = 16
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query: How to get the size occupied by indexes [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14779: Query: How to get the size occupied by indexes
URL: https://github.com/apache/pinot/issues/14779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query: How to get the size occupied by indexes [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #14779:
URL: https://github.com/apache/pinot/issues/14779#issuecomment-2579320565
There are 2 ways to get that info:
1. Use the segment metadata controller rest API, which should be able to return the per index type size info
2. Open the local segment on server, and the size info is stored in `index_map` file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Query: How to get the size occupied by indexes [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #14779:
URL: https://github.com/apache/pinot/issues/14779#issuecomment-2590325901
@Vasu7052 You want to put `columns` query parameter. Put `*` if you want to read index info for all columns.
From `index_map`, you also can tell the indexes and their sizes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org