You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2023/11/16 20:10:02 UTC
[I] Column footprints and capabilities in segmentMetadata query (druid)
gianm opened a new issue, #15386:
URL: https://github.com/apache/druid/issues/15386
For understanding and optimizing data footprint, it's valuable to know the footprint (in bytes) and capabilities (dictionary encoded, has index, etc) of each column on a per-segment basis and also on an aggregated basis across an entire datasource.
Today, the main way people get footprint information is by reading `meta.smoosh` files, which are a CSV descriptor that is part of the physical segment package, and which looks like the following. Each row (other than the first) represents a column part, and is four fields: column name, column part number, start offset, and end offset. The footprint in bytes of a column is the difference between the fourth and third fields.
```
v1,2147483647,1
__time,0,0,5466
channel,0,5727,8512
isRobot,0,5466,5727
```
Instead of needing to dig through `meta.smoosh` files, we'd like users to be able to use `segmentMetadata` queries (https://druid.apache.org/docs/latest/querying/segmentmetadataquery/) to retrieve the column footprints. For `merge: false`, users would get column footprints from each segment individually. For `merge: true`, users would get total footprint for each column across all segments.
On the capabilities side, certain capabilities, like `hasMultipleValues`, are exposed in `segmentMetadata` queries today. However, they aren't all exposed. We should expose all of them, so people can see which columns have dictionaries, which dictionaries are sorted, etc.
As a follow-up to this feature, we could add column footprint and capability to system tables in SQL. System tables are built on top of `segmentMetadata` queries, so adding the fields to `segmentMetadata` is the first step. (Note that adding to `segmentMetadata` is also valuable by itself, since users can issue `segmentMetadata` queries directly. The SQL would be merely a convenience.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Column footprints and capabilities in segmentMetadata query (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15386:
URL: https://github.com/apache/druid/issues/15386#issuecomment-2305953428
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Column footprints and capabilities in segmentMetadata query (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15386:
URL: https://github.com/apache/druid/issues/15386#issuecomment-2923782356
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Column footprints and capabilities in segmentMetadata query (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on issue #15386:
URL: https://github.com/apache/druid/issues/15386#issuecomment-2306024175
i think this would be satisfied by #16132 once I get back to it, hopefully not too much time from now, but i've a few things I'm planning on doing first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Column footprints and capabilities in segmentMetadata query (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15386:
URL: https://github.com/apache/druid/issues/15386#issuecomment-4035368628
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Column footprints and capabilities in segmentMetadata query (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15386:
URL: https://github.com/apache/druid/issues/15386#issuecomment-4203052530
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Column footprints and capabilities in segmentMetadata query (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15386: Column footprints and capabilities in segmentMetadata query
URL: https://github.com/apache/druid/issues/15386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org