You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "zhoujinsong (via GitHub)" <gi...@apache.org> on 2024/08/15 09:07:10 UTC
[I] [Feature]: Add table summary metrics [amoro]
zhoujinsong opened a new issue, #3104:
URL: https://github.com/apache/amoro/issues/3104
### Description
Add table summary metrics for each table to help users better understand the detail of tables.
### Use case/motivation
Amoro has currently implemented the Metric System to provide various metric information externally.
However, it is also crucial to have metrics for important aspects such as data size, file count, and record numbers related to tables. By adding metrics for these aspects, we aim to assist users in gaining a better understanding of the various situations on the tables.
### Describe the solution
We should define proper table summary metrics first.
Then AMS will refresh the table runtime every 3 minutes(by default), it can update the table summary metrics in this stage.
### Subtasks
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2291120477
> The meanings of "table_summary_total_files" and "table_summary_data_files" are whether the quantities are in the latest Snapshot or the quantities of all the files existing on HDFS.
IMO, they refer to the summary of the current snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2291041445
Thanks for driving this, the summary is very helpful for users to understand the underlying things, apart from the table-level metric, 1) Could we add the metric for the tasks -- we can give more information from the task-level metrics and optimize the planning process; 2) could we add a metric for the committing process
We've added some metrics above about this in our inner version and can contribute them if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2290960421
Here are some table summary metrics for Iceberg tables:
| Metric Name | Type | Tags | Description |
|-------------------------------------------|--------|------------------|--------------------------------------------------|
| table_summary_total_files | Gauge | catalog, database, table | Total number of files in the table |
| table_summary_data_files | Gauge | catalog, database, table | Number of data files in the table |
| table_summary_equality_delete_files | Gauge | catalog, database, table | Number of equality delete files in the table |
| table_summary_position_delete_files | Gauge | catalog, database, table | Number of position delete files in the table |
| table_summary_total_files_size | Gauge | catalog, database, table | Total size of files in the table |
| table_summary_data_files_size | Gauge | catalog, database, table | Size of data files in the table |
| table_summary_equality_delete_files_size | Gauge | catalog, database, table | Size of equality delete files in the table |
| table_summary_position_delete_files_size | Gauge | catalog, database, table | Size of position delete files in the table |
| table_summary_total_records | Gauge | catalog, database, table | Total records in the table |
| table_summary_data_files_records | Gauge | catalog, database, table | Records of data files in the tablep |
| table_summary_equality_delete_files_records | Gauge | catalog, database, table | Records of equality delete files in the table |
| table_summary_position_delete_files_records | Gauge | catalog, database, table | Records of position delete files in the table |
| table_summary_snapshots | Gauge | catalog, database, table | Number of snapshots in the table |
Please feel free to share your idea about table summary metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2291000652
The meanings of "table_summary_total_files" and "table_summary_data_files" are whether the quantities are in the latest Snapshot or the quantities of all the files existing on HDFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong closed issue #3104: [Feature]: Add table summary metrics
URL: https://github.com/apache/amoro/issues/3104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2291138065
> 1) Could we add the metric for the tasks -- we can give more information from the task-level metrics and optimize the planning process; 2) could we add a metric for the committing process
Great suggestion for plan/commit metrics.
Adding metrics related to queries and commits is indeed very valuable. However, it may depend on Amoro to implement the Iceberg [MetricsReporter](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/metrics/MetricsReporter.java) for collecting plan/commit information from different engines. This is indeed a planned feature for Amoro in the future, but it may not be included in this current feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "hhippodnsla (via GitHub)" <gi...@apache.org>.
hhippodnsla commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2294752688
In my opinion: we can use the concepts from DW such as identifying the dimension and facts.
e.g.
the dimensions such as:
* which table type? pure iceberg, mixed iceberg, mixed hive
* which data type? data or metadata
* is it in use? files expired or in use?
* which file type for data/metadata? data/eq-del/pos-del/manifest/manifest-list and so on...
* has partition and which parition?
the facts such as:
* number of files
* total size of files
* 90% file size
* median file size
* max file size
Here are our exp sience 0.4:
(let me bring it from my working computer on Monday....)
Furthermore, the idea to reference metrics used in iceberg like czy006 said might be good idea, thus we can get more detail view from the data inside the table, but need more consideration when the table format is mixed hive.
And we need to consider the capability of promethues reporter, since we've stepped on the pit here... (large number of metrics in single page will cause the performance issue)
On the other hand, I've totally agree with klion26, so we can have a better understanding of what's the situation when self-optimizing working. (e.g. add to OptimizerGroupMetric?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2291117795
> Whether to consider making the metrics more generalized to support different table formats?
I would be glad to make the metric name more general if it could be used for multiple formats, like table_summary_total_files or table_summary_data_files.
However, I also agree with adding specific metrics for different table formats, such as table_summary_equality_delete_files and table_summary_position_delete_files. We should allow different table formats to have different metric items.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "nqvuong1998 (via GitHub)" <gi...@apache.org>.
nqvuong1998 commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2292665426
Hi @zhoujinsong , We can use [this AWS Solution](https://aws.amazon.com/vi/blogs/big-data/monitoring-apache-iceberg-metadata-layer-using-aws-lambda-aws-glue-and-aws-cloudwatch/) as a reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2291001544
Whether to consider making the metrics more generalized to support different table formats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature]: Add table summary metrics [amoro]
Posted by "czy006 (via GitHub)" <gi...@apache.org>.
czy006 commented on issue #3104:
URL: https://github.com/apache/amoro/issues/3104#issuecomment-2293067619
> Hi @zhoujinsong , We can use [this AWS Solution](https://aws.amazon.com/vi/blogs/big-data/monitoring-apache-iceberg-metadata-layer-using-aws-lambda-aws-glue-and-aws-cloudwatch/) as a reference.
Thanks for your feedback and found,We can refer to this part of the indicators to achieve
https://github.com/aws-samples/monitoring-apache-iceberg-table-metadata-layer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org