You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "MonsterChenzhuo (via GitHub)" <gi...@apache.org> on 2024/08/26 13:50:22 UTC

[I] [Feature] Introduce New Metrics for Compaction [paimon]

MonsterChenzhuo opened a new issue, #4063:
URL: https://github.com/apache/paimon/issues/4063

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   During recent usage, we have observed that DelVector's operations are dependent on compaction, making it a critical factor for data visibility. However, the current metrics related to compaction are still limited. To address this, I propose introducing the following new metrics to enrich our compaction-related metrics:
   
   Standard Deviation of Compaction Time: Measures the variability in compaction times to help identify any extreme cases (e.g., some tasks running particularly slow).
   
   Ratio of Compacted Input to Output Files: Helps to understand the effectiveness of compaction.
   
   Change Rate in Total File Size Before and After Compaction: Assesses the impact of compaction on disk space, especially when compressing many small files into fewer large files.
   
   Memory Usage: Measures memory usage (such as heap memory) at the start and end of a compaction task to calculate changes in memory usage during the compaction process.
   
   Number of Pending Compaction Tasks: Monitors the length of the task queue submitted to the executor to understand if there are task congestion issues.
   
   Compaction Failure Rate: Records the proportion of compaction tasks that encounter errors, which is important for monitoring the stability and reliability of compaction tasks.
   
   CPU and I/O Utilization: Combines system monitoring tools to record CPU and I/O utilization during the execution of compaction tasks. This helps to identify the impact of compaction tasks on system resources.
   
   Average Compaction Time: Calculates the average execution time of compaction tasks over a period.
   
   Minimum Compaction Time: Records the shortest compaction time observed.
   
   Maximum Compaction Time: Records the longest compaction time observed.
   
   I believe these metrics will provide us with a more comprehensive understanding of compaction performance and its impact on our system.
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Introduce New Metrics for Compaction [paimon]

Posted by "MonsterChenzhuo (via GitHub)" <gi...@apache.org>.
MonsterChenzhuo commented on issue #4063:
URL: https://github.com/apache/paimon/issues/4063#issuecomment-2310273458

   cc @JingsongLi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Introduce New Metrics for Compaction [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on issue #4063:
URL: https://github.com/apache/paimon/issues/4063#issuecomment-2311623626

   Thanks @MonsterChenzhuo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Introduce New Metrics for Compaction [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed issue #4063: [Feature]  Introduce New Metrics for Compaction
URL: https://github.com/apache/paimon/issues/4063


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org