[GitHub] [hudi] yesemsanthoshkumar commented on a diff in pull request #6350: [HUDI-4610] Document list of metrics emitted by Hudi

[GitBox <gi...@apache.org>]

yesemsanthoshkumar commented on code in PR #6350:
URL: https://github.com/apache/hudi/pull/6350#discussion_r971631023


##########
website/docs/metrics.md:
##########
@@ -201,3 +201,32 @@ These `HoodieMetrics` can then be plotted on a standard tool like grafana. Below
 <figure>
     <img className="docimage" src={require("/assets/images/hudi_commit_duration.png").default} alt="hudi_commit_duration.png"  />
 </figure>
+
+## List of metrics:
+
+The below metrics are available in all timeline operations that involves a commit such as deltacommit, compaction, clustering and rollback.
+
+Name  |  Description
+--- | ---
+commitFreshnessInMs | Milliseconds from the commit end time and the maximum event time of the incoming records
+commitLatencyInMs | Milliseconds from the commit end time and the minimum event time of incoming records
+commitTime  | Time of commit in epoch milliseconds
+duration  | Total time taken for the commit/rollback in milliseconds
+numFilesDeleted | Number of files deleted during a clean/rollback
+numFilesFinalized | Number of files finalized in a write
+totalBytesWritten | Bytes written in a HoodieCommit
+totalCompactedRecordsUpdated  | Number of records updated in a compaction operation
+totalCreateTime | Time taken for file creation during a Hoodie Insert operation
+totalFilesInsert  | Number of newly written files in a HoodieCommit
+totalFilesUpdate  | Number of files updated in a HoodieCommit
+totalInsertRecordsWritten | Number of records inserted or converted to updates(for small file handling) in a HoodieCommit

Review Comment:
   @yihua This metric is emitted during both clustering and compaction op inside SparkRDDWriteClient.
   https://github.com/apache/hudi/blob/71b81740589c9c3ec1ee1bd4910342fde0656790/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java#L320 for compaction reference.
   
   Moreover, the https://github.com/apache/hudi/blob/4f7ea8c79a9d13accf72d094296993c588d87beb/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/HoodieMetrics.java#L162 updatecommitmetrics function calls fetchtotalInsertRecordsWritten defined as https://github.com/apache/hudi/blob/4f7ea8c79a9d13accf72d094296993c588d87beb/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieCommitMetadata.java#L278
   This inturn calls numInserts on the writestat.
   
   numInserts is documented as https://github.com/apache/hudi/blob/8ab6f17149e7edbdc80351bdf0d2e5c4c4a33908/hudi-common/src/main/java/org/apache/hudi/common/model/HoodieWriteStat.java#L69
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org