You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org> on 2025/08/29 00:55:29 UTC
[PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
vamshikrishnakyatham opened a new pull request, #13804:
URL: https://github.com/apache/hudi/pull/13804
### Change Logs
This PR introduces a new comprehensive `show_timeline` procedure for Hudi Spark SQL that provides detailed timeline information for all table operations. The procedure displays timeline instants including commits, deltacommits, compactions, clustering, cleaning, and rollback operations etc with support for both active and archived timelines and completed, pending state instants
Key features implemented:
- **Comprehensive timeline view**: Shows all timeline instants with detailed metadata including state transitions (REQUESTED, INFLIGHT, COMPLETED)
- **Time-based filtering**: Support for `startTime` and `endTime` parameters to filter results within specific time ranges
- **Archive timeline support**: `showArchived` parameter to include archived timeline data for complete historical view
- **Generic SQL filtering**: `filter` parameter supporting SQL expressions for flexible result filtering
- **Rich metadata output**: Includes formatted timestamps, rollback information, and table type details
The procedure replaces multiple fragmented timeline-related procedures with a single unified interface that provides both pending and completed instant information with partition-specific metadata support.
### Impact
**Public API Changes:**
- **New procedure**: `show_timeline` with comprehensive parameter support (table, path, limit, showArchived, filter, startTime, endTime)
- **Enhanced schema**: 8-column output including instant_time, action, state, requested_time, inflight_time, completed_time, timeline_type, rollback_info
- **Backward compatibility**: Existing timeline procedures remain functional (deprecated with guidance to use new procedure)
**User-facing Features:**
- **Unified timeline interface**: Single procedure for all timeline operations instead of multiple specialized procedures
- **Advanced filtering capabilities**: Time-based and SQL expression filtering for precise result control
- **Historical data access**: Archive timeline support for complete audit trails
**Performance Impact:**
- **Optimized timeline scanning**: Efficient file system scanning with proper extension filtering
- **Configurable limits**: Default 20-entry limit if there is no start and end time mentioned with user override capability
- **Selective archive access**: Archive timeline only loaded when explicitly requested
### Risk level: **Low**
**Verification performed:**
- **Comprehensive test coverage**: 4 focused test cases covering basic functionality, MoR tables, rollback operations, and state transitions
- **Schema validation**: All output fields properly typed and validated
- **Error handling**: Graceful handling of invalid filters, missing tables, and timeline access failures
- **Timeline consistency**: Proper handling of both active and archived timelines with correct state mapping
### Documentation Update
- Add `show_timeline` procedure to Hudi Spark SQL procedures documentation
- Update timeline management examples to use new procedure
- Add advanced filtering examples and use cases
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org>.
vamshikrishnakyatham commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2353606706
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
+
+ limitedInstants.map { instant =>
+ createTimelineEntry(instant, timelineType, metaClient, instantInfoMap, rollbackInfoMap)
+ }
+ }
+
+ private def getRolledBackInstantInfo(timeline: HoodieTimeline, metaClient: HoodieTableMetaClient): Map[String, List[String]] = {
+ val rollbackInfoMap = scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[String]]()
+
+ val rollbackInstants = timeline.filter(instant =>
+ HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)).getInstants
+
+ rollbackInstants.asScala.foreach { rollbackInstant =>
+ try {
+ if (rollbackInstant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, rollbackInstant.getAction, rollbackInstant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ val rolledBackInstant = metadata.getInstantToRollback.getCommitTime
+ rollbackInfoMap.getOrElseUpdate(rolledBackInstant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ } else {
+ val metadata = timeline.readRollbackMetadata(rollbackInstant)
+ metadata.getCommitsRollback.asScala.foreach { instant =>
+ rollbackInfoMap.getOrElseUpdate(instant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ }
+ }
+ } catch {
+ case _: Exception => // Skip invalid rollback instants
+ }
+ }
+ rollbackInfoMap.map { case (k, v) => k -> v.toList }.toMap
+ }
+
+ private def getRollbackInfo(instant: HoodieInstant, timeline: HoodieTimeline, rollbackInfoMap: Map[String, List[String]], metaClient: HoodieTableMetaClient): String = {
+ try {
+ if (HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)) {
+ if (instant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, instant.getAction, instant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ s"Rolls back ${metadata.getInstantToRollback.getCommitTime}"
+ } else {
+ val metadata = timeline.readRollbackMetadata(instant)
+ val rolledBackInstants = metadata.getCommitsRollback.asScala.mkString(", ")
+ s"Rolled back: $rolledBackInstants"
+ }
+ } else {
+ val instantTimestamp = instant.requestedTime()
+ if (rollbackInfoMap.contains(instantTimestamp)) {
+ s"Rolled back by: ${rollbackInfoMap(instantTimestamp).mkString(", ")}"
+ } else {
+ null
+ }
+ }
+ } catch {
+ case _: Exception => null
+ }
+ }
+
+ private case class HoodieInstantWithModTime(state: HoodieInstant.State,
+ action: String,
+ requestedTime: String,
+ completionTime: String,
+ modificationTimeMs: Long
+ ) {
+ def getModificationTime: Long = modificationTimeMs
+ }
+
+ private def buildInstantInfoFromTimeline(metaClient: HoodieTableMetaClient): Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]] = {
Review Comment:
sure
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3235625185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1827abc2cd6fede450ee052c7e48fbf2dc12ecd4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319211324
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
Review Comment:
Java Doc Please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319364941
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
+
+ limitedInstants.map { instant =>
+ createTimelineEntry(instant, timelineType, metaClient, instantInfoMap, rollbackInfoMap)
+ }
+ }
+
+ private def getRolledBackInstantInfo(timeline: HoodieTimeline, metaClient: HoodieTableMetaClient): Map[String, List[String]] = {
+ val rollbackInfoMap = scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[String]]()
+
+ val rollbackInstants = timeline.filter(instant =>
+ HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)).getInstants
+
+ rollbackInstants.asScala.foreach { rollbackInstant =>
+ try {
+ if (rollbackInstant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, rollbackInstant.getAction, rollbackInstant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ val rolledBackInstant = metadata.getInstantToRollback.getCommitTime
+ rollbackInfoMap.getOrElseUpdate(rolledBackInstant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ } else {
+ val metadata = timeline.readRollbackMetadata(rollbackInstant)
+ metadata.getCommitsRollback.asScala.foreach { instant =>
+ rollbackInfoMap.getOrElseUpdate(instant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ }
+ }
+ } catch {
+ case _: Exception => // Skip invalid rollback instants
+ }
+ }
+ rollbackInfoMap.map { case (k, v) => k -> v.toList }.toMap
+ }
+
+ private def getRollbackInfo(instant: HoodieInstant, timeline: HoodieTimeline, rollbackInfoMap: Map[String, List[String]], metaClient: HoodieTableMetaClient): String = {
+ try {
+ if (HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)) {
+ if (instant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, instant.getAction, instant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ s"Rolls back ${metadata.getInstantToRollback.getCommitTime}"
+ } else {
+ val metadata = timeline.readRollbackMetadata(instant)
+ val rolledBackInstants = metadata.getCommitsRollback.asScala.mkString(", ")
+ s"Rolled back: $rolledBackInstants"
+ }
+ } else {
+ val instantTimestamp = instant.requestedTime()
+ if (rollbackInfoMap.contains(instantTimestamp)) {
+ s"Rolled back by: ${rollbackInfoMap(instantTimestamp).mkString(", ")}"
+ } else {
+ null
+ }
+ }
+ } catch {
+ case _: Exception => null
+ }
+ }
+
+ private case class HoodieInstantWithModTime(state: HoodieInstant.State,
+ action: String,
+ requestedTime: String,
+ completionTime: String,
+ modificationTimeMs: Long
+ ) {
+ def getModificationTime: Long = modificationTimeMs
+ }
+
+ private def buildInstantInfoFromTimeline(metaClient: HoodieTableMetaClient): Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]] = {
+ try {
+ val storage = metaClient.getStorage
+ val timelinePath = metaClient.getTimelinePath
+ val instantFileNameParser = metaClient.getInstantFileNameParser
+ val instantGenerator = metaClient.getInstantGenerator
+
+ val instantMap = scala.collection.mutable.Map[String, scala.collection.mutable.Map[HoodieInstant.State, HoodieInstantWithModTime]]()
+
+ val fileStream = HoodieTableMetaClient.scanFiles(storage, timelinePath, path => {
+ val extension = instantFileNameParser.getTimelineFileExtension(path.getName)
+ metaClient.getActiveTimeline.getValidExtensionsInActiveTimeline.contains(extension)
+ })
+
+ fileStream.forEach { storagePathInfo =>
+ try {
+ val instant = instantGenerator.createNewInstant(storagePathInfo)
+ val instantWithModTime = HoodieInstantWithModTime(
+ instant.getState,
+ instant.getAction,
+ instant.requestedTime(),
+ instant.getCompletionTime,
+ storagePathInfo.getModificationTime
+ )
+
+ instantMap.getOrElseUpdate(instant.requestedTime(), scala.collection.mutable.Map[HoodieInstant.State, HoodieInstantWithModTime]())
+ .put(instant.getState, instantWithModTime)
+ } catch {
+ case _: Exception => // Skip invalid files
+ }
+ }
+ instantMap.map { case (timestamp, stateMap) =>
+ timestamp -> stateMap.toMap
+ }.toMap
+ } catch {
+ case _: Exception => Map.empty[String, Map[HoodieInstant.State, HoodieInstantWithModTime]]
+ }
+ }
+
+ private def getFormattedDateForState(instantTimestamp: String, state: HoodieInstant.State,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]]): String = {
+ val stateMap = instantInfoMap.get(instantTimestamp)
+ if (stateMap.isDefined) {
+ val stateInfo = stateMap.get.get(state)
+ if (stateInfo.isDefined) {
+ val modificationTime = stateInfo.get.getModificationTime
+ if (modificationTime > 0) {
+ val date = new java.util.Date(modificationTime)
+ val formatter = new java.text.SimpleDateFormat("MM-dd HH:mm:ss")
+ formatter.format(date)
+ } else {
+ instantTimestamp
+ }
+ } else {
+ null
+ }
+ } else {
+ null
+ }
+ }
+
+ private def createTimelineEntry(instant: HoodieInstant, timelineType: String, metaClient: HoodieTableMetaClient,
Review Comment:
Java Doc Please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2321432287
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
Review Comment:
Is that possible to use timeline.findInstantsInRange(startTime, endTime)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3313917815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
}, {
"hash" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270",
"triggerID" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"triggerType" : "PUSH"
}, {
"hash" : "521ed76db16f0950e7995c45543ddda0787c3ee9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "521ed76db16f0950e7995c45543ddda0787c3ee9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270)
* 521ed76db16f0950e7995c45543ddda0787c3ee9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319310185
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
Review Comment:
Java Doc Please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3235461750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe4ca1c9531933c724ae213415036a5296da6f5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240)
* 1827abc2cd6fede450ee052c7e48fbf2dc12ecd4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #13804: [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines
URL: https://github.com/apache/hudi/pull/13804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3235383503
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe4ca1c9531933c724ae213415036a5296da6f5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240)
* 1827abc2cd6fede450ee052c7e48fbf2dc12ecd4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319324719
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
Review Comment:
Why does the limit only take effect when the user does not set the startTime and endTime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319214058
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
+
+ limitedInstants.map { instant =>
+ createTimelineEntry(instant, timelineType, metaClient, instantInfoMap, rollbackInfoMap)
+ }
+ }
+
+ private def getRolledBackInstantInfo(timeline: HoodieTimeline, metaClient: HoodieTableMetaClient): Map[String, List[String]] = {
+ val rollbackInfoMap = scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[String]]()
+
+ val rollbackInstants = timeline.filter(instant =>
+ HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)).getInstants
+
+ rollbackInstants.asScala.foreach { rollbackInstant =>
+ try {
+ if (rollbackInstant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, rollbackInstant.getAction, rollbackInstant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ val rolledBackInstant = metadata.getInstantToRollback.getCommitTime
+ rollbackInfoMap.getOrElseUpdate(rolledBackInstant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ } else {
+ val metadata = timeline.readRollbackMetadata(rollbackInstant)
+ metadata.getCommitsRollback.asScala.foreach { instant =>
+ rollbackInfoMap.getOrElseUpdate(instant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ }
+ }
+ } catch {
+ case _: Exception => // Skip invalid rollback instants
+ }
+ }
+ rollbackInfoMap.map { case (k, v) => k -> v.toList }.toMap
+ }
+
+ private def getRollbackInfo(instant: HoodieInstant, timeline: HoodieTimeline, rollbackInfoMap: Map[String, List[String]], metaClient: HoodieTableMetaClient): String = {
+ try {
+ if (HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)) {
+ if (instant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, instant.getAction, instant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ s"Rolls back ${metadata.getInstantToRollback.getCommitTime}"
+ } else {
+ val metadata = timeline.readRollbackMetadata(instant)
+ val rolledBackInstants = metadata.getCommitsRollback.asScala.mkString(", ")
+ s"Rolled back: $rolledBackInstants"
+ }
+ } else {
+ val instantTimestamp = instant.requestedTime()
+ if (rollbackInfoMap.contains(instantTimestamp)) {
+ s"Rolled back by: ${rollbackInfoMap(instantTimestamp).mkString(", ")}"
+ } else {
+ null
+ }
+ }
+ } catch {
+ case _: Exception => null
+ }
+ }
+
+ private case class HoodieInstantWithModTime(state: HoodieInstant.State,
+ action: String,
+ requestedTime: String,
+ completionTime: String,
+ modificationTimeMs: Long
+ ) {
+ def getModificationTime: Long = modificationTimeMs
+ }
+
+ private def buildInstantInfoFromTimeline(metaClient: HoodieTableMetaClient): Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]] = {
Review Comment:
JavaDoc Please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3235374889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe4ca1c9531933c724ae213415036a5296da6f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3235377798
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe4ca1c9531933c724ae213415036a5296da6f5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3235380573
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ffe4ca1c9531933c724ae213415036a5296da6f5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319351848
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
Review Comment:
Maybe we need to redesign the use of limit. IMO, the limit should have nothing to do with the start time and end time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319177516
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
+
+ limitedInstants.map { instant =>
+ createTimelineEntry(instant, timelineType, metaClient, instantInfoMap, rollbackInfoMap)
+ }
+ }
+
+ private def getRolledBackInstantInfo(timeline: HoodieTimeline, metaClient: HoodieTableMetaClient): Map[String, List[String]] = {
+ val rollbackInfoMap = scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[String]]()
+
+ val rollbackInstants = timeline.filter(instant =>
+ HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)).getInstants
+
+ rollbackInstants.asScala.foreach { rollbackInstant =>
+ try {
+ if (rollbackInstant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, rollbackInstant.getAction, rollbackInstant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ val rolledBackInstant = metadata.getInstantToRollback.getCommitTime
+ rollbackInfoMap.getOrElseUpdate(rolledBackInstant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ } else {
+ val metadata = timeline.readRollbackMetadata(rollbackInstant)
+ metadata.getCommitsRollback.asScala.foreach { instant =>
+ rollbackInfoMap.getOrElseUpdate(instant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ }
+ }
+ } catch {
+ case _: Exception => // Skip invalid rollback instants
+ }
+ }
+ rollbackInfoMap.map { case (k, v) => k -> v.toList }.toMap
+ }
+
+ private def getRollbackInfo(instant: HoodieInstant, timeline: HoodieTimeline, rollbackInfoMap: Map[String, List[String]], metaClient: HoodieTableMetaClient): String = {
+ try {
+ if (HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)) {
+ if (instant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, instant.getAction, instant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ s"Rolls back ${metadata.getInstantToRollback.getCommitTime}"
+ } else {
+ val metadata = timeline.readRollbackMetadata(instant)
+ val rolledBackInstants = metadata.getCommitsRollback.asScala.mkString(", ")
+ s"Rolled back: $rolledBackInstants"
+ }
+ } else {
+ val instantTimestamp = instant.requestedTime()
+ if (rollbackInfoMap.contains(instantTimestamp)) {
+ s"Rolled back by: ${rollbackInfoMap(instantTimestamp).mkString(", ")}"
+ } else {
+ null
+ }
+ }
+ } catch {
+ case _: Exception => null
+ }
+ }
+
+ private case class HoodieInstantWithModTime(state: HoodieInstant.State,
+ action: String,
+ requestedTime: String,
+ completionTime: String,
+ modificationTimeMs: Long
+ ) {
+ def getModificationTime: Long = modificationTimeMs
Review Comment:
Looks like we do not need this function `def getModificationTime: Long = modificationTimeMs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2319373859
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
+
+ limitedInstants.map { instant =>
+ createTimelineEntry(instant, timelineType, metaClient, instantInfoMap, rollbackInfoMap)
+ }
+ }
+
+ private def getRolledBackInstantInfo(timeline: HoodieTimeline, metaClient: HoodieTableMetaClient): Map[String, List[String]] = {
+ val rollbackInfoMap = scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[String]]()
+
+ val rollbackInstants = timeline.filter(instant =>
+ HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)).getInstants
+
+ rollbackInstants.asScala.foreach { rollbackInstant =>
+ try {
+ if (rollbackInstant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, rollbackInstant.getAction, rollbackInstant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ val rolledBackInstant = metadata.getInstantToRollback.getCommitTime
+ rollbackInfoMap.getOrElseUpdate(rolledBackInstant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ } else {
+ val metadata = timeline.readRollbackMetadata(rollbackInstant)
+ metadata.getCommitsRollback.asScala.foreach { instant =>
+ rollbackInfoMap.getOrElseUpdate(instant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ }
+ }
+ } catch {
+ case _: Exception => // Skip invalid rollback instants
+ }
+ }
+ rollbackInfoMap.map { case (k, v) => k -> v.toList }.toMap
+ }
+
+ private def getRollbackInfo(instant: HoodieInstant, timeline: HoodieTimeline, rollbackInfoMap: Map[String, List[String]], metaClient: HoodieTableMetaClient): String = {
Review Comment:
Java Doc please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org>.
vamshikrishnakyatham commented on code in PR #13804:
URL: https://github.com/apache/hudi/pull/13804#discussion_r2353606156
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
+ limit: Int,
+ showArchived: Boolean,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instantInfoMap = buildInstantInfoFromTimeline(metaClient)
+
+ val activeRollbackInfoMap = getRolledBackInstantInfo(metaClient.getActiveTimeline, metaClient)
+ val archivedRollbackInfoMap = if (showArchived) {
+ getRolledBackInstantInfo(metaClient.getArchivedTimeline, metaClient)
+ } else {
+ Map.empty[String, List[String]]
+ }
+
+ val activeEntries = getTimelineEntriesFromTimeline(
+ metaClient.getActiveTimeline, "ACTIVE", metaClient, instantInfoMap, activeRollbackInfoMap, limit, startTime, endTime
+ )
+
+ val finalEntries = if (showArchived) {
+ val archivedEntries = getTimelineEntriesFromTimeline(
+ metaClient.getArchivedTimeline, "ARCHIVED", metaClient, instantInfoMap, archivedRollbackInfoMap, limit, startTime, endTime
+ )
+ val combinedEntries = (activeEntries ++ archivedEntries)
+ .sortWith((a, b) => a.getString(0) > b.getString(0))
+
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ combinedEntries
+ } else {
+ combinedEntries.take(limit)
+ }
+ } else {
+ if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ activeEntries
+ } else {
+ activeEntries.take(limit)
+ }
+ }
+
+ finalEntries
+ }
+
+ private def getTimelineEntriesFromTimeline(timeline: HoodieTimeline,
+ timelineType: String,
+ metaClient: HoodieTableMetaClient,
+ instantInfoMap: Map[String, Map[HoodieInstant.State, HoodieInstantWithModTime]],
+ rollbackInfoMap: Map[String, List[String]],
+ limit: Int,
+ startTime: String,
+ endTime: String): Seq[Row] = {
+
+ val instants = timeline.getInstants.iterator().asScala.toSeq
+
+ val filteredInstants = if (startTime.trim.nonEmpty || endTime.trim.nonEmpty) {
+ instants.filter { instant =>
+ val instantTime = instant.requestedTime()
+ val withinStartTime = if (startTime.trim.nonEmpty) instantTime >= startTime else true
+ val withinEndTime = if (endTime.trim.nonEmpty) instantTime <= endTime else true
+ withinStartTime && withinEndTime
+ }
+ } else {
+ instants
+ }
+
+ val sortedInstants = filteredInstants.sortWith((a, b) => a.requestedTime() > b.requestedTime())
+
+ val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+ sortedInstants
+ } else {
+ sortedInstants.take(limit)
+ }
+
+ limitedInstants.map { instant =>
+ createTimelineEntry(instant, timelineType, metaClient, instantInfoMap, rollbackInfoMap)
+ }
+ }
+
+ private def getRolledBackInstantInfo(timeline: HoodieTimeline, metaClient: HoodieTableMetaClient): Map[String, List[String]] = {
+ val rollbackInfoMap = scala.collection.mutable.Map[String, scala.collection.mutable.ListBuffer[String]]()
+
+ val rollbackInstants = timeline.filter(instant =>
+ HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)).getInstants
+
+ rollbackInstants.asScala.foreach { rollbackInstant =>
+ try {
+ if (rollbackInstant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, rollbackInstant.getAction, rollbackInstant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ val rolledBackInstant = metadata.getInstantToRollback.getCommitTime
+ rollbackInfoMap.getOrElseUpdate(rolledBackInstant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ } else {
+ val metadata = timeline.readRollbackMetadata(rollbackInstant)
+ metadata.getCommitsRollback.asScala.foreach { instant =>
+ rollbackInfoMap.getOrElseUpdate(instant, scala.collection.mutable.ListBuffer[String]())
+ .append(rollbackInstant.requestedTime())
+ }
+ }
+ } catch {
+ case _: Exception => // Skip invalid rollback instants
+ }
+ }
+ rollbackInfoMap.map { case (k, v) => k -> v.toList }.toMap
+ }
+
+ private def getRollbackInfo(instant: HoodieInstant, timeline: HoodieTimeline, rollbackInfoMap: Map[String, List[String]], metaClient: HoodieTableMetaClient): String = {
+ try {
+ if (HoodieTimeline.ROLLBACK_ACTION.equalsIgnoreCase(instant.getAction)) {
+ if (instant.isInflight) {
+ val instantToUse = metaClient.createNewInstant(
+ HoodieInstant.State.REQUESTED, instant.getAction, instant.requestedTime())
+ val metadata = timeline.readRollbackPlan(instantToUse)
+ s"Rolls back ${metadata.getInstantToRollback.getCommitTime}"
+ } else {
+ val metadata = timeline.readRollbackMetadata(instant)
+ val rolledBackInstants = metadata.getCommitsRollback.asScala.mkString(", ")
+ s"Rolled back: $rolledBackInstants"
+ }
+ } else {
+ val instantTimestamp = instant.requestedTime()
+ if (rollbackInfoMap.contains(instantTimestamp)) {
+ s"Rolled back by: ${rollbackInfoMap(instantTimestamp).mkString(", ")}"
+ } else {
+ null
+ }
+ }
+ } catch {
+ case _: Exception => null
+ }
+ }
+
+ private case class HoodieInstantWithModTime(state: HoodieInstant.State,
+ action: String,
+ requestedTime: String,
+ completionTime: String,
+ modificationTimeMs: Long
+ ) {
+ def getModificationTime: Long = modificationTimeMs
Review Comment:
I am using it to have get the modification time to be in a formatted way - just structured hoodie instant to be in a more better reusable way
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala:
##########
@@ -0,0 +1,381 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command.procedures
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.common.table.timeline.{HoodieInstant, HoodieTimeline}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.types.{DataTypes, Metadata, StructField, StructType}
+
+import java.util.function.Supplier
+
+import scala.collection.JavaConverters._
+
+/**
+ * Spark SQL procedure to show timeline information for a Hudi table.
+ *
+ * This procedure displays comprehensive information about all timeline instants including commits,
+ * compactions, clustering, cleaning, and other table operations. It supports both active and
+ * archived timelines with time-based filtering and generic SQL filters.
+ *
+ * == Parameters ==
+ * - `table`: Optional. The name of the Hudi table to query (mutually exclusive with `path`)
+ * - `path`: Optional. The base path of the Hudi table (mutually exclusive with `table`)
+ * - `limit`: Optional. Maximum number of timeline entries to return (default: 20)
+ * - `showArchived`: Optional. Whether to include archived timeline data (default: false)
+ * - `filter`: Optional. SQL expression to filter results (default: empty string)
+ * - `startTime`: Optional. Start timestamp for filtering results (format: yyyyMMddHHmmss)
+ * - `endTime`: Optional. End timestamp for filtering results (format: yyyyMMddHHmmss)
+ *
+ * == Output Schema ==
+ * - `instant_time`: Timestamp when the operation was performed
+ * - `action`: The action type (commit, deltacommit, compaction, clustering, clean, rollback, etc.)
+ * - `state`: Current state of the operation (REQUESTED, INFLIGHT, COMPLETED)
+ * - `requested_time`: Formatted date when the operation was requested (MM-dd HH:mm:ss format)
+ * - `inflight_time`: Formatted date when the operation became inflight (MM-dd HH:mm:ss format)
+ * - `completed_time`: Formatted date when the operation completed (MM-dd HH:mm format, "-" if not completed)
+ * - `timeline_type`: Whether the data is from ACTIVE or ARCHIVED timeline
+ * - `rollback_info`: Information about rollback operations (what was rolled back or what rolled back this operation)
+ *
+ * == Error Handling ==
+ * - Throws `IllegalArgumentException` for invalid filter expressions
+ * - Throws `HoodieException` for table access issues
+ * - Returns empty result set if no timeline entries match the criteria
+ * - Gracefully handles archived timeline access failures with warning logs
+ *
+ * == Filter Support ==
+ * The `filter` parameter supports SQL expressions for filtering results on any output column.
+ * The filter uses Spark SQL syntax and supports various data types and operations.
+ *
+ * == Usage Examples ==
+ * {{{
+ * -- Basic usage: Show timeline entries
+ * CALL show_timeline(table => 'hudi_table')
+ *
+ * -- Show timeline with custom limit
+ * CALL show_timeline(table => 'hudi_table', limit => 50)
+ *
+ * -- Include archived timeline data
+ * CALL show_timeline(table => 'hudi_table', showArchived => true)
+ *
+ * -- Filter for specific actions
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * filter => "action = 'commit'"
+ * )
+ *
+ * -- Show timeline with time range and filtering
+ * CALL show_timeline(
+ * table => 'hudi_table',
+ * startTime => '20251201000000',
+ * endTime => '20251231235959',
+ * filter => "action = 'commit' AND state = 'COMPLETED'"
+ * )
+ * }}}
+ *
+ * @see [[HoodieProcedureFilterUtils]] for detailed filter expression syntax
+ */
+class ShowTimelineProcedure extends BaseProcedure with ProcedureBuilder with SparkAdapterSupport with Logging {
+
+ private val PARAMETERS = Array[ProcedureParameter](
+ ProcedureParameter.optional(0, "table", DataTypes.StringType),
+ ProcedureParameter.optional(1, "path", DataTypes.StringType),
+ ProcedureParameter.optional(2, "limit", DataTypes.IntegerType, 20),
+ ProcedureParameter.optional(3, "showArchived", DataTypes.BooleanType, false),
+ ProcedureParameter.optional(4, "filter", DataTypes.StringType, ""),
+ ProcedureParameter.optional(5, "startTime", DataTypes.StringType, ""),
+ ProcedureParameter.optional(6, "endTime", DataTypes.StringType, "")
+ )
+
+ private val OUTPUT_TYPE = new StructType(Array[StructField](
+ StructField("instant_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("action", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("state", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("requested_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("inflight_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("completed_time", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("timeline_type", DataTypes.StringType, nullable = true, Metadata.empty),
+ StructField("rollback_info", DataTypes.StringType, nullable = true, Metadata.empty)
+ ))
+
+ def parameters: Array[ProcedureParameter] = PARAMETERS
+
+ def outputType: StructType = OUTPUT_TYPE
+
+ override def call(args: ProcedureArgs): Seq[Row] = {
+ super.checkArgs(PARAMETERS, args)
+
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ val showArchived = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[Boolean]
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val startTime = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+ val endTime = getArgValueOrDefault(args, PARAMETERS(6)).get.asInstanceOf[String]
+
+ validateFilter(filter, outputType)
+
+ val basePath: String = getBasePath(tableName, tablePath)
+ val metaClient = createMetaClient(jsc, basePath)
+
+ val timelineEntries = getTimelineEntries(metaClient, limit, showArchived, startTime, endTime)
+
+ applyFilter(timelineEntries, filter, outputType)
+ }
+
+ override def build: Procedure = new ShowTimelineProcedure()
+
+ private def getTimelineEntries(metaClient: HoodieTableMetaClient,
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3313922188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
}, {
"hash" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270",
"triggerID" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"triggerType" : "PUSH"
}, {
"hash" : "521ed76db16f0950e7995c45543ddda0787c3ee9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8659",
"triggerID" : "521ed76db16f0950e7995c45543ddda0787c3ee9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 521ed76db16f0950e7995c45543ddda0787c3ee9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8659)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3238113111
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
}, {
"hash" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270",
"triggerID" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3237845774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
}, {
"hash" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270",
"triggerID" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1827abc2cd6fede450ee052c7e48fbf2dc12ecd4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242)
* afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8270)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13804:
URL: https://github.com/apache/hudi/pull/13804#issuecomment-3237840410
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8240",
"triggerID" : "ffe4ca1c9531933c724ae213415036a5296da6f5",
"triggerType" : "PUSH"
}, {
"hash" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242",
"triggerID" : "1827abc2cd6fede450ee052c7e48fbf2dc12ecd4",
"triggerType" : "PUSH"
}, {
"hash" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1827abc2cd6fede450ee052c7e48fbf2dc12ecd4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8242)
* afd7d6bc6b89e09a03ce6b7da9e197337dbcd1de UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org