You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org> on 2025/08/27 20:01:51 UTC
[PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
vamshikrishnakyatham opened a new pull request, #13790:
URL: https://github.com/apache/hudi/pull/13790
### Change Logs
Supporting basePath and the generic filter option for all the non action Show Spark Procedures
### Impact
Advanced filtering support for restricting the results of Show Spark Procedures and support of base path of table as an argument (if table name is not known) to get the results.
### Risk level (write none, low medium or high below)
low
### Documentation Update
Hudi website documentation needs to be updated to include the support in the procedures section
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13790:
URL: https://github.com/apache/hudi/pull/13790#discussion_r2305300955
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowColumnStatsOverlapProcedure.scala:
##########
@@ -123,7 +135,12 @@ class ShowColumnStatsOverlapProcedure extends BaseProcedure with ProcedureBuilde
addStatisticsToRows(groupedPoints, fileSlicesSizeByPartition.toMap, rows)
// The returned results are sorted by column name and average value
- rows.asScala.toList.sortBy(row => (row.getString(1), row.getDouble(2)))
+ val results = rows.asScala.toList.sortBy(row => (row.getString(1), row.getDouble(2)))
+ if (filter != null && filter.trim.nonEmpty) {
+ HoodieProcedureFilterUtils.evaluateFilter(results, filter, OUTPUT_TYPE, sparkSession)
+ } else {
+ results
+ }
}
private def getTargetColumnsSeq(args: ProcedureArgs): Seq[String] = {
Review Comment:
This needs to be updated with the new argument index.
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowColumnStatsOverlapProcedure.scala:
##########
@@ -95,17 +97,27 @@ class ShowColumnStatsOverlapProcedure extends BaseProcedure with ProcedureBuilde
override def call(args: ProcedureArgs): Seq[Row] = {
super.checkArgs(PARAMETERS, args)
- val table = getArgValueOrDefault(args, PARAMETERS(0))
- val partitions = getArgValueOrDefault(args, PARAMETERS(1)).getOrElse("").toString
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val partitions = getArgValueOrDefault(args, PARAMETERS(2)).getOrElse("").toString
val partitionsSeq = partitions.split(",").filter(_.nonEmpty).toSeq
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+
+ if (filter != null && filter.trim.nonEmpty) {
+ HoodieProcedureFilterUtils.validateFilterExpression(filter, OUTPUT_TYPE, sparkSession) match {
+ case Left(errorMessage) =>
+ throw new IllegalArgumentException(s"Invalid filter expression: $errorMessage")
+ case Right(_) => // Validation passed, continue
+ }
+ }
val targetColumnsSeq = getTargetColumnsSeq(args)
- val basePath = getBasePath(table)
+ val basePath = getBasePath(tableName, tablePath)
val metadataConfig = HoodieMetadataConfig.newBuilder().enable(true).build
val metaClient = createMetaClient(jsc, basePath)
val schema = getSchema(metaClient)
val columnStatsIndex = new ColumnStatsIndexSupport(spark, schema, metadataConfig, metaClient)
- val fsView = buildFileSystemView(table)
+ val fsView = buildFileSystemView(tableName, tablePath)
Review Comment:
Should this take resolved `basePath`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3233721704
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7963b75c9256d74ea3a95d25be7859d8befbffb4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13790:
URL: https://github.com/apache/hudi/pull/13790
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229813449
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179)
* c7dc67b763b6af33fdee649ca67578eb2cd076a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229824647
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7dc67b763b6af33fdee649ca67578eb2cd076a0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "vamshikrishnakyatham (via GitHub)" <gi...@apache.org>.
vamshikrishnakyatham commented on code in PR #13790:
URL: https://github.com/apache/hudi/pull/13790#discussion_r2305312533
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowColumnStatsOverlapProcedure.scala:
##########
@@ -95,17 +97,27 @@ class ShowColumnStatsOverlapProcedure extends BaseProcedure with ProcedureBuilde
override def call(args: ProcedureArgs): Seq[Row] = {
super.checkArgs(PARAMETERS, args)
- val table = getArgValueOrDefault(args, PARAMETERS(0))
- val partitions = getArgValueOrDefault(args, PARAMETERS(1)).getOrElse("").toString
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val partitions = getArgValueOrDefault(args, PARAMETERS(2)).getOrElse("").toString
val partitionsSeq = partitions.split(",").filter(_.nonEmpty).toSeq
+ val filter = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+
+ if (filter != null && filter.trim.nonEmpty) {
+ HoodieProcedureFilterUtils.validateFilterExpression(filter, OUTPUT_TYPE, sparkSession) match {
+ case Left(errorMessage) =>
+ throw new IllegalArgumentException(s"Invalid filter expression: $errorMessage")
+ case Right(_) => // Validation passed, continue
+ }
+ }
val targetColumnsSeq = getTargetColumnsSeq(args)
- val basePath = getBasePath(table)
+ val basePath = getBasePath(tableName, tablePath)
val metadataConfig = HoodieMetadataConfig.newBuilder().enable(true).build
val metaClient = createMetaClient(jsc, basePath)
val schema = getSchema(metaClient)
val columnStatsIndex = new ColumnStatsIndexSupport(spark, schema, metadataConfig, metaClient)
- val fsView = buildFileSystemView(table)
+ val fsView = buildFileSystemView(tableName, tablePath)
Review Comment:
it does the same thing inside the `buildFileSystemView` method but yeah, we can definitely pass the basePath directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229755671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3231933837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
}, {
"hash" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98515d60d4826e446305f27776ee8b81520d4875 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184)
* 1fc1102ba9e504c65b78ed6f3b482c723275652d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229908898
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7dc67b763b6af33fdee649ca67578eb2cd076a0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181)
* 98515d60d4826e446305f27776ee8b81520d4875 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3232142289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
}, {
"hash" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202",
"triggerID" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"triggerType" : "PUSH"
}, {
"hash" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205",
"triggerID" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fc1102ba9e504c65b78ed6f3b482c723275652d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202)
* 7963b75c9256d74ea3a95d25be7859d8befbffb4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3232136895
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
}, {
"hash" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202",
"triggerID" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"triggerType" : "PUSH"
}, {
"hash" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fc1102ba9e504c65b78ed6f3b482c723275652d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202)
* 7963b75c9256d74ea3a95d25be7859d8befbffb4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3233957733
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205",
"triggerID" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7963b75c9256d74ea3a95d25be7859d8befbffb4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229582625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229952002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c7dc67b763b6af33fdee649ca67578eb2cd076a0 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181)
* 98515d60d4826e446305f27776ee8b81520d4875 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3232020951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
}, {
"hash" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202",
"triggerID" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1fc1102ba9e504c65b78ed6f3b482c723275652d Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3230082524
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98515d60d4826e446305f27776ee8b81520d4875 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3233731701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205",
"triggerID" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7963b75c9256d74ea3a95d25be7859d8befbffb4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229819397
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179)
* c7dc67b763b6af33fdee649ca67578eb2cd076a0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13790:
URL: https://github.com/apache/hudi/pull/13790#discussion_r2305288706
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowArchivedCommitsProcedure.scala:
##########
@@ -78,13 +80,22 @@ class ShowArchivedCommitsProcedure(includeExtraMetadata: Boolean) extends BasePr
override def call(args: ProcedureArgs): Seq[Row] = {
super.checkArgs(PARAMETERS, args)
- val table = getArgValueOrDefault(args, PARAMETERS(0)).get.asInstanceOf[String]
- val limit = getArgValueOrDefault(args, PARAMETERS(1)).get.asInstanceOf[Int]
- var startTs = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[String]
- var endTs = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[String]
+ val tableName = getArgValueOrDefault(args, PARAMETERS(0))
+ val tablePath = getArgValueOrDefault(args, PARAMETERS(1))
+ val limit = getArgValueOrDefault(args, PARAMETERS(2)).get.asInstanceOf[Int]
+ var startTs = getArgValueOrDefault(args, PARAMETERS(3)).get.asInstanceOf[String]
+ var endTs = getArgValueOrDefault(args, PARAMETERS(4)).get.asInstanceOf[String]
+ val filter = getArgValueOrDefault(args, PARAMETERS(5)).get.asInstanceOf[String]
+
+ if (filter != null && filter.trim.nonEmpty) {
+ HoodieProcedureFilterUtils.validateFilterExpression(filter, OUTPUT_TYPE, sparkSession) match {
+ case Left(errorMessage) =>
+ throw new IllegalArgumentException(s"Invalid filter expression: $errorMessage")
+ case Right(_) => // Validation passed, continue
+ }
+ }
Review Comment:
Should this be extract as a common util in `BaseProcedure`?
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowArchivedCommitsProcedure.scala:
##########
@@ -105,6 +116,11 @@ class ShowArchivedCommitsProcedure(includeExtraMetadata: Boolean) extends BasePr
// clear the instant details from memory after printing to reduce usage
archivedTimeline.clearInstantDetailsFromMemory(startTs, endTs)
}
+ if (filter != null && filter.trim.nonEmpty) {
+ HoodieProcedureFilterUtils.evaluateFilter(results, filter, OUTPUT_TYPE, sparkSession)
+ } else {
+ results
+ }
Review Comment:
Similar here on extracting the logic for simplicity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3232248488
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
}, {
"hash" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202",
"triggerID" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"triggerType" : "PUSH"
}, {
"hash" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205",
"triggerID" : "7963b75c9256d74ea3a95d25be7859d8befbffb4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7963b75c9256d74ea3a95d25be7859d8befbffb4 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8205)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3229589345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9746] Support basePath and the generic filter option for all the non action Show Spark Procedures [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13790:
URL: https://github.com/apache/hudi/pull/13790#issuecomment-3231938851
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8179",
"triggerID" : "abcbe416e69f72b17c14d0c0a5d8e6a27885dbe4",
"triggerType" : "PUSH"
}, {
"hash" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8181",
"triggerID" : "c7dc67b763b6af33fdee649ca67578eb2cd076a0",
"triggerType" : "PUSH"
}, {
"hash" : "98515d60d4826e446305f27776ee8b81520d4875",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184",
"triggerID" : "98515d60d4826e446305f27776ee8b81520d4875",
"triggerType" : "PUSH"
}, {
"hash" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202",
"triggerID" : "1fc1102ba9e504c65b78ed6f3b482c723275652d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 98515d60d4826e446305f27776ee8b81520d4875 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8184)
* 1fc1102ba9e504c65b78ed6f3b482c723275652d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=8202)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org