You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "beliefer (via GitHub)" <gi...@apache.org> on 2025/11/27 08:02:48 UTC
[PR] [WIP][GLUTEN-11203][CORE][VL][CH] Validate input format for Hive table's partition [incubator-gluten]
beliefer opened a new pull request, #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205
## What changes are proposed in this pull request?
This PR proposes to validate input format for Hive table's partition.
Fixes #11203
## How was this patch tested?
GA tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [WIP][GLUTEN-11203][CORE][VL][CH] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3584982461
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3588760997
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#discussion_r2571188779
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -221,7 +230,8 @@ object VeloxBackendSettings extends BackendSettingsApi {
val validationChecks = Seq(
validateScheme(),
- validateFormat(),
+ validateFormat(format),
Review Comment:
I think we should keep it. Because all the partition's formats are possible not same as table's format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE][VL][CH] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3587825587
ping @wForget @zhztheplayer @jackylee-ch @jinchengchenghh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#discussion_r2579351795
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -183,6 +184,14 @@ object VeloxBackendSettings extends BackendSettingsApi {
}
}
+ def validatePartitionFormat(): Option[String] =
+ partitionFileFormats.iterator
Review Comment:
Do we need to filter out table format?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer merged PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3600440806
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#discussion_r2571192806
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveTableScanExecTransformer.scala:
##########
@@ -113,6 +116,9 @@ case class HiveTableScanExecTransformer(
@transient private lazy val partitions: Seq[Partition] = partitionWithReadFileFormats.unzip._1
+ @transient private lazy val distinctReadFileFormatsCache: Set[ReadFileFormat] =
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3588743708
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3600364431
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#discussion_r2571188779
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -221,7 +230,8 @@ object VeloxBackendSettings extends BackendSettingsApi {
val validationChecks = Seq(
validateScheme(),
- validateFormat(),
+ validateFormat(format),
Review Comment:
~I think we should keep it. Because all the partition's formats are possible not same as table's format.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [WIP][GLUTEN-11203][CORE][VL][CH] Validate input format for Hive table's partition [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3584609585
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE][VL][CH] Validate input format for Hive table's partition [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#discussion_r2570994647
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -221,7 +230,8 @@ object VeloxBackendSettings extends BackendSettingsApi {
val validationChecks = Seq(
validateScheme(),
- validateFormat(),
+ validateFormat(format),
Review Comment:
Do we still need validateFormat? Looks like validatePartitionFormat is enough.
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/hive/HiveTableScanExecTransformer.scala:
##########
@@ -113,6 +116,9 @@ case class HiveTableScanExecTransformer(
@transient private lazy val partitions: Seq[Partition] = partitionWithReadFileFormats.unzip._1
+ @transient private lazy val distinctReadFileFormatsCache: Set[ReadFileFormat] =
Review Comment:
distinctReadFileFormatsCache -> distinctReadFileFormats
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on code in PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#discussion_r2583402494
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala:
##########
@@ -221,7 +230,8 @@ object VeloxBackendSettings extends BackendSettingsApi {
val validationChecks = Seq(
validateScheme(),
- validateFormat(),
+ validateFormat(format),
Review Comment:
Now, I remove `validateFormat` and using `validateFormats` to check them together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11203][CORE] Validate input format for Hive table's partition [incubator-gluten]
Posted by "beliefer (via GitHub)" <gi...@apache.org>.
beliefer commented on PR #11205:
URL: https://github.com/apache/incubator-gluten/pull/11205#issuecomment-3609750572
@PHILO-HE @wForget @jinchengchenghh Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org