You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "wecharyu (via GitHub)" <gi...@apache.org> on 2026/05/08 11:16:48 UTC
[PR] [GLUTEN-11921] Support NullType Parquet read/write [gluten]
wecharyu opened a new pull request, #12056:
URL: https://github.com/apache/gluten/pull/12056
## What changes are proposed in this pull request?
Enable parquet NullType read test.
## How was this patch tested?
Pass UT in `GlutenParquetIOSuite`.
## Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5489841665
@wecharyu Just a little more comments, please check it again. Thanks for your work~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3902472954
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12056: [GLUTEN-11921] Enable Parquet read/write test for NullType
URL: https://github.com/apache/gluten/pull/12056
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5437624331
Gentle ping @philo-he @zhouyuan, pls help reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3901220538
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
Oh, got it. Then we can just change it with `.exclude("SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only")`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5476322177
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3902369413
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
The duplicated exclusions are acceptable for now. Backend-specific exclusions are reasonable, since different backends may have different implementations or optimization coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Support NullType Parquet read/write [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3252527294
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
I'd prefer exclude it here because this test failure is not Velox-specific, it's caused by Gluten instead of backends.
Spark test collect leaf scan node `FileSourceScanExecTransformer` and run `executeColumnar()` in `VerifyNoAdditionalScanOutputExec`: https://github.com/apache/gluten/blob/356fda012d859f693fc613e6ac3585b7661bdbc5/shims/spark41/src/main/scala/org/apache/spark/sql/execution/AbstractFileSourceScanExec.scala#L139-L142
The inputRDD is `RDD[UnsafeRow]` for `parquetVectorizedReaderEnabled=false` test path, so it would always failed by:
```bash
java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-4480904302
> The failure is expected, we need first support it in Velox, PR is [facebookincubator/velox#17391](https://github.com/facebookincubator/velox/pull/17391)
@wecharyu, thank you for the clarification. If you need to validate the Velox PR, you can push a temporary commit with the PR ID added in the code below.
https://github.com/apache/gluten/blob/97632d8435a8df1279478629122fdffee82bbeab/ep/build-velox/src/get-velox.sh#L28
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3871158934
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
+1 for `excludeByPrefix` in `VeloxTestSettings.scala`. Keeping all skipped/excluded tests centralized makes them easier to track and revisit. Otherwise, such exclusions can easily be forgotten when the context is only known to a few contributors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5483308414
@wecharyu, thank you for the update. Could you check if this comment makes sense? https://github.com/apache/gluten/pull/12056#discussion_r3889036234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3901220538
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
Oh, got it. Then we can just change it with `.exclude("SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only")`. Can we use this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-4964740599
This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3900336441
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
This PR aims to enable all "SPARK-54220" prefix tests, even if it makes some adoption for gluten, so I think we do not need track it as exclusion test because the test is really not excluded. cc @philo-he
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3902229169
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
If exclude it in `VeloxTestSettings.scala`, we will also need exclude it in `ClickHouseTestSettings.scala` once the clickhouse ci enables gluten-ut for Spark 4.1, because the original Spark test is not compatible with gluten as discussed in https://github.com/apache/gluten/pull/12056#discussion_r3252527294.
If the duplicated excludes in both backends are acceptable, I'm happy to change this exclude to `VeloxTestSettings.scala`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he merged PR #12056:
URL: https://github.com/apache/gluten/pull/12056
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5491776440
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5438126209
@wecharyu just opened this patch, would you please do a rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3879427473
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
@jackylee-ch Thanks for the suggestion. This is not a Velox-backend-specific issue, and the test is not excluded; it is adapted for Gluten. Keeping the exclusion in `VeloxTestSettings.scala` would make it appear that this test is disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5479859539
@philo-he Added `spark.sql.legacy.parquet.returnNullStructIfAllFieldsMissing` config support to solve the failed test, pls take a look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3889036234
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
`excludeByPrefix` does not match the `Gluten - ` prefix added by `testGluten`; consequently, the `.excludeByPrefix("SPARK-54220")` setting in `VeloxTestSettings` effectively allows your variant to keep running—a point @philo-he raised back in May.
More importantly, the `testNameBlackList` mechanism is difficult to track: it takes precedence over all backend-specific settings, meaning no backend can re-enable the test, and the skipped tests won't be reflected in #11550—easily leading to genuine issues being obscured within the test suite files.
It would be more appropriate to consolidate skip configurations applicable to all backends into `GlutenTestSettings`; I will file an issue to see if we can improve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-4478950264
The failure is expected, we need first support it in Velox, PR is https://github.com/facebookincubator/velox/pull/17391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Support NullType Parquet read/write [gluten]
Posted by "philo-he (via GitHub)" <gi...@apache.org>.
philo-he commented on code in PR #12056:
URL: https://github.com/apache/gluten/pull/12056#discussion_r3243071946
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
`testGluten("SPARK-54220: xxx` will add a prefix into the test name. I assume `excludeByPrefix("SPARK-54220")` will not exclude this test variant.
##########
gluten-ut/spark41/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/GlutenParquetIOSuite.scala:
##########
@@ -17,14 +17,51 @@
package org.apache.spark.sql.execution.datasources.parquet
import org.apache.spark.sql._
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
/** A test suite that tests basic Parquet I/O. */
class GlutenParquetIOSuite extends ParquetIOSuite with GlutenSQLTestsBaseTrait {
+ override def testNameBlackList: Seq[String] =
Review Comment:
Can we keep this test excluded in VeloxTestSettings.scala? That way, it won't affect other backends' test enabling controls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5290644525
@philo-he Could you help reopen this PR? The Velox support is landed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11921] Enable Parquet read/write test for NullType [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12056:
URL: https://github.com/apache/gluten/pull/12056#issuecomment-5476315747
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org