You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "dmsuehir (via GitHub)" <gi...@apache.org> on 2025/06/13 21:19:06 UTC
[PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
dmsuehir opened a new pull request, #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969
## What changes were proposed in this pull request?
This PR:
* Clarifies the documentation regarding how to set the `--spark_home` arg when running the `gen-function-support-docs.py` script to explain that it can either be Spark built from source or an installed Spark release with additional resource files copied to the install directory
* Similarly, clarifies the documentation for running `spark-ut` unit tests to mention the same for the `spark.test.home` arg.
* Updates the `GlutenSQLQueryTestSuite`, where the original error was reported to check for the path existing before the code trying to get files from the directory. This is what was causing the null pointer exception previously. With the updated error handling, running the test run will still fail if the resource files are missing, but the message will be more clear and include the path to the missing directory.
* After running `gen-function-support-docs.py` to check my updates, there was one update to the Gluten support docs, so that is included in this PR.
(Fixes: \#9328)
## How was this patch tested?
Ran `gen-function-support-docs.py` and the `spark-ut` tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "dcoliversun (via GitHub)" <gi...@apache.org>.
dcoliversun commented on code in PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#discussion_r2149082816
##########
docs/developers/HowTo.md:
##########
@@ -125,7 +125,27 @@ To run a Scala/Java test class, you can use the below mvn command (take Velox ba
mvn test -Pspark-3.5 -Pspark-ut -Pbackends-velox -DargLine="-Dspark.test.home=/path/to/spark/source/code/home/" -DwildcardSuites=xxx
```
-Please set `wildcardSuites` with a fully qualified class name. `spark.test.home` is optional to set. It is only required for some test suites to use Spark resources.
+Please set `wildcardSuites` with a fully qualified class name.
+
Review Comment:
I think this is unnecessary to split one line into three lines. Please keep it as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#issuecomment-2977045726
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "PHILO-HE (via GitHub)" <gi...@apache.org>.
PHILO-HE merged PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "dmsuehir (via GitHub)" <gi...@apache.org>.
dmsuehir commented on code in PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#discussion_r2150262992
##########
docs/developers/HowTo.md:
##########
@@ -125,7 +125,27 @@ To run a Scala/Java test class, you can use the below mvn command (take Velox ba
mvn test -Pspark-3.5 -Pspark-ut -Pbackends-velox -DargLine="-Dspark.test.home=/path/to/spark/source/code/home/" -DwildcardSuites=xxx
```
-Please set `wildcardSuites` with a fully qualified class name. `spark.test.home` is optional to set. It is only required for some test suites to use Spark resources.
+Please set `wildcardSuites` with a fully qualified class name.
+
Review Comment:
Thank you for the review, I pushed an update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "dmsuehir (via GitHub)" <gi...@apache.org>.
dmsuehir commented on PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#issuecomment-2986141172
@PHILO-HE Thanks, I've tried that out and ran into a couple of issues:
* The install directory for `install_spark_resources.sh` is hardcoded to `/opt`. When trying to run this without sudo, I get a permission denied error
* After running `install_spark_resources.sh`, the directory has the Spark test resource files, but it does not have the Spark binaries. This works for running unit tests, but without binaries the doc generation script fails. You can see from this [PR](https://github.com/apache/incubator-gluten/pull/9499/files#diff-cc9b23a1706d7aac3ffaa94c96692d477c2cfb375b994d23289190d0f581e3c4R125), which is trying to add a GHA weekly run for the doc generation script, that it's needing to download the Spark binaries and then combine that with the test resource files from the `install_spark_resources.sh` directory.
I can mitigate these issues with a couple of changes to the `install_spark_resource.sh` script:
* Adding an optional second positional arg to the script for the spark dir (if it's not provided, it can still default to `/opt`)
* Extracting the bin/python dir from the spark .tgz file (right now the script only grabs `jars`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#issuecomment-2971708020
https://github.com/apache/incubator-gluten/issues/9328
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#issuecomment-2986172086
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#issuecomment-2971708403
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-9328][VL] Update GlutenSQLQueryTestSuite error handling for missing resource files and doc updates [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9969:
URL: https://github.com/apache/incubator-gluten/pull/9969#issuecomment-2977041383
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org