You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/08/12 01:38:24 UTC
[PR] [SPARK-49195][CONNECT] Embed script level parsing logic into SparkSubmitCommandBuilder [spark]
HyukjinKwon opened a new pull request, #47702:
URL: https://github.com/apache/spark/pull/47702
### What changes were proposed in this pull request?
This PR embeds the configuration validation logic in scripts (added in https://github.com/apache/spark/pull/47402) to `SparkSubmitCommandBuilder` to be consistent.
### Why are the changes needed?
To have the consistent one place that handles configurations.
Those configuration validation at the script level was a temporary workaround that we should remove it away.
### Does this PR introduce _any_ user-facing change?
No, refactoring.
### How was this patch tested?
Manually tested:
```
./bin/spark-shell --remote "local[*]"
...
spark.range(10).show()
```
```
./sbin/start-connect-server.sh --wait
...
./bin/spark-shell --remote "sc://localhost"
...
spark.range(10).show()
```
```
./bin/spark-shell --remote "local[*]" --conf spark.sql.ansi.enabled=false
...
spark.conf.get("spark.sql.ansi.enabled")
```
```
./bin/spark-shell --remote "local[*]" --conf spark.sql.ansi.enabled=false --conf spark.abcabc=abc --conf spark.sql.debug=true
...
spark.conf.get("spark.sql.debug")
spark.conf.get("spark.sql.ansi.enabled")
```
```
./sbin/start-connect-server.sh --wait
...
./bin/spark-shell --remote "sc://localhost" --conf spark.sql.ansi.enabled=false --conf spark.abcabc=abc --conf spark.sql.debug=true
...
spark.conf.get("spark.sql.debug")
spark.conf.get("spark.sql.ansi.enabled")
```
```
SPARK_REMOTE="sc://localhost" ./bin/spark-shell
...
spark.range(1).collect()
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49195][CONNECT] Embed script level parsing logic into SparkSubmitCommandBuilder [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #47702: [SPARK-49195][CONNECT] Embed script level parsing logic into SparkSubmitCommandBuilder
URL: https://github.com/apache/spark/pull/47702
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49195][CONNECT] Embed script level parsing logic into SparkSubmitCommandBuilder [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #47702:
URL: https://github.com/apache/spark/pull/47702#issuecomment-2284584812
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49195][CONNECT] Embed script level parsing logic into SparkSubmitCommandBuilder [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #47702:
URL: https://github.com/apache/spark/pull/47702#discussion_r1713191075
##########
bin/spark-shell:
##########
@@ -44,42 +44,11 @@ Scala REPL options, Spark Classic only:
# through spark.driver.extraClassPath is not automatically propagated.
SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dscala.usejavacp=true"
-# In order to start Spark Connect shell, we should identify if spark.remote
-# or --remote is set. Spark Connect does not support loading configurations
-# yet.
-connect_shell=false
-cur_arg="$0"
-for arg in "${@:1}"
-do
- # --conf spark.remote=... or -c spark.remote=...
- if [[ $cur_arg == "--conf" || $cur_arg == "-c" ]]; then
- if [[ $arg == "spark.remote"* ]]; then
- connect_shell=true
- fi
- fi
-
- # --conf=spark.remote=... or -c=spark.remote=...
- if [[ $arg == "--conf=spark.remote"* || $arg == "-c=spark.remote"* ]]; then
- connect_shell=true
- fi
-
- # --remote= or --remote
- if [[ $arg == "--remote"* ]]; then
- connect_shell=true
- fi
- cur_arg=$arg
-done
-
-if [ -n "${SPARK_REMOTE}" ]; then
- connect_shell=true
-fi
-
function main() {
- if $connect_shell; then
- export SPARK_SUBMIT_OPTS
- export SPARK_CONNECT_SHELL=1
- "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.sql.application.ConnectRepl --name "Connect shell" "$@"
- elif $cygwin; then
+ export SPARK_SCALA_SHELL=1
Review Comment:
This will be removed with https://github.com/apache/spark/pull/47434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org