You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2024/11/22 02:58:13 UTC
[PR] [DO-NOT-MERGE] Support SQL API mode [spark]
HyukjinKwon opened a new pull request, #48930:
URL: https://github.com/apache/spark/pull/48930
### What changes were proposed in this pull request?
TBD
### Why are the changes needed?
TBD
### Does this PR introduce _any_ user-facing change?
TBD
### How was this patch tested?
TBD
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode, prototype 1 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #48930: [DO-NOT-MERGE] Support SQL API mode, prototype 1
URL: https://github.com/apache/spark/pull/48930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1866935353
##########
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala:
##########
@@ -90,7 +95,11 @@ object PythonRunner {
env.put(s"PYSPARK_REMOTE_INIT_CONF_$idx", compact(render(group)))
}
}
- sparkConf.getOption("spark.remote").foreach(url => env.put("SPARK_REMOTE", url))
+ if (isAPIModeClassic) {
Review Comment:
This implies that the default API mode is connect... That is a suggestion...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1866937430
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -620,8 +620,17 @@ object SparkConnectClient {
* Configure the builder using the env SPARK_REMOTE environment variable.
*/
def loadFromEnvironment(): Builder = {
+ lazy val isAPIModeConnect = Option(System.getProperty("spark.api.mode"))
+ .exists(_.toLowerCase(Locale.ROOT) == "connect")
Option(System.getProperty("spark.remote")) // Set from Spark Submit
.orElse(sys.env.get(SparkConnectClient.SPARK_REMOTE))
+ .orElse {
+ if (isAPIModeConnect) {
+ Option(System.getProperty("spark.master")).orElse(sys.env.get("MASTER"))
Review Comment:
A spark master URL is not compatible with a connect URL... How is this supposed to work. More generally, why should the client have to deal with spark master URLs, at this point in time you should be able to pass a valid remote URL.
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -620,8 +620,17 @@ object SparkConnectClient {
* Configure the builder using the env SPARK_REMOTE environment variable.
*/
def loadFromEnvironment(): Builder = {
+ lazy val isAPIModeConnect = Option(System.getProperty("spark.api.mode"))
+ .exists(_.toLowerCase(Locale.ROOT) == "connect")
Option(System.getProperty("spark.remote")) // Set from Spark Submit
.orElse(sys.env.get(SparkConnectClient.SPARK_REMOTE))
+ .orElse {
+ if (isAPIModeConnect) {
+ Option(System.getProperty("spark.master")).orElse(sys.env.get("MASTER"))
Review Comment:
A spark master URL is not compatible with a connect URL... How is this supposed to work? More generally, why should the client have to deal with spark master URLs, at this point in time you should be able to pass a valid remote URL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode, prototype 1 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48930:
URL: https://github.com/apache/spark/pull/48930#issuecomment-2522407420
Tests should pass now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1866936049
##########
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala:
##########
@@ -52,9 +53,13 @@ object PythonRunner {
// Format python file paths before adding them to the PYTHONPATH
val formattedPythonFile = formatPath(pythonFile)
val formattedPyFiles = resolvePyFiles(formatPaths(pyFiles))
+ lazy val isAPIModeClassic = sparkConf.getOption(
+ "spark.api.mode").exists(_.toLowerCase(Locale.ROOT) == "classic")
+ lazy val isAPIModeConnect = sparkConf.getOption(
+ "spark.api.mode").exists(_.toLowerCase(Locale.ROOT) == "connect")
var gatewayServer: Option[Py4JServer] = None
- if (sparkConf.getOption("spark.remote").isEmpty) {
+ if (sparkConf.getOption("spark.remote").isEmpty || isAPIModeClassic) {
Review Comment:
Can you please throw exceptions when you encounter incompatible configurations. For example: Classic should only work with master set, and not with remote set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1867272462
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -620,8 +620,17 @@ object SparkConnectClient {
* Configure the builder using the env SPARK_REMOTE environment variable.
*/
def loadFromEnvironment(): Builder = {
+ lazy val isAPIModeConnect = Option(System.getProperty("spark.api.mode"))
+ .exists(_.toLowerCase(Locale.ROOT) == "connect")
Option(System.getProperty("spark.remote")) // Set from Spark Submit
.orElse(sys.env.get(SparkConnectClient.SPARK_REMOTE))
+ .orElse {
+ if (isAPIModeConnect) {
+ Option(System.getProperty("spark.master")).orElse(sys.env.get("MASTER"))
Review Comment:
Oh yeah, this can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode, prototype 1 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48930:
URL: https://github.com/apache/spark/pull/48930#issuecomment-2522238521
Tests should pass now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode, prototype 1 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1870519689
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -620,8 +620,17 @@ object SparkConnectClient {
* Configure the builder using the env SPARK_REMOTE environment variable.
*/
def loadFromEnvironment(): Builder = {
+ lazy val isAPIModeConnect = Option(System.getProperty("spark.api.mode"))
+ .exists(_.toLowerCase(Locale.ROOT) == "connect")
Option(System.getProperty("spark.remote")) // Set from Spark Submit
.orElse(sys.env.get(SparkConnectClient.SPARK_REMOTE))
+ .orElse {
+ if (isAPIModeConnect) {
+ Option(System.getProperty("spark.master")).orElse(sys.env.get("MASTER"))
Review Comment:
Actually for now, we need this because we directly start ConnectRepl for Spark Connect shell. Once we have a layer for Spark Connect (e.g., at https://github.com/apache/spark/pull/49053), we would be able to separate it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1867263381
##########
core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala:
##########
@@ -52,9 +53,13 @@ object PythonRunner {
// Format python file paths before adding them to the PYTHONPATH
val formattedPythonFile = formatPath(pythonFile)
val formattedPyFiles = resolvePyFiles(formatPaths(pyFiles))
+ lazy val isAPIModeClassic = sparkConf.getOption(
+ "spark.api.mode").exists(_.toLowerCase(Locale.ROOT) == "classic")
+ lazy val isAPIModeConnect = sparkConf.getOption(
+ "spark.api.mode").exists(_.toLowerCase(Locale.ROOT) == "connect")
var gatewayServer: Option[Py4JServer] = None
- if (sparkConf.getOption("spark.remote").isEmpty) {
+ if (sparkConf.getOption("spark.remote").isEmpty || isAPIModeClassic) {
Review Comment:
That is actually thrown (fail fast) at `SparkSubmitArguments.validateSubmitArguments`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48930:
URL: https://github.com/apache/spark/pull/48930#discussion_r1866938190
##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala:
##########
@@ -620,8 +620,17 @@ object SparkConnectClient {
* Configure the builder using the env SPARK_REMOTE environment variable.
*/
def loadFromEnvironment(): Builder = {
+ lazy val isAPIModeConnect = Option(System.getProperty("spark.api.mode"))
Review Comment:
We should throw an error iif api mode is not connect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE] Support SQL API mode, prototype 1 [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48930:
URL: https://github.com/apache/spark/pull/48930#issuecomment-2550298987
gonna close this for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org