You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cdkrot (via GitHub)" <gi...@apache.org> on 2023/07/26 20:52:46 UTC
[GitHub] [spark] cdkrot opened a new pull request, #42175: [SPARK-44558] Export Spark Log Level
cdkrot opened a new pull request, #42175:
URL: https://github.com/apache/spark/pull/42175
### What changes were proposed in this pull request?
Export Spark Connect Log Level in pyspark.
### Why are the changes needed?
This is convenient for software dependent on spark-connect so it can be possible to enable debug logging just in one place.
### Does this PR introduce _any_ user-facing change?
New api is suggested.
### How was this patch tested?
Checked it works from shell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42175: [SPARK-44558] Export Spark Log Level
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42175:
URL: https://github.com/apache/spark/pull/42175#discussion_r1275597694
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -17,6 +17,7 @@
__all__ = [
"ChannelBuilder",
"SparkConnectClient",
+ "spark_connect_log_level",
Review Comment:
```suggestion
"getLogLevel",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42175: [SPARK-44558] Export Spark Log Level
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42175:
URL: https://github.com/apache/spark/pull/42175#discussion_r1275597200
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -114,6 +115,19 @@ def _configure_logging() -> logging.Logger:
logger = _configure_logging()
+def spark_connect_log_level() -> Optional[int]:
Review Comment:
```suggestion
def getLogLevel() -> Optional[int]:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42175: [SPARK-44558][CONNECT][PYTHON] Export Spark Log Level
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42175:
URL: https://github.com/apache/spark/pull/42175#issuecomment-1654887577
Merged to master and branch-3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a diff in pull request #42175: [SPARK-44558] Export Spark Log Level
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #42175:
URL: https://github.com/apache/spark/pull/42175#discussion_r1275596325
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -114,6 +115,19 @@ def _configure_logging() -> logging.Logger:
logger = _configure_logging()
+def spark_connect_log_level() -> Optional[int]:
+ """
+ This returns this log level as integer, or none (if no logging is enabled).
+
+ Spark Connect logging can be configured with environment variable 'SPARK_CONNECT_LOG_LEVEL'
+ .. versionadded:: 3.5.0
+ """
+
+ if logger.disabled:
+ return None
+ return logger.level
Review Comment:
```suggestion
if not logger.disabled:
return logger.level
```
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -114,6 +115,19 @@ def _configure_logging() -> logging.Logger:
logger = _configure_logging()
+def spark_connect_log_level() -> Optional[int]:
+ """
+ This returns this log level as integer, or none (if no logging is enabled).
+
+ Spark Connect logging can be configured with environment variable 'SPARK_CONNECT_LOG_LEVEL'
Review Comment:
```suggestion
Spark Connect logging can be configured with environment variable 'SPARK_CONNECT_LOG_LEVEL'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cdkrot commented on a diff in pull request #42175: [SPARK-44558][CONNECT][PYTHON] Export Spark Log Level
Posted by "cdkrot (via GitHub)" <gi...@apache.org>.
cdkrot commented on code in PR #42175:
URL: https://github.com/apache/spark/pull/42175#discussion_r1276026310
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -114,6 +115,19 @@ def _configure_logging() -> logging.Logger:
logger = _configure_logging()
+def spark_connect_log_level() -> Optional[int]:
+ """
+ This returns this log level as integer, or none (if no logging is enabled).
+
+ Spark Connect logging can be configured with environment variable 'SPARK_CONNECT_LOG_LEVEL'
+ .. versionadded:: 3.5.0
+ """
+
+ if logger.disabled:
+ return None
+ return logger.level
Review Comment:
Need return statement for codestyle:
```
annotations failed mypy checks:
python/pyspark/sql/connect/client/core.py:118: error: Missing return statement [return]
```
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -104,7 +105,7 @@ def _configure_logging() -> logging.Logger:
# Check the environment variables for log levels:
if "SPARK_CONNECT_LOG_LEVEL" in os.environ:
- logger.setLevel(os.getenv("SPARK_CONNECT_LOG_LEVEL", "error").upper())
Review Comment:
Note original version has a typo, the default value is never used due to above 'if'. I'm just removing the typo without changing the logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #42175: [SPARK-44558][CONNECT][PYTHON] Export Spark Log Level
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #42175: [SPARK-44558][CONNECT][PYTHON] Export Spark Log Level
URL: https://github.com/apache/spark/pull/42175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org