You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2023/01/12 09:24:45 UTC
[spark] branch master updated: [SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 04836babb7a [SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas
04836babb7a is described below
commit 04836babb7a1a2aafa7c65393c53c42937ef75a4
Author: Stefaan Lippens <st...@vito.be>
AuthorDate: Thu Jan 12 18:24:30 2023 +0900
[SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas
### What changes were proposed in this pull request?
See https://issues.apache.org/jira/browse/SPARK-41989 for in depth explanation
Short summary: `pyspark/pandas/__init__.py` uses, at import time, `logging.warning()` which might silently call `logging.basicConfig()`.
So by importing `pyspark.pandas` (directly or indirectly) a user might unknowingly break their own logging setup (e.g. when based on `logging.basicConfig()` or related). `logging.getLogger(...).warning()` does not trigger this behavior.
### Does this PR introduce _any_ user-facing change?
User-defined logging setups will be more predictable.
### How was this patch tested?
Manual testing so far.
I'm not sure it's worthwhile to cover this with a unit test
Closes #39516 from soxofaan/SPARK-41989-pyspark-pandas-logging-setup.
Authored-by: Stefaan Lippens <st...@vito.be>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/pandas/__init__.py | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/python/pyspark/pandas/__init__.py b/python/pyspark/pandas/__init__.py
index 5864025ad2a..980aeab2bee 100644
--- a/python/pyspark/pandas/__init__.py
+++ b/python/pyspark/pandas/__init__.py
@@ -47,9 +47,7 @@ if (
LooseVersion(pyarrow.__version__) >= LooseVersion("2.0.0")
and "PYARROW_IGNORE_TIMEZONE" not in os.environ
):
- import logging
-
- logging.warning(
+ warnings.warn(
"'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is required to "
"set this environment variable to '1' in both driver and executor sides if you use "
"pyarrow>=2.0.0. "
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org