You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/10 14:22:18 UTC
[GitHub] [spark] Yikun opened a new pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Yikun opened a new pull request #32867:
URL: https://github.com/apache/spark/pull/32867
### What changes were proposed in this pull request?
Add path level discover for python unittests.
### Why are the changes needed?
Now we need to specify the python test cases by manually when we add a new testcase. Sometime, we forgot to add the testcase to module list, the testcase would not be executed.
Such as:
- pyspark-core pyspark.tests.test_pin_thread
- pyspark-pandas pyspark.pandas.tests.data_type_ops.testing_utils
Thus we need some auo-discover way to find all testcase rather than specified by manually.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add below code in end of `dev/sparktestsupport/modules.py`
```python
for m in sorted(all_modules):
for g in sorted(m.python_test_goals):
print(m.name, g)
```
Compare the result before and after.
https://www.diffchecker.com/gaMvbLmk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659799167
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Emm, looks like it a diffcult to add this doctest, the doctest requires real import of pyspark, so it would be failed when running the testcase in run-test.py, but we still could run the local test at least. So, I will add the doctest in next commits.
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Emm, looks like it a diffcult to add this doctest, the doctest requires real import of pyspark, so it would be failed when running the testcase in run-test.py.
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Yep
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
https://github.com/apache/spark/pull/32867/commits/ef460a70716b3a4b4c9835b8d7fd73f839500e24#diff-baa74aac64952412da46580040412f216c4174a61a9f9bd201d41f0b84a50254R783-R793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862154577
**[Test build #139862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139862/testReport)** for PR 32867 at commit [`9f4388a`](https://github.com/apache/spark/commit/9f4388a6ea8cb1f038b00a7f72ce92dd6bbb7845).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652290250
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,28 @@
import itertools
import os
import re
+import glob
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ if not paths:
+ return set([])
+ tests = set([])
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the test*.py in every path
+ files = glob.glob(os.path.join(pyspark_path, path, "test_*.py"))
+ for f in files:
+ # Convert 'pyspark_path/pyspark/tests/test_abc.py' to 'pyspark.tests.test_abc'
+ file2module = os.path.relpath(f, pyspark_path)[:-3].replace("/", ".")
Review comment:
Hmm .. I like the idea but the implementation is slightly flaky ... e.g.) I wonder if we can import pyspark and go through the sub packages manually instead of going through the files ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862107926
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861150138
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44317/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858675605
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139642/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861132170
**[Test build #139791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139791/testReport)** for PR 32867 at commit [`23f2c87`](https://github.com/apache/spark/commit/23f2c87c7ee16d433b3d59dbb71dae00203fea97).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r651402839
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,28 @@
import itertools
import os
import re
+import glob
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ if not paths:
+ return set([])
+ tests = set([])
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the test*.py in every path
+ files = glob.glob(os.path.join(pyspark_path, path, "test*.py"))
Review comment:
Thanks for reminder, it should be test_*.py"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864329093
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870201584
Just wondering we can only verify it manually? If by any chance, some tests are not found, can we easily know it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin edited a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
ueshin edited a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870809700
I'm sorry the the late review, but is this triggering unit tests? I don't see any test cases running.
- https://github.com/apache/spark/runs/2940354315
```
========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.6', 'python3.9', 'pypy3']
Will test the following Python modules: ['pyspark-pandas-slow']
python3.6 python_implementation is CPython
python3.6 version is: Python 3.6.13
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.5
pypy3 python_implementation is PyPy
pypy3 version is: Python 3.7.10 (7.3.5+dfsg-1~ppa1~ubuntu20.04, May 23 2021, 14:57:07)
[PyPy 7.3.5 with GCC 9.3.0]
Starting test(python3.6): pyspark.pandas.generic
Starting test(python3.6): pyspark.pandas.frame
Finished test(python3.6): pyspark.pandas.generic (127s)
Starting test(python3.6): pyspark.pandas.series
Finished test(python3.6): pyspark.pandas.series (197s)
Starting test(python3.9): pyspark.pandas.frame
Finished test(python3.6): pyspark.pandas.frame (465s)
Starting test(python3.9): pyspark.pandas.generic
Finished test(python3.9): pyspark.pandas.generic (120s)
Starting test(python3.9): pyspark.pandas.series
Finished test(python3.9): pyspark.pandas.frame (440s)
Finished test(python3.9): pyspark.pandas.series (179s)
Tests passed in 766 seconds
```
- before the commit https://github.com/apache/spark/runs/2939324068
```
========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.6', 'python3.9', 'pypy3']
Will test the following Python modules: ['pyspark-pandas-slow']
python3.6 python_implementation is CPython
python3.6 version is: Python 3.6.13
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.5
pypy3 python_implementation is PyPy
pypy3 version is: Python 3.7.10 (7.3.5+dfsg-1~ppa1~ubuntu20.04, May 23 2021, 14:57:07)
[PyPy 7.3.5 with GCC 9.3.0]
Starting test(python3.6): pyspark.pandas.tests.indexes.test_base
Starting test(python3.6): pyspark.pandas.tests.indexes.test_datetime
Finished test(python3.6): pyspark.pandas.tests.indexes.test_datetime (262s)
Starting test(python3.6): pyspark.pandas.tests.test_dataframe
Finished test(python3.6): pyspark.pandas.tests.indexes.test_base (301s)
Starting test(python3.6): pyspark.pandas.tests.test_groupby
Finished test(python3.6): pyspark.pandas.tests.test_dataframe (394s) ... 3 tests were skipped
Starting test(python3.6): pyspark.pandas.tests.test_indexing
Finished test(python3.6): pyspark.pandas.tests.test_indexing (180s) ... 5 tests were skipped
Starting test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames
Finished test(python3.6): pyspark.pandas.tests.test_groupby (559s)
Starting test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames_groupby
Finished test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (202s)
Starting test(python3.6): pyspark.pandas.tests.test_series
Finished test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames (392s) ... 2 tests were skipped
Starting test(python3.6): pyspark.pandas.tests.test_stats
Finished test(python3.6): pyspark.pandas.tests.test_stats (102s)
Starting test(python3.9): pyspark.pandas.tests.indexes.test_base
Finished test(python3.6): pyspark.pandas.tests.test_series (301s) ... 2 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.indexes.test_datetime
Finished test(python3.9): pyspark.pandas.tests.indexes.test_base (270s)
Starting test(python3.9): pyspark.pandas.tests.test_dataframe
Finished test(python3.9): pyspark.pandas.tests.indexes.test_datetime (243s)
Starting test(python3.9): pyspark.pandas.tests.test_groupby
Finished test(python3.9): pyspark.pandas.tests.test_dataframe (368s) ... 1 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.test_indexing
Finished test(python3.9): pyspark.pandas.tests.test_groupby (514s)
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames
Finished test(python3.9): pyspark.pandas.tests.test_indexing (200s) ... 5 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (193s)
Starting test(python3.9): pyspark.pandas.tests.test_series
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames (430s)
Starting test(python3.9): pyspark.pandas.tests.test_stats
Finished test(python3.9): pyspark.pandas.tests.test_series (291s) ... 1 tests were skipped
Starting test(python3.6): pyspark.pandas.frame
Finished test(python3.9): pyspark.pandas.tests.test_stats (104s)
Starting test(python3.6): pyspark.pandas.generic
Finished test(python3.6): pyspark.pandas.generic (113s)
Starting test(python3.6): pyspark.pandas.series
Finished test(python3.6): pyspark.pandas.series (162s)
Starting test(python3.9): pyspark.pandas.frame
Finished test(python3.6): pyspark.pandas.frame (386s)
Starting test(python3.9): pyspark.pandas.generic
Finished test(python3.9): pyspark.pandas.generic (102s)
Starting test(python3.9): pyspark.pandas.series
Finished test(python3.9): pyspark.pandas.series (152s)
Finished test(python3.9): pyspark.pandas.frame (364s)
Tests passed in 3301 seconds
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659540353
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864329084
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864120019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864347581
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44540/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858675571
**[Test build #139642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139642/testReport)** for PR 32867 at commit [`be3877a`](https://github.com/apache/spark/commit/be3877a49203bc4f68234476b9b12802e4d4e1b0).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` file2class = os.path.relpath(f, pyspark_path)[:-3].replace(\"/\", \".\")`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-869430292
> BTW, this change will likely affect many other vendors who maintain their forks so I will take a close look few more times. Thanks for bearing with me in advance .. ;-).
Sure, thanks for your patient review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862187930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139865/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858754898
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862107926
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139854/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864335412
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861161585
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44317/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864329093
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870200640
Looks pretty good to me too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862217134
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44383/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858674433
**[Test build #139642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139642/testReport)** for PR 32867 at commit [`be3877a`](https://github.com/apache/spark/commit/be3877a49203bc4f68234476b9b12802e4d4e1b0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864357505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858668874
cc @HyukjinKwon
It would be good if you could give some inputs on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864347588
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44540/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870200303
@Yikun sorry but mind resolving the conflicts please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862154577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864334363
**[Test build #140004 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140004/testReport)** for PR 32867 at commit [`17dda87`](https://github.com/apache/spark/commit/17dda875e3195b4909fd4d711f62b40bebe7c9a9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class ShuffleBlockInfo(shuffleId: Int, mapId: Long) `
* `case class ShuffleMergedDataBlockId(appId: String, shuffleId: Int, reduceId: Int) extends BlockId `
* `case class ShuffleMergedIndexBlockId(`
* `case class ShuffleMergedMetaBlockId(`
* `class DecimalOps(FractionalOps):`
* `class KubernetesLocalDiskShuffleDataIO(sparkConf: SparkConf) extends ShuffleDataIO `
* `class KubernetesLocalDiskShuffleExecutorComponents(sparkConf: SparkConf)`
* `case class TempResolvedColumn(child: Expression, nameParts: Seq[String]) extends UnaryExpression`
* `class ExpressionContainmentOrdering extends Ordering[Expression] `
* `case class MakeDTInterval(`
* `case class MakeYMInterval(years: Expression, months: Expression)`
* `case class DayTimeIntervalType(startField: Byte, endField: Byte) extends AtomicType `
* `case class YearMonthIntervalType(startField: Byte, endField: Byte) extends AtomicType `
* `public class ParquetVectorUpdaterFactory `
* `case class CommandResult(`
* `case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric `
* `case class StateStoreCustomSizeMetric(name: String, desc: String) extends StateStoreCustomMetric `
* `case class StateStoreCustomTimingMetric(name: String, desc: String) extends StateStoreCustomMetric `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659422872
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
+ """
+
+ def add_suite(testcases, modules, slow):
+ """Gather the suite module names"""
+ if hasattr(testcases, '__iter__'):
+ for test_case in testcases:
+ add_suite(test_case, modules, slow)
+ else:
+ name = testcases.__module__
+ module = _get_module_from_name(name)
+ if slow and hasattr(module, '_slow_test'):
+ modules.add(name)
+ if not slow and not hasattr(module, '_slow_test'):
+ modules.add(name)
+
+ if not paths:
+ return []
+ _modules = set()
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the unittest in every path
+ suite = unittest.defaultTestLoader.discover(
+ os.path.join(pyspark_path, path),
+ top_level_dir=pyspark_path
+ )
+ add_suite(suite, _modules, discover_slow)
+
+ return sorted(list(_modules))
Review comment:
```suggestion
return sorted(list(modules))
```
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
+ """
+
+ def add_suite(testcases, modules, slow):
+ """Gather the suite module names"""
+ if hasattr(testcases, '__iter__'):
+ for test_case in testcases:
+ add_suite(test_case, modules, slow)
+ else:
+ name = testcases.__module__
+ module = _get_module_from_name(name)
+ if slow and hasattr(module, '_slow_test'):
+ modules.add(name)
+ if not slow and not hasattr(module, '_slow_test'):
+ modules.add(name)
+
+ if not paths:
+ return []
+ _modules = set()
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the unittest in every path
+ suite = unittest.defaultTestLoader.discover(
+ os.path.join(pyspark_path, path),
+ top_level_dir=pyspark_path
+ )
+ add_suite(suite, _modules, discover_slow)
Review comment:
```suggestion
add_suite(suite, modules, discover_slow)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659423134
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -608,58 +594,19 @@ def __hash__(self):
"pyspark.pandas.spark.accessors",
"pyspark.pandas.spark.utils",
"pyspark.pandas.typedef.typehints",
- # unittests
- "pyspark.pandas.tests.data_type_ops.test_base",
- "pyspark.pandas.tests.data_type_ops.test_binary_ops",
- "pyspark.pandas.tests.data_type_ops.test_boolean_ops",
- "pyspark.pandas.tests.data_type_ops.test_categorical_ops",
- "pyspark.pandas.tests.data_type_ops.test_complex_ops",
- "pyspark.pandas.tests.data_type_ops.test_date_ops",
- "pyspark.pandas.tests.data_type_ops.test_datetime_ops",
- "pyspark.pandas.tests.data_type_ops.test_decimal_ops",
- "pyspark.pandas.tests.data_type_ops.test_null_ops",
- "pyspark.pandas.tests.data_type_ops.test_num_ops",
- "pyspark.pandas.tests.data_type_ops.test_string_ops",
- "pyspark.pandas.tests.data_type_ops.test_udt_ops",
- "pyspark.pandas.tests.indexes.test_category",
- "pyspark.pandas.tests.plot.test_frame_plot",
- "pyspark.pandas.tests.plot.test_frame_plot_matplotlib",
- "pyspark.pandas.tests.plot.test_frame_plot_plotly",
- "pyspark.pandas.tests.plot.test_series_plot",
- "pyspark.pandas.tests.plot.test_series_plot_matplotlib",
- "pyspark.pandas.tests.plot.test_series_plot_plotly",
- "pyspark.pandas.tests.test_categorical",
- "pyspark.pandas.tests.test_config",
- "pyspark.pandas.tests.test_csv",
- "pyspark.pandas.tests.test_dataframe_conversion",
- "pyspark.pandas.tests.test_dataframe_spark_io",
- "pyspark.pandas.tests.test_default_index",
- "pyspark.pandas.tests.test_expanding",
- "pyspark.pandas.tests.test_extension",
- "pyspark.pandas.tests.test_frame_spark",
- "pyspark.pandas.tests.test_indexops_spark",
- "pyspark.pandas.tests.test_internal",
- "pyspark.pandas.tests.test_namespace",
- "pyspark.pandas.tests.test_numpy_compat",
- "pyspark.pandas.tests.test_ops_on_diff_frames_groupby_expanding",
- "pyspark.pandas.tests.test_ops_on_diff_frames_groupby_rolling",
- "pyspark.pandas.tests.test_repr",
- "pyspark.pandas.tests.test_reshape",
- "pyspark.pandas.tests.test_rolling",
- "pyspark.pandas.tests.test_series_conversion",
- "pyspark.pandas.tests.test_series_datetime",
- "pyspark.pandas.tests.test_series_string",
- "pyspark.pandas.tests.test_sql",
- "pyspark.pandas.tests.test_typedef",
- "pyspark.pandas.tests.test_utils",
- "pyspark.pandas.tests.test_window",
- ],
+ ] + _discover_python_unittests(
+ "pyspark/pandas/tests",
+ "pyspark/pandas/tests/data_type_ops",
+ "pyspark/pandas/tests/indexes",
+ "pyspark/pandas/tests/plot"
Review comment:
qq: why can't we just put `pyspark/pandas/tests` alone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659422686
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
+ """
+
+ def add_suite(testcases, modules, slow):
+ """Gather the suite module names"""
+ if hasattr(testcases, '__iter__'):
Review comment:
nit: maybe:
```python
from collections.abc import Iterable
if isinstance(testcases, Iterable):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-860817989
CC @ueshin @itholic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858668874
cc @HyukjinKwon @xinrong-databricks
It would be good if you could give some inputs on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652421042
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -16,13 +16,65 @@
#
from functools import total_ordering
+from importlib import import_module
+import inspect
import itertools
import os
+from pkgutil import iter_modules
import re
+import unittest
+
+from sparktestsupport import SPARK_HOME
+
all_modules = []
+def _contain_unittests_class(module_name):
+ """
+ Check if the module with specific module_name has classes are derived from unittest.TestCase.
+
+ Such as:
+ pyspark.tests.test_appsubmit, it will return True, because there is SparkSubmitTests which is
+ included under the module of pyspark.tests.test_appsubmit, inherits from unittest.TestCase.
+ ``
+
+ :param module_name: the complete name of module to be checked.
+ :return: True if contains unittest classes otherwise False.
+ An ``ModuleNotFoundError`` will raise if the module is not found
+ """
+ _module = import_module(module_name)
Review comment:
```Python
Traceback (most recent call last):
File "./dev/run-tests.py", line 32, in <module>
import sparktestsupport.modules as modules
File "/home/runner/work/spark/spark/dev/sparktestsupport/modules.py", line 425, in <module>
pyspark_core = Module(
File "/home/runner/work/spark/spark/dev/sparktestsupport/modules.py", line 122, in __init__
discovered_goals = _discover_python_unittests(python_test_paths)
File "/home/runner/work/spark/spark/dev/sparktestsupport/modules.py", line 73, in _discover_python_unittests
if _contain_unittests_class(module.name):
File "/home/runner/work/spark/spark/dev/sparktestsupport/modules.py", line 46, in _contain_unittests_class
_module = import_module(module_name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'pyspark'
```
It should be changed to path based.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862107867
**[Test build #139854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139854/testReport)** for PR 32867 at commit [`eb835c5`](https://github.com/apache/spark/commit/eb835c5d73b4f844cdd6dff1fd37a2317079c5e6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-859217722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862187894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r649236682
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -35,7 +53,7 @@ class Module(object):
def __init__(self, name, dependencies, source_file_regexes, build_profile_flags=(),
environ=None, sbt_test_goals=(), python_test_goals=(),
excluded_python_implementations=(), test_tags=(), should_run_r_tests=False,
- should_run_build_tests=False):
+ should_run_build_tests=False, python_test_paths=(), python_excluded_tests=()):
Review comment:
`python_excluded_tests` is for something like I want to running all tests except some slow case. See L607 and L621.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-872072514
The root reason of failed to discover is that the deps of PySpark modules is not installed, so we get the wrong list when we do discover.
I re-propose it in https://github.com/apache/spark/pull/33174 , the mainly changed:
- add the error check for discover to make sure the discover work, that means if the error like this pr happend would be raised with exception.
- Move the discover from `dev/run-tests.py` to `python/run-tests.py`, we don't need to discover python test and install python test deps in other modules.
- Add doctest to make sure discover work as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864143367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-868934485
@HyukjinKwon Ready for review, it would be good if you could take a look again. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r654349836
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,48 @@
import itertools
import os
import re
+import unittest
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ """
+ Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ :param paths: paths of module to be discovered.
+ :return: A set of complete test module name discovered udner the paths
Review comment:
We should use numpydoc string style ... Feel free to make another PR to fix other docstrings later ..
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -35,10 +73,9 @@ class Module(object):
def __init__(self, name, dependencies, source_file_regexes, build_profile_flags=(),
environ=None, sbt_test_goals=(), python_test_goals=(),
excluded_python_implementations=(), test_tags=(), should_run_r_tests=False,
- should_run_build_tests=False):
+ should_run_build_tests=False, python_test_paths=(), python_excluded_tests=()):
"""
Define a new module.
-
Review comment:
I think we should keep this newline between description and parameter descriptions.
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -58,14 +95,19 @@ def __init__(self, name, dependencies, source_file_regexes, build_profile_flags=
is not explicitly changed.
:param should_run_r_tests: If true, changes in this module will trigger all R tests.
:param should_run_build_tests: If true, changes in this module will trigger build tests.
+ :param python_test_paths: A set of python test paths to be discovered
+ :param python_excluded_tests: A set of excluded Python tests
"""
self.name = name
self.dependencies = dependencies
self.source_file_prefixes = source_file_regexes
self.sbt_test_goals = sbt_test_goals
self.build_profile_flags = build_profile_flags
self.environ = environ or {}
- self.python_test_goals = python_test_goals
+ discovered_goals = _discover_python_unittests(python_test_paths)
+ # Final goals = Manual specified goals + Discoverd goals - Excluded goals
+ all_goals = set(python_test_goals) | set(discovered_goals)
+ self.python_test_goals = sorted(list(set(all_goals) - set(python_excluded_tests)))
Review comment:
@Yikun what about we add a private variable at these test slow modules, and detect them by checking if that variable is defined?
For example,
```diff
diff --git a/python/pyspark/pandas/tests/test_dataframe.py b/python/pyspark/pandas/tests/test_dataframe.py
index e54b7835c22..22f2b5da93e 100644
--- a/python/pyspark/pandas/tests/test_dataframe.py
+++ b/python/pyspark/pandas/tests/test_dataframe.py
@@ -50,6 +50,10 @@ from pyspark.testing.sqlutils import SQLTestUtils
from pyspark.pandas.utils import name_like_string
+# This is used in run-tests.py blah blah. see modules.py blah blah
+_slow_test = True
+
+
class DataFrameTest(PandasOnSparkTestCase, SQLTestUtils):
@property
def pdf(self):
```
and then we can filter like:
```python
is_slow_test = hasattr(module, "_slow_test")
```
In this way, we can remove `pyspark_pandas_slow_unittests` define here too.
Also, can we remove `python_test_paths` and `python_excluded_tests` here too?
We could run `_discover_python_unittests` and directly assign to `python_test_goals` I guess. For example,
```python
python_test_goals = [
...
] + _discover_python_unittests("pyspark/tests")
```
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -488,12 +495,10 @@ def __hash__(self):
python_test_goals=[
# doctests
"pyspark.streaming.util",
Review comment:
BTW, do you plan to do it for doctests later? Doing it in a separate PR is fine. I'm just curious about your plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659422401
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
It is possible to add a simple doctests at https://github.com/apache/spark/blob/master/dev/run-tests.py? The doctests there are ran before running the script.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32867:
URL: https://github.com/apache/spark/pull/32867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861169623
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44317/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870201584
Just wondering we can only verify it manually? If by any chance, some tests are not found, can we easily know it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659422752
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
+ """
+
+ def add_suite(testcases, modules, slow):
+ """Gather the suite module names"""
+ if hasattr(testcases, '__iter__'):
+ for test_case in testcases:
+ add_suite(test_case, modules, slow)
+ else:
+ name = testcases.__module__
+ module = _get_module_from_name(name)
+ if slow and hasattr(module, '_slow_test'):
+ modules.add(name)
+ if not slow and not hasattr(module, '_slow_test'):
+ modules.add(name)
+
+ if not paths:
+ return []
+ _modules = set()
Review comment:
```suggestion
modules = set()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659799167
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Emm, looks like it a diffcult to add this doctest, the doctest requires real import of pyspark, so it would be failed when running the testcase in run-test.py.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-871032645
Thanks for revert for it. I'm so sorry for the trouble, I will recheck and fix soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652290945
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,28 @@
import itertools
import os
import re
+import glob
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ if not paths:
+ return set([])
+ tests = set([])
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the test*.py in every path
+ files = glob.glob(os.path.join(pyspark_path, path, "test_*.py"))
+ for f in files:
+ # Convert 'pyspark_path/pyspark/tests/test_abc.py' to 'pyspark.tests.test_abc'
+ file2module = os.path.relpath(f, pyspark_path)[:-3].replace("/", ".")
Review comment:
or .. define a private variable in each file, and decide which tests are slow, etc. based on that .. ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659799167
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Emm, looks like it a diffcult to add this doctest, the doctest requires real import of pyspark, so it would be failed when running the testcase in run-test.py, but we still could run the local test at least. So, I will add the doctest in next commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-859245463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-871004830
Uhoh, thanks @ueshin for reverting this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864323338
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659541568
##########
File path: python/pyspark/pandas/tests/test_stats.py
##########
@@ -31,6 +31,11 @@
from pyspark.testing.sqlutils import SQLTestUtils
+# This is used in run-tests.py to discover the slow test. See more in the doc of
+# _discover_python_unittests of dev/sparktestsupport/modules.py
+_slow_test = True
Review comment:
Sure, it make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870847052
Reverted. https://github.com/apache/spark/commit/1f6e2f55d7896c9128f80a8f1ed4c317244d013b
@Yikun Could you submit another PR to fix it? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870221373
> Just wondering we can only verify it manually? If by any chance, some tests are not found, can we easily know it?
It just same as before, if we forgot to add the path of tests, these tests would not be ran.
But compare to original implementations, the path level discover need less redundant work to add the test file name one by one, and also it decrease the possibility of forgetting to add test on some level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-859245463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858674433
**[Test build #139642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139642/testReport)** for PR 32867 at commit [`be3877a`](https://github.com/apache/spark/commit/be3877a49203bc4f68234476b9b12802e4d4e1b0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870410762
Merged to master, thanks @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862187930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139865/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862154622
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun edited a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun edited a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-872072514
The root reason of failed to discover is that the deps of PySpark modules is not installed, so we get the wrong list when we do discover.
I re-propose it in https://github.com/apache/spark/pull/33174 , the mainly changed:
- Add the error check for discover to make sure the discover work, that means if the error like this pr happend would be raised with exception.
- Move the discover from `dev/run-tests.py` to `python/run-tests.py`, we don't need to discover python test and install python test deps in other modules.
- Add doctest to make sure discover work as expected.
Thank you for your patience. I will fight this test discover to the end to make up for my mistake. : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864335412
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xinrong-databricks commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
xinrong-databricks commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r651096165
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,28 @@
import itertools
import os
import re
+import glob
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ if not paths:
+ return set([])
+ tests = set([])
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the test*.py in every path
+ files = glob.glob(os.path.join(pyspark_path, path, "test*.py"))
Review comment:
Do we intentionally include `pyspark/pandas/tests.data_type_ops.testing_utils`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861195984
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r654522707
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -58,14 +95,19 @@ def __init__(self, name, dependencies, source_file_regexes, build_profile_flags=
is not explicitly changed.
:param should_run_r_tests: If true, changes in this module will trigger all R tests.
:param should_run_build_tests: If true, changes in this module will trigger build tests.
+ :param python_test_paths: A set of python test paths to be discovered
+ :param python_excluded_tests: A set of excluded Python tests
"""
self.name = name
self.dependencies = dependencies
self.source_file_prefixes = source_file_regexes
self.sbt_test_goals = sbt_test_goals
self.build_profile_flags = build_profile_flags
self.environ = environ or {}
- self.python_test_goals = python_test_goals
+ discovered_goals = _discover_python_unittests(python_test_paths)
+ # Final goals = Manual specified goals + Discoverd goals - Excluded goals
+ all_goals = set(python_test_goals) | set(discovered_goals)
+ self.python_test_goals = sorted(list(set(all_goals) - set(python_excluded_tests)))
Review comment:
Yes, great suggestion, it makes module implementation more clean.
If we want to **only** discover the slow tests
```
python_test_goals = [
...
] + _discover_python_unittests("pyspark/tests", discover_slow=True)
```
otherwise will skip slow tests.
I complete it in my next commit. : )
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -488,12 +495,10 @@ def __hash__(self):
python_test_goals=[
# doctests
"pyspark.streaming.util",
Review comment:
Frankly, not yet, but I'd like to start to do some investigation. This unittestPR is started when I forgot many times to add the unittests in module. ^_^
And the `doctests` discover is a bit special, it's not subclass of doctest (just like the unitest), so looks like there are 2 potential solution:
1. Flag based. Add some flag like `_doc_test = True` in every test we want to test.
2. Special name based. Discover which module contains `_doctest` method (we should rename _test to _doctest to make doc test name unique).
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -488,12 +495,10 @@ def __hash__(self):
python_test_goals=[
# doctests
"pyspark.streaming.util",
Review comment:
BTW, do you think the footer method (excute doctest or unitests in __main__ with repeat code in many files) in python code is a good way enough? Do we consider do some re-arch (or to say, already have some work)on testing framework?
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -488,12 +495,10 @@ def __hash__(self):
python_test_goals=[
# doctests
"pyspark.streaming.util",
Review comment:
BTW, I'm not sure here is the good place to discussion. : )
Do you think the footer method (excute doctest or unitests in __main__ with repeat code in many files) in python code is a good way enough? Do we consider do some re-arch (or to say, already have some work) on testing framework?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861132170
**[Test build #139791 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139791/testReport)** for PR 32867 at commit [`23f2c87`](https://github.com/apache/spark/commit/23f2c87c7ee16d433b3d59dbb71dae00203fea97).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652400691
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,28 @@
import itertools
import os
import re
+import glob
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ if not paths:
+ return set([])
+ tests = set([])
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the test*.py in every path
+ files = glob.glob(os.path.join(pyspark_path, path, "test_*.py"))
+ for f in files:
+ # Convert 'pyspark_path/pyspark/tests/test_abc.py' to 'pyspark.tests.test_abc'
+ file2module = os.path.relpath(f, pyspark_path)[:-3].replace("/", ".")
Review comment:
> I wonder if we can import pyspark and go through the sub packages manually instead of going through the files ..
Yep, it's more accurate, I change the rules from "test_* discover" to "unittest module discover", it will only discover the modules which are interit from unittest.TestCase. See my latest change: https://github.com/apache/spark/pull/32867/commits/eb835c5d73b4f844cdd6dff1fd37a2317079c5e6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864120019
**[Test build #139996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139996/testReport)** for PR 32867 at commit [`b5871b0`](https://github.com/apache/spark/commit/b5871b06f23bc9abb4f83163bafc2f8e88101def).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r660252198
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
kk that's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870809700
Is this triggering unit tests? I don't see any test cases running. https://github.com/apache/spark/runs/2940354315
```
========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.6', 'python3.9', 'pypy3']
Will test the following Python modules: ['pyspark-pandas-slow']
python3.6 python_implementation is CPython
python3.6 version is: Python 3.6.13
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.5
pypy3 python_implementation is PyPy
pypy3 version is: Python 3.7.10 (7.3.5+dfsg-1~ppa1~ubuntu20.04, May 23 2021, 14:57:07)
[PyPy 7.3.5 with GCC 9.3.0]
Starting test(python3.6): pyspark.pandas.generic
Starting test(python3.6): pyspark.pandas.frame
Finished test(python3.6): pyspark.pandas.generic (127s)
Starting test(python3.6): pyspark.pandas.series
Finished test(python3.6): pyspark.pandas.series (197s)
Starting test(python3.9): pyspark.pandas.frame
Finished test(python3.6): pyspark.pandas.frame (465s)
Starting test(python3.9): pyspark.pandas.generic
Finished test(python3.9): pyspark.pandas.generic (120s)
Starting test(python3.9): pyspark.pandas.series
Finished test(python3.9): pyspark.pandas.frame (440s)
Finished test(python3.9): pyspark.pandas.series (179s)
Tests passed in 766 seconds
```
- before the commit https://github.com/apache/spark/runs/2939324068
```
========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.6', 'python3.9', 'pypy3']
Will test the following Python modules: ['pyspark-pandas-slow']
python3.6 python_implementation is CPython
python3.6 version is: Python 3.6.13
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.5
pypy3 python_implementation is PyPy
pypy3 version is: Python 3.7.10 (7.3.5+dfsg-1~ppa1~ubuntu20.04, May 23 2021, 14:57:07)
[PyPy 7.3.5 with GCC 9.3.0]
Starting test(python3.6): pyspark.pandas.tests.indexes.test_base
Starting test(python3.6): pyspark.pandas.tests.indexes.test_datetime
Finished test(python3.6): pyspark.pandas.tests.indexes.test_datetime (262s)
Starting test(python3.6): pyspark.pandas.tests.test_dataframe
Finished test(python3.6): pyspark.pandas.tests.indexes.test_base (301s)
Starting test(python3.6): pyspark.pandas.tests.test_groupby
Finished test(python3.6): pyspark.pandas.tests.test_dataframe (394s) ... 3 tests were skipped
Starting test(python3.6): pyspark.pandas.tests.test_indexing
Finished test(python3.6): pyspark.pandas.tests.test_indexing (180s) ... 5 tests were skipped
Starting test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames
Finished test(python3.6): pyspark.pandas.tests.test_groupby (559s)
Starting test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames_groupby
Finished test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (202s)
Starting test(python3.6): pyspark.pandas.tests.test_series
Finished test(python3.6): pyspark.pandas.tests.test_ops_on_diff_frames (392s) ... 2 tests were skipped
Starting test(python3.6): pyspark.pandas.tests.test_stats
Finished test(python3.6): pyspark.pandas.tests.test_stats (102s)
Starting test(python3.9): pyspark.pandas.tests.indexes.test_base
Finished test(python3.6): pyspark.pandas.tests.test_series (301s) ... 2 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.indexes.test_datetime
Finished test(python3.9): pyspark.pandas.tests.indexes.test_base (270s)
Starting test(python3.9): pyspark.pandas.tests.test_dataframe
Finished test(python3.9): pyspark.pandas.tests.indexes.test_datetime (243s)
Starting test(python3.9): pyspark.pandas.tests.test_groupby
Finished test(python3.9): pyspark.pandas.tests.test_dataframe (368s) ... 1 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.test_indexing
Finished test(python3.9): pyspark.pandas.tests.test_groupby (514s)
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames
Finished test(python3.9): pyspark.pandas.tests.test_indexing (200s) ... 5 tests were skipped
Starting test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames_groupby (193s)
Starting test(python3.9): pyspark.pandas.tests.test_series
Finished test(python3.9): pyspark.pandas.tests.test_ops_on_diff_frames (430s)
Starting test(python3.9): pyspark.pandas.tests.test_stats
Finished test(python3.9): pyspark.pandas.tests.test_series (291s) ... 1 tests were skipped
Starting test(python3.6): pyspark.pandas.frame
Finished test(python3.9): pyspark.pandas.tests.test_stats (104s)
Starting test(python3.6): pyspark.pandas.generic
Finished test(python3.6): pyspark.pandas.generic (113s)
Starting test(python3.6): pyspark.pandas.series
Finished test(python3.6): pyspark.pandas.series (162s)
Starting test(python3.9): pyspark.pandas.frame
Finished test(python3.6): pyspark.pandas.frame (386s)
Starting test(python3.9): pyspark.pandas.generic
Finished test(python3.9): pyspark.pandas.generic (102s)
Starting test(python3.9): pyspark.pandas.series
Finished test(python3.9): pyspark.pandas.series (152s)
Finished test(python3.9): pyspark.pandas.frame (364s)
Tests passed in 3301 seconds
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862107867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652488153
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -16,13 +16,65 @@
#
from functools import total_ordering
+from importlib import import_module
+import inspect
import itertools
import os
+from pkgutil import iter_modules
import re
+import unittest
+
+from sparktestsupport import SPARK_HOME
+
all_modules = []
+def _contain_unittests_class(module_name):
+ """
+ Check if the module with specific module_name has classes are derived from unittest.TestCase.
+
+ Such as:
+ pyspark.tests.test_appsubmit, it will return True, because there is SparkSubmitTests which is
+ included under the module of pyspark.tests.test_appsubmit, inherits from unittest.TestCase.
+ ``
+
+ :param module_name: the complete name of module to be checked.
+ :return: True if contains unittest classes otherwise False.
+ An ``ModuleNotFoundError`` will raise if the module is not found
+ """
+ _module = import_module(module_name)
Review comment:
@HyukjinKwon the module import based discoverd depends on pyspark and serveral dependency of pyspark, so it's a bit hard to discover the modules.
Do you have any idea on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870824460
Ur, this might be what I said before, how we make sure the tests are run...
@Yikun Can you check it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858768593
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864357104
**[Test build #140014 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140014/testReport)** for PR 32867 at commit [`74e3dd1`](https://github.com/apache/spark/commit/74e3dd124cb6a0f75d4a75639e38b3804519fa1e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r660252198
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
kk that's fine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861179891
**[Test build #139791 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139791/testReport)** for PR 32867 at commit [`23f2c87`](https://github.com/apache/spark/commit/23f2c87c7ee16d433b3d59dbb71dae00203fea97).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864143367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858768593
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-859219067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652400691
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,28 @@
import itertools
import os
import re
+import glob
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _discover_python_unittests(paths):
+ if not paths:
+ return set([])
+ tests = set([])
+ pyspark_path = os.path.join(SPARK_HOME, "python")
+ for path in paths:
+ # Discover the test*.py in every path
+ files = glob.glob(os.path.join(pyspark_path, path, "test_*.py"))
+ for f in files:
+ # Convert 'pyspark_path/pyspark/tests/test_abc.py' to 'pyspark.tests.test_abc'
+ file2module = os.path.relpath(f, pyspark_path)[:-3].replace("/", ".")
Review comment:
> I wonder if we can import pyspark and go through the sub packages manually instead of going through the files ..
Yep, it's more accurate, I change the rules from "test_* discover" to "unittest module discover", it will only discover the modules which are interit from unittest.TestCase. See my latest change: https://github.com/apache/spark/pull/32867/commits/9f4388a6ea8cb1f038b00a7f72ce92dd6bbb7845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870221373
> Just wondering we can only verify it manually? If by any chance, some tests are not found, can we easily know it?
It just same as before, if we forgot to add the path of tests, these tests would not be ran.
But compare to original implementations, the path level discover need less redundant work to add the test file name one by one, and also it decrease the possibility of forgetting to add test on some level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659543103
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
https://github.com/apache/spark/pull/32867/commits/ef460a70716b3a4b4c9835b8d7fd73f839500e24#diff-baa74aac64952412da46580040412f216c4174a61a9f9bd201d41f0b84a50254R783-R793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-869288341
BTW, this change will likely affect many other vendors who maintain their forks so I will take a close look few more times. Thanks for bearing with me in advance .. ;-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861169623
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44317/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870200303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r652488153
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -16,13 +16,65 @@
#
from functools import total_ordering
+from importlib import import_module
+import inspect
import itertools
import os
+from pkgutil import iter_modules
import re
+import unittest
+
+from sparktestsupport import SPARK_HOME
+
all_modules = []
+def _contain_unittests_class(module_name):
+ """
+ Check if the module with specific module_name has classes are derived from unittest.TestCase.
+
+ Such as:
+ pyspark.tests.test_appsubmit, it will return True, because there is SparkSubmitTests which is
+ included under the module of pyspark.tests.test_appsubmit, inherits from unittest.TestCase.
+ ``
+
+ :param module_name: the complete name of module to be checked.
+ :return: True if contains unittest classes otherwise False.
+ An ``ModuleNotFoundError`` will raise if the module is not found
+ """
+ _module = import_module(module_name)
Review comment:
@HyukjinKwon the module import based discoverd depends on pyspark and serveral dependency of pyspark, so it's a bit hard to discover the modules in ./dev/run-tests.py.
Do you have any idea on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659423301
##########
File path: python/pyspark/pandas/tests/test_stats.py
##########
@@ -31,6 +31,11 @@
from pyspark.testing.sqlutils import SQLTestUtils
+# This is used in run-tests.py to discover the slow test. See more in the doc of
+# _discover_python_unittests of dev/sparktestsupport/modules.py
+_slow_test = True
Review comment:
sorry for a forth and back. I think we can just remove the leading underscore since this is referred in another module.
```suggestion
is_slow_test = True
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864343365
**[Test build #140014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140014/testReport)** for PR 32867 at commit [`74e3dd1`](https://github.com/apache/spark/commit/74e3dd124cb6a0f75d4a75639e38b3804519fa1e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659540353
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
Yep
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -19,10 +19,67 @@
import itertools
import os
import re
+import unittest
+import sys
+
+from sparktestsupport import SPARK_HOME
all_modules = []
+def _get_module_from_name(name):
+ __import__(name)
+ return sys.modules[name]
+
+
+def _discover_python_unittests(*paths, discover_slow=False):
+ """Discover the python module which contains unittests under paths.
+
+ Such as:
+ ['pyspark/tests'], it will return the set of module name under the path of pyspark/tests, like
+ {'pyspark.tests.test_appsubmit', 'pyspark.tests.test_broadcast', ...}
+
+ Parameters
+ ----------
+ paths : str
+ Paths of modules to be discovered.
+ discover_slow : bool
+ If True, will only discover slow tests
+ If False, will discover all tests except slow tests
+
+ Returns
+ -------
+ A set of complete test module name discovered under specified paths
Review comment:
https://github.com/apache/spark/pull/32867/commits/ef460a70716b3a4b4c9835b8d7fd73f839500e24#diff-baa74aac64952412da46580040412f216c4174a61a9f9bd201d41f0b84a50254R783-R793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ueshin commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
ueshin commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-870845772
Let me revert this for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862187894
**[Test build #139865 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139865/testReport)** for PR 32867 at commit [`22015a5`](https://github.com/apache/spark/commit/22015a526ae9fe66482d477fbc858ec6d73d18b4).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864357505
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140014/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864347588
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44540/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
Yikun commented on a change in pull request #32867:
URL: https://github.com/apache/spark/pull/32867#discussion_r659541347
##########
File path: dev/sparktestsupport/modules.py
##########
@@ -608,58 +594,19 @@ def __hash__(self):
"pyspark.pandas.spark.accessors",
"pyspark.pandas.spark.utils",
"pyspark.pandas.typedef.typehints",
- # unittests
- "pyspark.pandas.tests.data_type_ops.test_base",
- "pyspark.pandas.tests.data_type_ops.test_binary_ops",
- "pyspark.pandas.tests.data_type_ops.test_boolean_ops",
- "pyspark.pandas.tests.data_type_ops.test_categorical_ops",
- "pyspark.pandas.tests.data_type_ops.test_complex_ops",
- "pyspark.pandas.tests.data_type_ops.test_date_ops",
- "pyspark.pandas.tests.data_type_ops.test_datetime_ops",
- "pyspark.pandas.tests.data_type_ops.test_decimal_ops",
- "pyspark.pandas.tests.data_type_ops.test_null_ops",
- "pyspark.pandas.tests.data_type_ops.test_num_ops",
- "pyspark.pandas.tests.data_type_ops.test_string_ops",
- "pyspark.pandas.tests.data_type_ops.test_udt_ops",
- "pyspark.pandas.tests.indexes.test_category",
- "pyspark.pandas.tests.plot.test_frame_plot",
- "pyspark.pandas.tests.plot.test_frame_plot_matplotlib",
- "pyspark.pandas.tests.plot.test_frame_plot_plotly",
- "pyspark.pandas.tests.plot.test_series_plot",
- "pyspark.pandas.tests.plot.test_series_plot_matplotlib",
- "pyspark.pandas.tests.plot.test_series_plot_plotly",
- "pyspark.pandas.tests.test_categorical",
- "pyspark.pandas.tests.test_config",
- "pyspark.pandas.tests.test_csv",
- "pyspark.pandas.tests.test_dataframe_conversion",
- "pyspark.pandas.tests.test_dataframe_spark_io",
- "pyspark.pandas.tests.test_default_index",
- "pyspark.pandas.tests.test_expanding",
- "pyspark.pandas.tests.test_extension",
- "pyspark.pandas.tests.test_frame_spark",
- "pyspark.pandas.tests.test_indexops_spark",
- "pyspark.pandas.tests.test_internal",
- "pyspark.pandas.tests.test_namespace",
- "pyspark.pandas.tests.test_numpy_compat",
- "pyspark.pandas.tests.test_ops_on_diff_frames_groupby_expanding",
- "pyspark.pandas.tests.test_ops_on_diff_frames_groupby_rolling",
- "pyspark.pandas.tests.test_repr",
- "pyspark.pandas.tests.test_reshape",
- "pyspark.pandas.tests.test_rolling",
- "pyspark.pandas.tests.test_series_conversion",
- "pyspark.pandas.tests.test_series_datetime",
- "pyspark.pandas.tests.test_series_string",
- "pyspark.pandas.tests.test_sql",
- "pyspark.pandas.tests.test_typedef",
- "pyspark.pandas.tests.test_utils",
- "pyspark.pandas.tests.test_window",
- ],
+ ] + _discover_python_unittests(
+ "pyspark/pandas/tests",
+ "pyspark/pandas/tests/data_type_ops",
+ "pyspark/pandas/tests/indexes",
+ "pyspark/pandas/tests/plot"
Review comment:
Yep, we can just put `pyspark/pandas/tests` in here, it would be discovered all sub path modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-862154622
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-861195984
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864314782
**[Test build #140004 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140004/testReport)** for PR 32867 at commit [`17dda87`](https://github.com/apache/spark/commit/17dda875e3195b4909fd4d711f62b40bebe7c9a9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858725498
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44170/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32867:
URL: https://github.com/apache/spark/pull/32867
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864343365
**[Test build #140014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140014/testReport)** for PR 32867 at commit [`74e3dd1`](https://github.com/apache/spark/commit/74e3dd124cb6a0f75d4a75639e38b3804519fa1e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32867: [SPARK-35721][PYTHON] Path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-864314782
**[Test build #140004 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140004/testReport)** for PR 32867 at commit [`17dda87`](https://github.com/apache/spark/commit/17dda875e3195b4909fd4d711f62b40bebe7c9a9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32867: [SPARK-XXXX][PYTHON] path level discover for python unittests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32867:
URL: https://github.com/apache/spark/pull/32867#issuecomment-858675605
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139642/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org