You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by HyukjinKwon <gi...@git.apache.org> on 2018/11/13 10:04:27 UTC
[GitHub] spark pull request #23021: [SPARK-26032][PYTHON] Break large sql/tests.py fi...
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/23021
[SPARK-26032][PYTHON] Break large sql/tests.py files into smaller files
## What changes were proposed in this pull request?
This is the official first attempt to break huge single `tests.py` file - I did it locally before few times and gave up for some reasons. Now, currently it really makes the unittests super hard to read and difficult to check. To me, it even bothers me to to scroll down the big file. It's one single 7000 lines file!
This is not only readability issue. Since one big test takes most of tests time, the tests don't run in parallel fully, causing the test time increases.
We could pick up one example and follow. Given my investigation, the current style looks closer to NumPy structure and looks easier to follow. Please see https://github.com/numpy/numpy/tree/master/numpy.
Basically this PR proposes to break down `pyspark/sql/tests.py` into ...:
```bash
pyspark
...
├── sql
...
│ ├── tests # Includes all tests broken down from 'pyspark/sql/tests.py'
│ │ │ # Each matchs to module in 'pyspark/sql'. Additionally, some logical group can be added.
│ │ │ # For instance, 'test_arrow.py', 'test_datasources.py', 'test_serde.py' and 'test_submit.py'
│ │ │ # were added.
│ │ ├── __init__.py
│ │ ├── test_arrow.py
│ │ ├── test_catalog.py
│ │ ├── test_column.py
│ │ ├── test_conf.py
│ │ ├── test_context.py
│ │ ├── test_dataframe.py
│ │ ├── test_datasources.py
│ │ ├── test_functions.py
│ │ ├── test_group.py
│ │ ├── test_readwriter.py
│ │ ├── test_serde.py
│ │ ├── test_session.py
│ │ ├── test_streaming.py
│ │ ├── test_submit.py
│ │ ├── test_types.py
│ │ ├── test_udf.py
│ │ └── test_utils.py
...
├── testing # Includes testing utils that can be used in unittests.
│ ├── __init__.py
│ └── sqlutils.py
...
```
## How was this patch tested?
Existing tests should cover.
`cd python` and `./run-tests-with-coverage`. Manually checked they are actually being ran.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-25344
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23021.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23021
----
commit dee09b73757bbad73cbe274d6a9873d8cc143423
Author: hyukjinkwon <gu...@...>
Date: 2018-11-13T08:41:57Z
Break larga sql/tests.py files into smaller files
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4991/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98783/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23021: [SPARK-26032][PYTHON] Break large sql/tests.py fi...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23021#discussion_r233269827
--- Diff: python/pyspark/testing/sqlutils.py ---
@@ -0,0 +1,268 @@
+#
--- End diff --
Yea, similar thought. One thing is though testing util itself has some dependencies to each others (for instance, `from pyspark.tests import ReusedPySparkTestCase`). So, I thought it's okay to make it separate.
Another benefit from doing so is we can simply refer the example (NumPy) as is. As you already know, NumPy has been there more then 15 years and should be mature enough. It should be okay to trust their approach.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by squito <gi...@git.apache.org>.
Github user squito commented on the issue:
https://github.com/apache/spark/pull/23021
thanks for working on this @HyukjinKwon . Did you test on python3 as well?
I'm surprised elapsed time is the same -- I would have expected it to be faster as the tests could run in parallel.
aside: does anybody have a good recommendation for tools to help with viewing a diff like this? I feel like I want something really simple: a tool to show me where each original test got moved to, and highlight what was deleted or added. I tried p4merge and beyond compare, but neither was helpful (or I'm using them wrong).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23021: [SPARK-26032][PYTHON] Break large sql/tests.py fi...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/23021#discussion_r233183572
--- Diff: python/pyspark/testing/sqlutils.py ---
@@ -0,0 +1,268 @@
+#
--- End diff --
Maybe rename this file to `sql_testing_utils.py`? And is it possible to just place this in the `sql/tests` directory? Having the `testing` folder seems a little ambiguous to me
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
Elapsed time looks virtually same. All tests looks running fine. The last commit should show skipped tests fine as well. Should be ready for a look.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
adding @holdenk, @ueshin and @icexelloss as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98783/testReport)** for PR 23021 at commit [`00a9103`](https://github.com/apache/spark/commit/00a9103d768f61dfd09792e0ca142409b845f4eb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
> Did you test on python3 as well?
Of course!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4993/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
adding @rxin (derived from mailing list)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
> Could you add some descriptions to run a single test file or a single test case if exists?
Done!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98769/testReport)** for PR 23021 at commit [`8680579`](https://github.com/apache/spark/commit/8680579a171a07243a1b9eb84f527698e932ccbe).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4978/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98783/testReport)** for PR 23021 at commit [`00a9103`](https://github.com/apache/spark/commit/00a9103d768f61dfd09792e0ca142409b845f4eb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4979/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98782 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98782/testReport)** for PR 23021 at commit [`b8517a1`](https://github.com/apache/spark/commit/b8517a15b602850710a701ec3eb53368048d28df).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98765/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23021
One thing - I would put “pandas” right after test_ so you get the natural
logical grouping with sorting by file name.
On Tue, Nov 13, 2018 at 4:58 PM Hyukjin Kwon <no...@github.com>
wrote:
> I am going to push after testing and double checking. The line counts
> would look like this
>
> 54 ./test_utils.py
> 199 ./test_catalog.py
> 503 ./test_grouped_agg_pandas_udf.py
> 45 ./test_group.py
> 320 ./test_session.py
> 153 ./test_readwriter.py
> 806 ./test_scalar_pandas_udf.py
> 216 ./test_pandas_udf.py
> 566 ./test_streaming.py
> 55 ./test_conf.py
> 16 ./__init__.py
> 530 ./test_grouped_map_pandas_udf.py
> 157 ./test_column.py
> 654 ./test_udf.py
> 262 ./test_window_pandas_udf.py
> 278 ./test_functions.py
> 263 ./test_context.py
> 138 ./test_serde.py
> 170 ./test_datasources.py
> 399 ./test_arrow.py
> 96 ./test_appsubmit.py
> 944 ./test_types.py
> 737 ./test_dataframe.py
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/23021#issuecomment-438497006>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AATvPOg1IR6S5Fc4qv2mrPWTsDRRxf1Qks5uu2qagaJpZM4YbTYj>
> .
>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
@BryanCutler and @squito, Here is the official first attempt to break `pyspark/sql/tests.py` into multiple small files.
If there are no outstanding issues (for instance, if we are agreed upon the high level approach), I would like to push this quicker as this will super likely have conflicts easily.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
adding @icexelloss as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
I am going to push after testing and double checking. The line counts would look like this
```
54 ./test_utils.py
199 ./test_catalog.py
503 ./test_grouped_agg_pandas_udf.py
45 ./test_group.py
320 ./test_session.py
153 ./test_readwriter.py
806 ./test_scalar_pandas_udf.py
216 ./test_pandas_udf.py
566 ./test_streaming.py
55 ./test_conf.py
16 ./__init__.py
530 ./test_grouped_map_pandas_udf.py
157 ./test_column.py
654 ./test_udf.py
262 ./test_window_pandas_udf.py
278 ./test_functions.py
263 ./test_context.py
138 ./test_serde.py
170 ./test_datasources.py
399 ./test_arrow.py
96 ./test_appsubmit.py
944 ./test_types.py
737 ./test_dataframe.py
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
> I'd break the pandas udf one into smaller pieces too, as you suggested. We should also investigate why the runtime didn't improve ...
One suspection from my investigation is, it requires to stop and start the context for each test which is costly. I also expected it's going to slightly decrease the time but actually it looks that slightly increased the time (I guess 2 ~ 3 mins in total? - shouldn't be a big deal).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4977/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
Merged to master.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #23021: [SPARK-26032][PYTHON] Break large sql/tests.py fi...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/23021
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98808/testReport)** for PR 23021 at commit [`a990770`](https://github.com/apache/spark/commit/a990770ca17e49ac6a04fa5f3e62455af5ef7170).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/4992/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5010/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/23021
Great initiative!
I'd break the pandas udf one into smaller pieces too, as you suggested. We should also investigate why the runtime didn't improve ...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98767 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98767/testReport)** for PR 23021 at commit [`c6481d5`](https://github.com/apache/spark/commit/c6481d5033bd219846cf00b488b8d587fce54183).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98782 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98782/testReport)** for PR 23021 at commit [`b8517a1`](https://github.com/apache/spark/commit/b8517a15b602850710a701ec3eb53368048d28df).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
FWIW, I at least double checked if they are any tests missing, and if they are actually being ran (via coverage).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98765/testReport)** for PR 23021 at commit [`dee09b7`](https://github.com/apache/spark/commit/dee09b73757bbad73cbe274d6a9873d8cc143423).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98767 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98767/testReport)** for PR 23021 at commit [`c6481d5`](https://github.com/apache/spark/commit/c6481d5033bd219846cf00b488b8d587fce54183).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
I am merging this in - maybe I am rushing it but please allow me to go ahead since it's going to block other PySpark PRs.
At worst case, I am willing to revert and propose this again if there's something obviously wrong. Please feel free to leave some comments even after this is merged. I would appreciate it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98765/testReport)** for PR 23021 at commit [`dee09b7`](https://github.com/apache/spark/commit/dee09b73757bbad73cbe274d6a9873d8cc143423).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
Yup!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98808/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
Yup, will break pandas one into smaller ones as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/23021
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98767/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/23021
For your information, here's the line counts for each file:
```
52 ./test_utils.py
197 ./test_catalog.py
43 ./test_group.py
318 ./test_session.py
151 ./test_readwriter.py
2180 ./test_pandas_udf.py
564 ./test_streaming.py
53 ./test_conf.py
155 ./test_column.py
652 ./test_udf.py
276 ./test_functions.py
261 ./test_context.py
136 ./test_serde.py
168 ./test_datasources.py
397 ./test_arrow.py
94 ./test_appsubmit.py
942 ./test_types.py
735 ./test_dataframe.py
```
It's rather evenly distributed, except for `./test_pandas_udf.py`. Maybe we should make `test_scalar_pandas_udf.py` and etc. but I would like to avoid this for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98808/testReport)** for PR 23021 at commit [`a990770`](https://github.com/apache/spark/commit/a990770ca17e49ac6a04fa5f3e62455af5ef7170).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #23021: [SPARK-26032][PYTHON] Break large sql/tests.py files int...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/23021
**[Test build #98769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/98769/testReport)** for PR 23021 at commit [`8680579`](https://github.com/apache/spark/commit/8680579a171a07243a1b9eb84f527698e932ccbe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org