You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/03 08:13:25 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

HyukjinKwon opened a new pull request #29634:
URL: https://github.com/apache/spark/pull/29634


   ### What changes were proposed in this pull request?
   
   This PR proposes to add a page to describe how to test PySpark. Note that it avoids duplication of https://spark.apache.org/developer-tools.html and it more aims to add put the relevant links together.
   
   ### Why are the changes needed?
   
   To guide PySpark developers easily test.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, it will adds a new documentation page:
   
   ![Screen Shot 2020-09-03 at 5 10 08 PM](https://user-images.githubusercontent.com/6477701/92088806-561b4c00-ee08-11ea-9069-a44bfb54e9e4.png)
   
   ### How was this patch tested?
   
   Manually tested.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686937104






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686896263


   **[Test build #128274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128274/testReport)** for PR 29634 at commit [`46934b3`](https://github.com/apache/spark/commit/46934b3470480d7c1cda711289546ae9ba419a6b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686350365






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686349814


   **[Test build #128245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128245/testReport)** for PR 29634 at commit [`ee723db`](https://github.com/apache/spark/commit/ee723db379bf571a2f51939df60bf3958517aba3).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686889107






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686925870






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-687024460


   Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686350365






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686331474






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686889107






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #29634:
URL: https://github.com/apache/spark/pull/29634


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686334763


   **[Test build #128245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128245/testReport)** for PR 29634 at commit [`ee723db`](https://github.com/apache/spark/commit/ee723db379bf571a2f51939df60bf3958517aba3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686925870






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686331023


   @srowen, @viirya, @BryanCutler WDYT?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686925219


   **[Test build #128277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128277/testReport)** for PR 29634 at commit [`9dac672`](https://github.com/apache/spark/commit/9dac672d0d0545fd7060e7ee9f14ac50428bbbce).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686335329






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] srowen commented on a change in pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #29634:
URL: https://github.com/apache/spark/pull/29634#discussion_r482980446



##########
File path: python/docs/source/development/testing.rst
##########
@@ -0,0 +1,61 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+===============
+Testing PySpark
+===============
+
+In order to run PySpark tests, you should build Spark itself first via Maven
+or SBT. For example,
+
+.. code-block:: bash
+
+    build/mvn -DskipTests clean package
+
+After that, the PySpark test cases can be run via using ``python/run-tests``. For example,
+
+.. code-block:: bash
+
+    python/run-tests --python-executable=python3
+
+Note that:
+
+* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``.
+* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading <https://spark.apache.org/docs/latest/#downloading>`_.
+
+Please see the guidance on how to `build Spark <https://github.com/apache/spark#building-spark>`_,
+`run tests for a module, or individual tests <https://spark.apache.org/developer-tools.html>`_.
+
+
+Running Individual PySpark Tests
+--------------------------------
+
+You can run a specific test via using ``python/run-tests``, for example, as below:
+
+.. code-block:: bash
+
+    python/run-tests --testnames pyspark.sql.tests.test_arrow
+
+Please refer `Testing PySpark <https://spark.apache.org/developer-tools.html>`_ for more details.

Review comment:
       refer to

##########
File path: python/docs/source/development/testing.rst
##########
@@ -0,0 +1,61 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+===============
+Testing PySpark
+===============
+
+In order to run PySpark tests, you should build Spark itself first via Maven
+or SBT. For example,
+
+.. code-block:: bash
+
+    build/mvn -DskipTests clean package
+
+After that, the PySpark test cases can be run via using ``python/run-tests``. For example,
+
+.. code-block:: bash
+
+    python/run-tests --python-executable=python3
+
+Note that:
+
+* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``.
+* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading <https://spark.apache.org/docs/latest/#downloading>`_.

Review comment:
       Just curious is this specifically necessary for debugging? shouldn't be necessary in Spark 3 in general.

##########
File path: python/docs/source/development/testing.rst
##########
@@ -0,0 +1,61 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+===============
+Testing PySpark
+===============
+
+In order to run PySpark tests, you should build Spark itself first via Maven
+or SBT. For example,
+
+.. code-block:: bash
+
+    build/mvn -DskipTests clean package
+
+After that, the PySpark test cases can be run via using ``python/run-tests``. For example,
+
+.. code-block:: bash
+
+    python/run-tests --python-executable=python3
+
+Note that:
+
+* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``.
+* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading <https://spark.apache.org/docs/latest/#downloading>`_.
+
+Please see the guidance on how to `build Spark <https://github.com/apache/spark#building-spark>`_,
+`run tests for a module, or individual tests <https://spark.apache.org/developer-tools.html>`_.
+
+
+Running Individual PySpark Tests
+--------------------------------
+
+You can run a specific test via using ``python/run-tests``, for example, as below:
+
+.. code-block:: bash
+
+    python/run-tests --testnames pyspark.sql.tests.test_arrow
+
+Please refer `Testing PySpark <https://spark.apache.org/developer-tools.html>`_ for more details.
+
+
+Running tests using GitHub Actions
+----------------------------------
+
+You can run the full PySpark tests by using GitHub Actions in your own forked GitHub

Review comment:
       with a few clicks
   refer to




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686888751


   **[Test build #128274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128274/testReport)** for PR 29634 at commit [`46934b3`](https://github.com/apache/spark/commit/46934b3470480d7c1cda711289546ae9ba419a6b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686888751


   **[Test build #128274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128274/testReport)** for PR 29634 at commit [`46934b3`](https://github.com/apache/spark/commit/46934b3470480d7c1cda711289546ae9ba419a6b).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686936689


   **[Test build #128277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128277/testReport)** for PR 29634 at commit [`9dac672`](https://github.com/apache/spark/commit/9dac672d0d0545fd7060e7ee9f14ac50428bbbce).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686334763


   **[Test build #128245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128245/testReport)** for PR 29634 at commit [`ee723db`](https://github.com/apache/spark/commit/ee723db379bf571a2f51939df60bf3958517aba3).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686896591






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686335329






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686896591






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686331474






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29634:
URL: https://github.com/apache/spark/pull/29634#discussion_r483369994



##########
File path: python/docs/source/development/testing.rst
##########
@@ -0,0 +1,61 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+===============
+Testing PySpark
+===============
+
+In order to run PySpark tests, you should build Spark itself first via Maven
+or SBT. For example,
+
+.. code-block:: bash
+
+    build/mvn -DskipTests clean package
+
+After that, the PySpark test cases can be run via using ``python/run-tests``. For example,
+
+.. code-block:: bash
+
+    python/run-tests --python-executable=python3
+
+Note that:
+
+* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``.
+* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading <https://spark.apache.org/docs/latest/#downloading>`_.

Review comment:
       Yeah, ideally we should let it work out of the box but actually it is required to set `tryReflectionSetAccessible` property to run with JDK 11 (see https://github.com/apache/spark/pull/26552). Otherwise, the Arrow related code paths fail.  See also ARROW-7223.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686925219


   **[Test build #128277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128277/testReport)** for PR 29634 at commit [`9dac672`](https://github.com/apache/spark/commit/9dac672d0d0545fd7060e7ee9f14ac50428bbbce).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29634:
URL: https://github.com/apache/spark/pull/29634#discussion_r483399207



##########
File path: python/docs/source/development/testing.rst
##########
@@ -0,0 +1,61 @@
+..  Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+..    http://www.apache.org/licenses/LICENSE-2.0
+
+..  Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+===============
+Testing PySpark
+===============
+
+In order to run PySpark tests, you should build Spark itself first via Maven
+or SBT. For example,
+
+.. code-block:: bash
+
+    build/mvn -DskipTests clean package
+
+After that, the PySpark test cases can be run via using ``python/run-tests``. For example,
+
+.. code-block:: bash
+
+    python/run-tests --python-executable=python3
+
+Note that:
+
+* If you are running tests on Mac OS, you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES``.
+* If you are using JDK 11, you should set ``-Dio.netty.tryReflectionSetAccessible=true`` for Arrow related features. See also `Downloading <https://spark.apache.org/docs/latest/#downloading>`_.

Review comment:
       Oh! but we set it in testing scripts by default so it is more required for debugging when we don't use our own testing script. Yes, I got the point now. Let me just remove this here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29634: [SPARK-32783][DOCS][PYTHON] Development - Testing PySpark

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29634:
URL: https://github.com/apache/spark/pull/29634#issuecomment-686937104






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org