You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/01/20 06:19:40 UTC
[spark-website] branch asf-site updated: Add some info about
OBJC_DISABLE_INITIALIZE_FORK_SAFETY and macOS High Sierra+ for PySpark
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/spark-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f523903 Add some info about OBJC_DISABLE_INITIALIZE_FORK_SAFETY and macOS High Sierra+ for PySpark
f523903 is described below
commit f5239036e7badb082aa05c7aa96bdd9551a0ef83
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Mon Jan 20 15:19:29 2020 +0900
Add some info about OBJC_DISABLE_INITIALIZE_FORK_SAFETY and macOS High Sierra+ for PySpark
To run some PySpark features like pandas UDF, it requires to set `OBJC_DISABLE_INITIALIZE_FORK_SAFETY` on macOS High Sierra+. See https://issues.apache.org/jira/browse/SPARK-25473 and https://bugs.python.org/issue33725
I had to explain this multiple times. I wasn't particularly worried about PySpark itself because the production env is not usually Mac.
I target to describe `OBJC_DISABLE_INITIALIZE_FORK_SAFETY` in testing page as Mac is usually dev env.
Author: HyukjinKwon <gu...@apache.org>
Closes #252 from HyukjinKwon/pyspark-doc.
---
developer-tools.md | 4 ++++
site/developer-tools.html | 4 ++++
2 files changed, 8 insertions(+)
diff --git a/developer-tools.md b/developer-tools.md
index 8ed4141..30206a7 100644
--- a/developer-tools.md
+++ b/developer-tools.md
@@ -134,6 +134,10 @@ To run individual PySpark tests, you can use `run-tests` script under `python` d
Note that, if you add some changes into Scala or Python side in Apache Spark, you need to manually build Apache Spark again before running PySpark tests in order to apply the changes.
Running PySpark testing script does not automatically build it.
+Also, note that there is an ongoing issue to use PySpark on macOS High Serria+. `OBJC_DISABLE_INITIALIZE_FORK_SAFETY`
+should be set to `YES` in order to run some of tests.
+See [PySpark issue](https://issues.apache.org/jira/browse/SPARK-25473) and [Python issue](https://bugs.python.org/issue33725) for more details.
+
To run test cases in a specific module:
```
diff --git a/site/developer-tools.html b/site/developer-tools.html
index c449e20..72c48f5 100644
--- a/site/developer-tools.html
+++ b/site/developer-tools.html
@@ -317,6 +317,10 @@ $ build/mvn package -DskipTests -pl core
Note that, if you add some changes into Scala or Python side in Apache Spark, you need to manually build Apache Spark again before running PySpark tests in order to apply the changes.
Running PySpark testing script does not automatically build it.</p>
+<p>Also, note that there is an ongoing issue to use PySpark on macOS High Serria+. <code class="highlighter-rouge">OBJC_DISABLE_INITIALIZE_FORK_SAFETY</code>
+should be set to <code class="highlighter-rouge">YES</code> in order to run some of tests.
+See <a href="https://issues.apache.org/jira/browse/SPARK-25473">PySpark issue</a> and <a href="https://bugs.python.org/issue33725">Python issue</a> for more details.</p>
+
<p>To run test cases in a specific module:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ python/run-tests --testnames pyspark.sql.tests.test_arrow
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org