You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by buckhx <gi...@git.apache.org> on 2016/04/14 16:53:49 UTC
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
GitHub user buckhx opened a pull request:
https://github.com/apache/spark/pull/12398
[SPARK-5929][PYSPARK] Context addPyPackage and addRequirementsFile
## What changes were proposed in this pull request?
Context.addPyPackage()
Context.addRequirementsFile()
Both of these methods take a package on the master and ship it to the workers when called instead of having to manually install packages on all workers.
## How was this patch tested?
Unit tests are written, but I do not believe they accurately reflect a distributed environment. The test_add_py_package is not using addPyPackage and still works. The addRequirementsFile method requires internet access to hit the global pypi server and won't work on the current Jenkins build system.
We have had this patch running at Palantir for about a year in production.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/buckhx/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12398.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12398
----
commit 0ed060df2ec5a1a0427df6c160bd51c7014b29da
Author: buck heroux <bh...@palantir.com>
Date: 2015-02-18T17:50:27Z
added requirements file to pyspark
commit 6b8bcde60378b58998f5c14d81d72de81f44d718
Author: buck heroux <bh...@palantir.com>
Date: 2015-02-18T23:30:28Z
tarfile has no contextmanager in python2.
commit 2773483ea6cc244cb7de02c7dc184391a94d29e6
Author: buck heroux <bh...@palantir.com>
Date: 2015-02-19T01:52:06Z
reqs fix
commit 0371ad9b13f96dcc534d897789ccd32f907d5ed9
Author: buck heroux <bh...@palantir.com>
Date: 2015-02-19T02:06:48Z
temp tar file
commit f2a46e5d6e309a5ba29259cc1f77e594d932b0f5
Author: buck heroux <bh...@palantir.com>
Date: 2015-03-05T00:51:46Z
bubbled up try finally
commit fca4be61c6542b807a0d5370f761ef031fc7eb86
Author: buck heroux <bh...@palantir.com>
Date: 2015-03-05T00:53:17Z
forgot to remove
commit d28752297ec5b10f6afce49713f4efe20f8d533f
Author: buck heroux <bh...@palantir.com>
Date: 2015-04-03T20:35:44Z
added requirementsFile tests and switch to __import__
commit 76ff63733b9d293c43218aa743a74bcce36a20c9
Author: buck heroux <bh...@palantir.com>
Date: 2015-04-03T20:41:30Z
merged tests
commit 565bf7fc0889390178f8f96d0ed9a917b287e164
Author: buck heroux <bh...@palantir.com>
Date: 2015-07-14T17:54:12Z
pep8 styling
commit 23771fde4674c73d5fb0bbb478385d9cf55c648b
Author: buck heroux <bh...@palantir.com>
Date: 2015-10-05T22:07:48Z
support namespace packages and extract addModule logic
commit cd21c5c06d6a611f4e738199f20582aeb6087fd7
Author: buck heroux <bh...@palantir.com>
Date: 2015-10-05T22:10:17Z
tmp_dir to mod
commit 39f26d93b7030f2d7bfe7782414dc5b016cdd852
Author: buck heroux <bh...@palantir.com>
Date: 2015-10-21T20:22:06Z
remove reqs from context constructor
commit 49a4ed0931b20dd2982c16e1ca3533cef65303af
Author: buck heroux <bh...@palantir.com>
Date: 2015-10-21T20:39:24Z
upstream merge
commit 1501d0f4744d4645a239f3c3fb931469c2038fa8
Author: buckhx <bu...@gmail.com>
Date: 2016-03-31T17:30:58Z
upstream merge
commit 3af35bb5f54518f2d07db7665c78d18c5fe2bdf2
Author: buck heroux <bh...@palantir.com>
Date: 2016-04-13T19:08:06Z
Merge remote-tracking branch 'upstream/master'
commit 88a1d6ceffaddfc5f3aa5afc07d65a1d9ddb54e9
Author: buck heroux <bh...@palantir.com>
Date: 2016-04-13T22:57:23Z
add_py_package test
commit 93b9e9fdd464744f90eb3a7eb325f853e2f113d4
Author: buck heroux <bh...@palantir.com>
Date: 2016-04-14T14:39:17Z
uncommented pip_requirements test
commit 82476a632c5da724c3ca45ed8b888a946d934104
Author: buck heroux <bh...@palantir.com>
Date: 2016-04-14T14:42:05Z
removed todo
commit ce9966e875a6d00bbe6325dc5fc9519ce5292c04
Author: buck heroux <bh...@palantir.com>
Date: 2016-04-14T14:47:36Z
spacing
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/12398#discussion_r59800417
--- Diff: python/pyspark/context.py ---
@@ -814,6 +817,40 @@ def addPyFile(self, path):
import importlib
importlib.invalidate_caches()
+ def addPyPackage(self, pkg):
+ """
+ Add a package to the spark context, the package must have already been
+ imported by the driver via __import__ semantics. Supports namespace
+ packages by simulating the loading __path__ as a set of modules from
+ the __path__ list in a single package.
+ """
+ tmp_dir = tempfile.mkdtemp()
+ try:
+ tar_path = os.path.join(tmp_dir, pkg.__name__+'.tar.gz')
+ tar = tarfile.open(tar_path, "w:gz")
+ for mod in pkg.__path__[::-1]:
+ # adds in reverse to simulate namespace loading path
+ tar.add(mod, arcname=os.path.basename(mod))
+ tar.close()
+ self.addPyFile(tar_path)
+ finally:
+ shutil.rmtree(tmp_dir)
+
+ def addRequirementsFile(self, path):
+ """
+ Add a pip requirements file to distribute dependencies for all tasks
+ on thie SparkContext in the future. An ImportError will be thrown if
+ a module in the file can't be downloaded.
+ See https://pip.pypa.io/en/latest/user_guide.html#requirements-files
+ Raises ImportError if the requirement can't be found
+ """
+ import pip
+ for req in pip.req.parse_requirements(path, session=uuid.uuid1()):
+ if not req.check_if_exists():
+ pip.main(['install', req.req.__str__()])
--- End diff --
So it seems that this can sometimesrequire elevated privileges based on the issues with the previous jenkins run. What about if at startup we created a fixed temp directory per context adding it to our path with `sys.path.insert(0, self.pipBase)` and at install did something along the lines of:
`pip.main(['install', req.req.__str__(), '--target', self.pipBase])` so that we don't have to have write permissions to the default pip target?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by buckhx <gi...@git.apache.org>.
Github user buckhx commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-217902925
Added an addPyPackage example to the docstring and that test formatting. We're looking into adding the spark-submit arg for --py-requirements.
The pip API takes a file path via pip.req.parse_requirements https://github.com/pypa/pip/blob/develop/pip/req/req_file.py#L64 but it looks like it might be possible to take a string and break it out line by with a mock file name via pip.req.process_line & pip.req.preprocess https://github.com/pypa/pip/blob/develop/pip/req/req_file.py#L110
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12398: [SPARK-5929][PYSPARK] Context addPyPackage and addPyRequ...
Posted by buckhx <gi...@git.apache.org>.
Github user buckhx commented on the issue:
https://github.com/apache/spark/pull/12398
@davies how's this looking?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12398: [SPARK-5929][PYSPARK] Context addPyPackage and addPyRequ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12398
**[Test build #3068 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3068/consoleFull)** for PR 12398 at commit [`9c37e06`](https://github.com/apache/spark/commit/9c37e0649a75d22d5712b08dcd6c3acee412f719).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12398: [SPARK-5929][PYSPARK] Context addPyPackage and addPyRequ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/12398
**[Test build #3068 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3068/consoleFull)** for PR 12398 at commit [`9c37e06`](https://github.com/apache/spark/commit/9c37e0649a75d22d5712b08dcd6c3acee412f719).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12398: [SPARK-5929][PYSPARK] Context addPyPackage and addPyRequ...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/12398
@buckhx any interest in updating this against master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12398#discussion_r61992299
--- Diff: python/pyspark/context.py ---
@@ -814,6 +817,40 @@ def addPyFile(self, path):
import importlib
importlib.invalidate_caches()
+ def addPyPackage(self, pkg):
+ """
+ Add a package to the spark context, the package must have already been
+ imported by the driver via __import__ semantics. Supports namespace
+ packages by simulating the loading __path__ as a set of modules from
+ the __path__ list in a single package.
+ """
+ tmp_dir = tempfile.mkdtemp()
+ try:
+ tar_path = os.path.join(tmp_dir, pkg.__name__+'.tar.gz')
+ tar = tarfile.open(tar_path, "w:gz")
+ for mod in pkg.__path__[::-1]:
+ # adds in reverse to simulate namespace loading path
+ tar.add(mod, arcname=os.path.basename(mod))
+ tar.close()
+ self.addPyFile(tar_path)
+ finally:
+ shutil.rmtree(tmp_dir)
+
+ def addRequirementsFile(self, path):
--- End diff --
Will it be better to pass the requirements as string?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12398#discussion_r61992444
--- Diff: python/pyspark/context.py ---
@@ -814,6 +817,40 @@ def addPyFile(self, path):
import importlib
importlib.invalidate_caches()
+ def addPyPackage(self, pkg):
+ """
+ Add a package to the spark context, the package must have already been
+ imported by the driver via __import__ semantics. Supports namespace
+ packages by simulating the loading __path__ as a set of modules from
+ the __path__ list in a single package.
+ """
--- End diff --
Could you add an example here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-210179401
Since @andrewor14 got Jenkins to run the earlier iteration of this PR maybe you could do the same here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by buckhx <gi...@git.apache.org>.
Github user buckhx commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-211420495
I could see the pipBase approach working.
The other testing blocker I have is that it seems like that context that's getting created in the test is able to use local packages. The test package that's created is able to be imported on the workers without distributing it via addPyPackage. https://github.com/apache/spark/pull/12398/files#diff-388a156f4ce454fe98d7a99a0f7f0012R1950
Is there a way to get the mocked workers to not use the master python path?
The pip part is nice, but I think the meat of this PR is in the addPyPackage.
I looked into adding a flag to the CLI, but kind of went down a rabbit hole trying to figure it out. I'd advocate adding it to the CLI, but think someone working in the spark core space could get that integrated much more efficiently than myself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-210171984
So looking back on the predecessor PR it seemed like @davies suggested adding support for specifying these at runtime with `spark-submit` & it seems like the pip install permissions issue might still be present for testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12398: [SPARK-5929][PYSPARK] Context addPyPackage and ad...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12398
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/12398#discussion_r61992184
--- Diff: python/pyspark/tests.py ---
@@ -1947,6 +1947,33 @@ def test_with_stop(self):
sc.stop()
self.assertEqual(SparkContext._active_spark_context, None)
+ def test_add_py_package(self):
+ name = "test_tmp"
+ try:
+ os.mkdir(name)
+ with open(os.path.join(name, "__init__.py"), 'w+') as temp:
+ temp.write("triple = lambda x: 3*x")
+ pkg = __import__(name)
+ with SparkContext() as sc:
+ #trips = sc.parallelize([0, 1, 2, 3]).map(test_tmp.triple)
+ #sc.addPyPackage(pkg)
+ trips = sc.parallelize([0, 1, 2, 3]).map(lambda x: pkg.triple(x))
+ self.assertSequenceEqual([0, 3, 6, 9], trips.collect())
+ finally:
+ shutil.rmtree(name)
+
--- End diff --
remove the extra empty line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-210172335
I think having this supported could be very useful :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-216749580
**[Test build #2967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2967/consoleFull)** for PR 12398 at commit [`ce9966e`](https://github.com/apache/spark/commit/ce9966e875a6d00bbe6325dc5fc9519ce5292c04).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-216751245
@buckhx These API seems useful, could you also add an argument for bin/spark-submit (only for requirement file) ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12398: [SPARK-5929][PYSPARK] Context addPyPackage and addPyRequ...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/12398
@buckhx Just following up to see if this is something you are still interested in working on?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12398: [SPARK-5929][PYSPARK] Context addPyPackage and addPyRequ...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/12398
ping @buckhx
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by buckhx <gi...@git.apache.org>.
Github user buckhx commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-218478989
@robert3005 added the --py-requirements to spark-submit. I'm looking at passing the requirements as a list or a line delimited string currently.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-209986241
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-216749512
**[Test build #2967 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2967/consoleFull)** for PR 12398 at commit [`ce9966e`](https://github.com/apache/spark/commit/ce9966e875a6d00bbe6325dc5fc9519ce5292c04).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-216711127
ok to test
@dvaies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by buckhx <gi...@git.apache.org>.
Github user buckhx commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-219136325
changed addRequirementsFile to addPyRequirements which takes a list of requirement strings
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5929][PYSPARK] Context addPyPackage and...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/12398#issuecomment-217919681
@buckhx Even the pip API only takes a file path, we could write the as temporary file, I think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org