You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2015/02/17 01:57:11 UTC
[jira] [Commented] (SPARK-3842) Remove the hacks for Python
callback server in py4j
[ https://issues.apache.org/jira/browse/SPARK-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323512#comment-14323512 ]
Josh Rosen commented on SPARK-3842:
-----------------------------------
It looks like this issue is still relevant as of Feb. 2015 and still blocked on upstream Py4J changes. Low-priority, though, but just thought I'd document that it's still relevant.
> Remove the hacks for Python callback server in py4j
> ----------------------------------------------------
>
> Key: SPARK-3842
> URL: https://issues.apache.org/jira/browse/SPARK-3842
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, Streaming
> Reporter: Davies Liu
> Assignee: Davies Liu
> Priority: Minor
>
> There are three hacks while create Python API for Streaming (https://github.com/apache/spark/pull/2538) :
> 1. daemonize the callback server thread, by 'thread.daemon = True' before start it https://github.com/bartdag/py4j/issues/147
> 2. let callback server bind to random port, then update the Java callback client with real port. https://github.com/bartdag/py4j/issues/148
> 3. start the callback server later. https://github.com/bartdag/py4j/issues/149
> These hacks should be removed after py4j has fix these issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org