You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Patrick Wendell (JIRA)" <ji...@apache.org> on 2014/04/23 03:51:15 UTC
[jira] [Updated] (SPARK-1579) PySpark should distinguish expected
IOExceptions from unexpected ones in the worker
[ https://issues.apache.org/jira/browse/SPARK-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell updated SPARK-1579:
-----------------------------------
Description:
I chatted with [~adav] a bit about this. Right now we drop IOExceptions because they are (in some cases) expected if a Python worker returns before consuming its entire input. The issue is this swallows legitimate IO exceptions when they occur.
One thought we had was to change the daemon.py file to, instead of closing the socket when the function is over, simply busy-wait on the socket being closed. We'd transfer the responsibility for closing the socket to the Java reader. The Java reader could, when it has finished consuming output form Python, set a flag on a volatile variable to indicate that Python has fully returned, and then close the socket. Then if an IOException is found, we only swallow it if we are expecting it.
This would also let us remove the warning message right now.
was:
I chatted with [~adav] a bit about this. Right now we drop IOExceptions because they are (in some cases) expected if a Python worker returns before consuming its entire input. The issue is this swallows legitimate IO exceptions when they occur.
One thought we had was to change the daemon.py file to, instead of closing the socket when the function is over, simply busy-wait on the socket being closed. We'd transfer the responsibility for closing the socket to the Java reader. The Java reader could, when it has finished consuming output form Python, set a flag on a volatile variable to indicate that Python has fully returned. Then if an IOException is found, we only swallow it if we are expecting it.
This would also let us remove the warning message right now.
> PySpark should distinguish expected IOExceptions from unexpected ones in the worker
> -----------------------------------------------------------------------------------
>
> Key: SPARK-1579
> URL: https://issues.apache.org/jira/browse/SPARK-1579
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Reporter: Patrick Wendell
> Assignee: Aaron Davidson
> Fix For: 1.1.0
>
>
> I chatted with [~adav] a bit about this. Right now we drop IOExceptions because they are (in some cases) expected if a Python worker returns before consuming its entire input. The issue is this swallows legitimate IO exceptions when they occur.
> One thought we had was to change the daemon.py file to, instead of closing the socket when the function is over, simply busy-wait on the socket being closed. We'd transfer the responsibility for closing the socket to the Java reader. The Java reader could, when it has finished consuming output form Python, set a flag on a volatile variable to indicate that Python has fully returned, and then close the socket. Then if an IOException is found, we only swallow it if we are expecting it.
> This would also let us remove the warning message right now.
--
This message was sent by Atlassian JIRA
(v6.2#6252)