[jira] [Commented] (SPARK-8170) Ctrl-C in pyspark shell doesn't kill running job

["Jacob Wellington (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14985949#comment-14985949 ] 

Jacob Wellington commented on SPARK-8170:
-----------------------------------------

I'm running into an issue when I'm trying to connect to spark via a django server that connects to spark by creating a new thread. When I comment out line 225 of the context.py file in the PR it works fine. Also works fine in 1.5.1.

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/asoni/.conda/envs/py35_spark_django/lib/python3.5/threading.py", line 923, in _bootstrap_inner
    self.run()
  File "/home/asoni/.conda/envs/py35_spark_django/lib/python3.5/threading.py", line 871, in run
    self._target(*self._args, **self._kwargs)
  File "/home/asoni/spark-server/spark_server/spark_jobs/views.py", line 67, in process_job
    from spark_jobs.job_runner import JobRunner
  File "/home/asoni/spark-server/spark_server/spark_jobs/job_runner.py", line 2, in <module>
    from spark_jobs.extraction_plan_group_runner import *
  File "/home/asoni/spark-server/spark_server/spark_jobs/extraction_plan_group_runner.py", line 1, in <module>
    from spark_jobs.spark_connection import SparkConnection
  File "/home/asoni/spark-server/spark_server/spark_jobs/spark_connection.py", line 5, in <module>
    class SparkConnection:
  File "/home/asoni/spark-server/spark_server/spark_jobs/spark_connection.py", line 7, in SparkConnection
    sc = SparkContext(conf=conf)
  File "/home/asoni/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/pyspark.zip/pyspark/context.py", line 114, in __init__
    conf, jsc, profiler_cls)
  File "/home/asoni/spark-1.6.0-SNAPSHOT-bin-custom-spark/python/lib/pyspark.zip/pyspark/context.py", line 225, in _do_init
    signal.signal(signal.SIGINT, signal_handler)
  File "/home/asoni/.conda/envs/py35_spark_django/lib/python3.5/signal.py", line 47, in signal
    handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread

> Ctrl-C in pyspark shell doesn't kill running job
> ------------------------------------------------
>
>                 Key: SPARK-8170
>                 URL: https://issues.apache.org/jira/browse/SPARK-8170
>             Project: Spark
>          Issue Type: Sub-task
>          Components: PySpark
>    Affects Versions: 1.3.1
>            Reporter: Ashwin Shankar
>            Assignee: Ashwin Shankar
>             Fix For: 1.6.0
>
>
> Hitting Ctrl-C in spark-sql(and other tools like presto) cancels any running job and starts a new input line on the prompt. It would be nice if pyspark shell also can do that. Otherwise, in case a user submits a job, say he made a mistake, and wants to cancel it, he needs to exit the shell and re-login to continue his work. Re-login can be a pain especially in Spark on yarn, since it takes a while to allocate AM container and initial executors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org