[jira] [Commented] (BEAM-3645) Support multi-process execution on the FnApiRunner

["Yifan Mai (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924585#comment-16924585 ] 

Yifan Mai commented on BEAM-3645:
---------------------------------

While testing this I noticed that the multi-process runner does not handle SIGINT gracefully. To reproduce, run wordcount.py using the "Run with multiprocessing mode" instructions from the comment above (in Python 3).

Expected: wordcount terminates gracefully when Ctrl-C is pressed during pipeline execution (similarly to default direct runner)
Actual: wordcount hangs forever after printing the following once per worker:

{code}
Exception in thread run_worker:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/google/home/yifanmai/venv/wordcount/lib/python3.6/site-packages/apache_beam/runners/portability/local_job_service.py", line 216, in run
    'Worker subprocess exited with return code %s' % p.returncode)
RuntimeError: Worker subprocess exited with return code 1
{code}

> Support multi-process execution on the FnApiRunner
> --------------------------------------------------
>
>                 Key: BEAM-3645
>                 URL: https://issues.apache.org/jira/browse/BEAM-3645
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py-core
>    Affects Versions: 2.2.0, 2.3.0
>            Reporter: Charles Chen
>            Assignee: Hannah Jiang
>            Priority: Major
>             Fix For: 2.15.0
>
>          Time Spent: 35h 20m
>  Remaining Estimate: 0h
>
> https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance gain over the previous DirectRunner.  We can do even better in multi-core environments by supporting multi-process execution in the FnApiRunner, to scale past Python GIL limitations.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)