You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Antoine Pitrou (Jira)" <ji...@apache.org> on 2021/04/07 07:22:00 UTC
[jira] [Commented] (ARROW-12241) [Python] Parallel csv reader
cancellation test kills pytest
[ https://issues.apache.org/jira/browse/ARROW-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316063#comment-17316063 ]
Antoine Pitrou commented on ARROW-12241:
----------------------------------------
> , looks the keyboard interrupt is generated deliberately to test the cancellation? But why it stops the pytest main process?
Hmm... probably because CSV parsing is too fast on your machine (especially in debug mode). We can try to tweak the timings and/or make the workload harder.
> [Python] Parallel csv reader cancellation test kills pytest
> -----------------------------------------------------------
>
> Key: ARROW-12241
> URL: https://issues.apache.org/jira/browse/ARROW-12241
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Yibo Cai
> Priority: Major
>
> CI job is okay, but it failed from my side. Tested on x86 skylake server with 32 cores, and Apple M1 with 8 cores.
> Maybe I missed something?
> Test steps:
> {code:bash}
> $ cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DARROW_COMPUTE=ON -DARROW_PARQUET=ON -DARROW_BUILD_TESTS=ON -DCMAKE_INSTALL_PREFIX=$(pwd)/_install -DCMAKE_INSTALL_LIBDIR=lib -DARROW_PYTHON=ON -DCMAKE_CXX_COMPILER=/usr/bin/clang++-9 -DCMAKE_C_COMPILER=/usr/bin/clang-9 ..
> $ ninja install
> $ cd ~/arrow/python
> # set LD_LIBRARY_PATH, ARROW_HOME to newly built binaries
> $ python setup.py build_ext --inplace
> $ pytest pyarrow
> {code}
> Error log:
> {code:bash}
> ======================================================================================= test session starts ========================================================================================
> platform linux -- Python 3.6.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
> rootdir: /home/cyb/arrow/python, configfile: setup.cfg
> plugins: lazy-fixture-0.6.3, hypothesis-5.41.2
> collected 3802 items / 3 skipped / 3799 selected
> pyarrow/tests/test_adhoc_memory_leak.py s [ 0%]
> pyarrow/tests/test_array.py ......................s...........................................................................................s............................................. [ 4%]
> ............................ss......... [ 5%]
> pyarrow/tests/test_builder.py .... [ 5%]
> pyarrow/tests/test_cffi.py ...... [ 5%]
> pyarrow/tests/test_compute.py .................................................................................................................................................... [ 9%]
> pyarrow/tests/test_convert_builtin.py ...................................................................................................................................................... [ 13%]
> ................................................................................................................................x..................................................x........ [ 18%]
> ...............ssssss.....................................................................................................sssssss [ 21%]
> pyarrow/tests/test_csv.py ....................................F.......................
> ============================================================================================= FAILURES =============================================================================================
> ______________________________________________________________________________ TestParallelCSVRead.test_cancellation _______________________________________________________________________________
> self = <pyarrow.tests.test_csv.TestParallelCSVRead testMethod=test_cancellation>
> def test_cancellation(self):
> if (threading.current_thread().ident !=
> threading.main_thread().ident):
> pytest.skip("test only works from main Python thread")
> if sys.version_info >= (3, 8):
> raise_signal = signal.raise_signal
> elif os.name == 'nt':
> # On Windows, os.kill() doesn't actually send a signal,
> # it just terminates the process with the given exit code.
> pytest.skip("test requires Python 3.8+ on Windows")
> else:
> # On Unix, emulate raise_signal() with os.kill().
> def raise_signal(signum):
> os.kill(os.getpid(), signum)
> large_csv = b"a,b,c\n" + b"1,2,3\n" * 30000000
> def signal_from_thread():
> time.sleep(0.2)
> raise_signal(signal.SIGINT)
> t1 = time.time()
> with pytest.raises(KeyboardInterrupt) as exc_info:
> threading.Thread(target=signal_from_thread).start()
> > self.read_bytes(large_csv)
> E Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
> pyarrow/tests/test_csv.py:927: Failed
> ========================================================================================= warnings summary =========================================================================================
> ../../archery/lib/python3.6/distutils/__init__.py:4
> /home/cyb/archery/lib/python3.6/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
> import imp
> -- Docs: https://docs.pytest.org/en/stable/warnings.html
> ===================================================================================== short test summary info ======================================================================================
> FAILED pyarrow/tests/test_csv.py::TestParallelCSVRead::test_cancellation - Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> pyarrow/error.pxi:221: KeyboardInterrupt
> (to show a full traceback on KeyboardInterrupt use --full-trace)
> ================================================================= 1 failed, 864 passed, 21 skipped, 2 xfailed, 1 warning in 4.19s ==================================================================
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)