You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Yibo Cai (Jira)" <ji...@apache.org> on 2021/04/07 07:07:00 UTC
[jira] [Comment Edited] (ARROW-12241) [Python] Parallel csv reader
cancellation test kills pytest
[ https://issues.apache.org/jira/browse/ARROW-12241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316047#comment-17316047 ]
Yibo Cai edited comment on ARROW-12241 at 4/7/21, 7:06 AM:
-----------------------------------------------------------
Hmm..., looks the keyboard interrupt is generated deliberately to test the cancellation? But why it stops the pytest main process?
was (Author: yibocai):
Hmm..., looks the keyboard interrupt is generated deliberately to test the cancellation. But why it stops the pytest main process?
> [Python] Parallel csv reader cancellation test kills pytest
> -----------------------------------------------------------
>
> Key: ARROW-12241
> URL: https://issues.apache.org/jira/browse/ARROW-12241
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Yibo Cai
> Priority: Major
>
> CI job is okay, but it failed from my side. Tested on x86 skylake server with 32 cores, and Apple M1 with 8 cores.
> Maybe I missed something?
> Test steps:
> {code:bash}
> $ cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DARROW_COMPUTE=ON -DARROW_PARQUET=ON -DARROW_BUILD_TESTS=ON -DCMAKE_INSTALL_PREFIX=$(pwd)/_install -DCMAKE_INSTALL_LIBDIR=lib -DARROW_PYTHON=ON -DCMAKE_CXX_COMPILER=/usr/bin/clang++-9 -DCMAKE_C_COMPILER=/usr/bin/clang-9 ..
> $ ninja install
> $ cd ~/arrow/python
> # set LD_LIBRARY_PATH, ARROW_HOME to newly built binaries
> $ python setup.py build_ext --inplace
> $ pytest pyarrow
> {code}
> Error log:
> {code:bash}
> ======================================================================================= test session starts ========================================================================================
> platform linux -- Python 3.6.9, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
> rootdir: /home/cyb/arrow/python, configfile: setup.cfg
> plugins: lazy-fixture-0.6.3, hypothesis-5.41.2
> collected 3802 items / 3 skipped / 3799 selected
> pyarrow/tests/test_adhoc_memory_leak.py s [ 0%]
> pyarrow/tests/test_array.py ......................s...........................................................................................s............................................. [ 4%]
> ............................ss......... [ 5%]
> pyarrow/tests/test_builder.py .... [ 5%]
> pyarrow/tests/test_cffi.py ...... [ 5%]
> pyarrow/tests/test_compute.py .................................................................................................................................................... [ 9%]
> pyarrow/tests/test_convert_builtin.py ...................................................................................................................................................... [ 13%]
> ................................................................................................................................x..................................................x........ [ 18%]
> ...............ssssss.....................................................................................................sssssss [ 21%]
> pyarrow/tests/test_csv.py ....................................F.......................
> ============================================================================================= FAILURES =============================================================================================
> ______________________________________________________________________________ TestParallelCSVRead.test_cancellation _______________________________________________________________________________
> self = <pyarrow.tests.test_csv.TestParallelCSVRead testMethod=test_cancellation>
> def test_cancellation(self):
> if (threading.current_thread().ident !=
> threading.main_thread().ident):
> pytest.skip("test only works from main Python thread")
> if sys.version_info >= (3, 8):
> raise_signal = signal.raise_signal
> elif os.name == 'nt':
> # On Windows, os.kill() doesn't actually send a signal,
> # it just terminates the process with the given exit code.
> pytest.skip("test requires Python 3.8+ on Windows")
> else:
> # On Unix, emulate raise_signal() with os.kill().
> def raise_signal(signum):
> os.kill(os.getpid(), signum)
> large_csv = b"a,b,c\n" + b"1,2,3\n" * 30000000
> def signal_from_thread():
> time.sleep(0.2)
> raise_signal(signal.SIGINT)
> t1 = time.time()
> with pytest.raises(KeyboardInterrupt) as exc_info:
> threading.Thread(target=signal_from_thread).start()
> > self.read_bytes(large_csv)
> E Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
> pyarrow/tests/test_csv.py:927: Failed
> ========================================================================================= warnings summary =========================================================================================
> ../../archery/lib/python3.6/distutils/__init__.py:4
> /home/cyb/archery/lib/python3.6/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
> import imp
> -- Docs: https://docs.pytest.org/en/stable/warnings.html
> ===================================================================================== short test summary info ======================================================================================
> FAILED pyarrow/tests/test_csv.py::TestParallelCSVRead::test_cancellation - Failed: DID NOT RAISE <class 'KeyboardInterrupt'>
> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
> pyarrow/error.pxi:221: KeyboardInterrupt
> (to show a full traceback on KeyboardInterrupt use --full-trace)
> ================================================================= 1 failed, 864 passed, 21 skipped, 2 xfailed, 1 warning in 4.19s ==================================================================
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)