You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Adam Holmberg (Jira)" <ji...@apache.org> on 2020/08/17 20:26:00 UTC
[jira] [Commented] (CASSANDRA-16053) cqlsh COPY functions broken in
Python 3.8 on Mac
[ https://issues.apache.org/jira/browse/CASSANDRA-16053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179209#comment-17179209 ]
Adam Holmberg commented on CASSANDRA-16053:
-------------------------------------------
CASSANDRA-11749 introduced passing the cluster to clean up sockets in forked child processes. That patch assumed that forking would be used everywhere but Windows.
Instead of making the parameter passing logic more complex to account for platform and runtime variations, I suggest we rely on the underlying driver fix which was later introduced, and stop trying to pass the Cluster.
[PYTHON-577|https://datastax-oss.atlassian.net/browse/PYTHON-577]
[patch|https://github.com/apache/cassandra/compare/trunk...aholmberg:CASSANDRA-16053?expand=1]
I've run the repro script from CASSANDRA-11749 several hundred times. Stefania indicated that it would previously reproduce within around a half dozen iterations.
> cqlsh COPY functions broken in Python 3.8 on Mac
> ------------------------------------------------
>
> Key: CASSANDRA-16053
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16053
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/cqlsh
> Reporter: Adam Holmberg
> Assignee: Adam Holmberg
> Priority: Normal
> Fix For: 4.0-beta2
>
>
> {noformat}
> aholmberg-rmbp16:cassandra adamholmberg$ pyenv shell
> 3.8.2
> aholmberg-rmbp16:cassandra adamholmberg$ bin/cqlsh --debug
> Using CQL driver: <module 'cassandra' from '/Users/adamholmberg/code/cassandra/bin/../lib/cassandra-driver-internal-only-3.23.0.post0-1a184b99.zip/cassandra-driver-3.23.0.post0-1a184b99/cassandra/__init__.py'>
> Using connect timeout: 5 seconds
> Using 'utf-8' encoding
> Using ssl: False
> Connected to Test Cluster at 127.0.0.1:9042.
> [cqlsh 5.0.1 | Cassandra 4.0-beta2-SNAPSHOT | CQL spec 3.4.5 | Native protocol v4]
> Use HELP for help.
> cqlsh> copy test.testcopyto to 'asdf';
> Detected 12 core(s)
> Using 11 child processes
> Starting copy of test.testcopyto with columns [a, b, c, d].
> Traceback (most recent call last):
> File "/Users/adamholmberg/code/cassandra/bin/cqlsh.py", line 937, in onecmd
> self.handle_statement(st, statementtext)
> File "/Users/adamholmberg/code/cassandra/bin/cqlsh.py", line 974, in handle_statement
> return custom_handler(parsed)
> File "/Users/adamholmberg/code/cassandra/bin/cqlsh.py", line 1563, in do_copy
> task.run()
> File "/Users/adamholmberg/code/cassandra/bin/../pylib/cqlshlib/copyutil.py", line 669, in run
> self.start_processes()
> File "/Users/adamholmberg/code/cassandra/bin/../pylib/cqlshlib/copyutil.py", line 471, in start_processes
> process.start()
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/process.py", line 121, in start
> self._popen = self._Popen(self)
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
> return _default_context.get_context().Process._Popen(process_obj)
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
> return Popen(process_obj)
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
> super().__init__(process_obj)
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
> self._launch(process_obj)
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
> reduction.dump(process_obj, fp)
> File "/Users/adamholmberg/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
> ForkingPickler(file, protocol).dump(obj)
> TypeError: cannot pickle '_thread.lock' object
> cqlsh>
> {noformat}
> multiprocessing uses a different default start method on Mac, and pickling fails trying to serialize the Cluster object.
> https://github.com/python/cpython/blob/db098bc1f05bd0773943e59f83489f05f28dedf8/Lib/multiprocessing/context.py#L313-L318
> https://bugs.python.org/issue33725
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org