You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Eila Oriel Research <ei...@orielresearch.org> on 2023/01/17 21:22:00 UTC
setup.py gsutil cp command Error 98
Hello,
I am using an old code from setup.py to copy a file from gs to the worker.
I am receiving the following error message:
['mkdir','-p','/opt/userowned/'],
["chmod", "777", "/opt/userowned/"],
["gsutil","cp","gs://ort_tools/anaconda/anaconda.sh", "/opt/userowned/"]
insertId: "2551510730791367555:450:0:37197"
jsonPayload: {
stream: "stderr"
message: "2023/01/17 21:06:47 RuntimeError: Command ['gsutil', 'cp',
'gs://ort_tools/anaconda/anaconda.sh', '/opt/userowned/'] failed: exit
code: 1 "
}
The command output is the following:
2023-01-17 16:06:47.914 EST
2023/01/17 21:06:47 Command output: b'Process SyncManager-2:\nTraceback
(most recent call last):\n File
"/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in
_bootstrap\n self.run()\n File
"/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run\n
self._target(*self._args, **self._kwargs)\n File
"/usr/local/lib/python3.9/multiprocessing/managers.py", line 583, in
_run_server\n server = cls._Server(registry, address, authkey,
serializer)\n File "/usr/local/lib/python3.9/multiprocessing/managers.py",
line 156, in __init__\n self.listener = Listener(address=address,
backlog=16)\n File
"/usr/local/lib/python3.9/multiprocessing/connection.py", line 453, in
__init__\n self._listener = SocketListener(address, family, backlog)\n File
"/usr/local/lib/python3.9/multiprocessing/connection.py", line 596, in
__init__\n self._socket.bind(address)\nOSError: [Errno 98] Address already
in use\nTraceback (most recent call last):\n File
"/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
<module>\n gsutil.RunMain()\n File
"/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gsutil.py", line 151,
in RunMain\n sys.exit(gslib.__main__.main())\n File
"/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gslib/__main__.py",
line 239, in main\n gslib.command.InitializeMultiprocessingVariables()\n
File "/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gslib/command.py",
line 337, in InitializeMultiprocessingVariables\n manager =
multiprocessing_context.Manager()\n File
"/usr/local/lib/python3.9/multiprocessing/context.py", line 57, in
Manager\n m.start()\n File
"/usr/local/lib/python3.9/multiprocessing/managers.py", line 558, in
start\n self._address = reader.recv()\n File
"/usr/local/lib/python3.9/multiprocessing/connection.py", line 255, in
recv\n buf = self._recv_bytes()\n File
"/usr/local/lib/python3.9/multiprocessing/connection.py", line 419, in
_recv_bytes\n buf = self._recv(4)\n File
"/usr/local/lib/python3.9/multiprocessing/connection.py", line 388, in
_recv\n raise EOFError\nEOFError\n'
I tried the gsutil cp command on local machine and didn't have any issues.
This code used to work with no issue.
Any suggestions why is it firing [Errno 98] Address already in use?
Thank you,
--
Eila Arich
Founder, CEO
Oriel Research Therapeutics (ORT)
https://www.orielresearch.com/blog
eila@orielresearch.com
www.orielresearch.com
Newton, MA / Mountain View, CA
[image: twitter] <https://twitter.com/eilalan1>
[image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
[image: instagram] <https://www.instagram.com/eilalan/>
Re: setup.py gsutil cp command Error 98
Posted by Eila Oriel Research <ei...@orielresearch.org>.
Hi All,
It seems that I will need to edit all the setup.py files' custom_commands
section with 'sudo' .
Is that correct? Were the privileges of the setup.py execution changed?
If yes, is there a simpler solution other than editing the seup.py files?
Thank you,
Eila
On Tue, Jan 17, 2023 at 1:22 PM Eila Oriel Research <ei...@orielresearch.org>
wrote:
> Hello,
>
> I am using an old code from setup.py to copy a file from gs to the worker.
> I am receiving the following error message:
>
>
> ['mkdir','-p','/opt/userowned/'],
> ["chmod", "777", "/opt/userowned/"],
> ["gsutil","cp","gs://ort_tools/anaconda/anaconda.sh", "/opt/userowned/"]
>
>
> insertId: "2551510730791367555:450:0:37197"
> jsonPayload: {
> stream: "stderr"
> message: "2023/01/17 21:06:47 RuntimeError: Command ['gsutil', 'cp',
> 'gs://ort_tools/anaconda/anaconda.sh', '/opt/userowned/'] failed: exit
> code: 1 "
> }
>
> The command output is the following:
> 2023-01-17 16:06:47.914 EST
> 2023/01/17 21:06:47 Command output: b'Process SyncManager-2:\nTraceback
> (most recent call last):\n File
> "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in
> _bootstrap\n self.run()\n File
> "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run\n
> self._target(*self._args, **self._kwargs)\n File
> "/usr/local/lib/python3.9/multiprocessing/managers.py", line 583, in
> _run_server\n server = cls._Server(registry, address, authkey,
> serializer)\n File "/usr/local/lib/python3.9/multiprocessing/managers.py",
> line 156, in __init__\n self.listener = Listener(address=address,
> backlog=16)\n File
> "/usr/local/lib/python3.9/multiprocessing/connection.py", line 453, in
> __init__\n self._listener = SocketListener(address, family, backlog)\n File
> "/usr/local/lib/python3.9/multiprocessing/connection.py", line 596, in
> __init__\n self._socket.bind(address)\nOSError: [Errno 98] Address already
> in use\nTraceback (most recent call last):\n File
> "/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gsutil", line 21, in
> <module>\n gsutil.RunMain()\n File
> "/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gsutil.py", line 151,
> in RunMain\n sys.exit(gslib.__main__.main())\n File
> "/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gslib/__main__.py",
> line 239, in main\n gslib.command.InitializeMultiprocessingVariables()\n
> File "/usr/local/gcloud/google-cloud-sdk/platform/gsutil/gslib/command.py",
> line 337, in InitializeMultiprocessingVariables\n manager =
> multiprocessing_context.Manager()\n File
> "/usr/local/lib/python3.9/multiprocessing/context.py", line 57, in
> Manager\n m.start()\n File
> "/usr/local/lib/python3.9/multiprocessing/managers.py", line 558, in
> start\n self._address = reader.recv()\n File
> "/usr/local/lib/python3.9/multiprocessing/connection.py", line 255, in
> recv\n buf = self._recv_bytes()\n File
> "/usr/local/lib/python3.9/multiprocessing/connection.py", line 419, in
> _recv_bytes\n buf = self._recv(4)\n File
> "/usr/local/lib/python3.9/multiprocessing/connection.py", line 388, in
> _recv\n raise EOFError\nEOFError\n'
>
> I tried the gsutil cp command on local machine and didn't have any issues.
> This code used to work with no issue.
>
> Any suggestions why is it firing [Errno 98] Address already in use?
>
> Thank you,
> --
> Eila Arich
>
> Founder, CEO
>
> Oriel Research Therapeutics (ORT)
>
> https://www.orielresearch.com/blog
> eila@orielresearch.com
> www.orielresearch.com
> Newton, MA / Mountain View, CA
> [image: twitter] <https://twitter.com/eilalan1>
> [image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
> [image: instagram] <https://www.instagram.com/eilalan/>
>
>
--
Eila Arich
Founder, CEO
Oriel Research Therapeutics (ORT)
https://www.orielresearch.com/blog
eila@orielresearch.com
www.orielresearch.com
Newton, MA / Mountain View, CA
[image: twitter] <https://twitter.com/eilalan1>
[image: linkedin] <https://www.linkedin.com/in/eilalandkof/>
[image: instagram] <https://www.instagram.com/eilalan/>