[GitHub] [airflow] PApostol commented on issue #8212: Can't read S3 remote logs when using gevent/eventlent webserver workers.

[GitBox <gi...@apache.org>]

PApostol commented on issue #8212:
URL: https://github.com/apache/airflow/issues/8212#issuecomment-1361128280

   Hi @potiuk, we've also had the same issue with S3 remote logs and I've tried out your fix. While the fix succeeds in fetching the logs from S3 on the UI, it seems to break the scheduler upon restart. The error is reproducible every time I try to start the scheduler after the fix is applied (note that webserver works fine). Removing the fix from `airflow/__init__.py` resolves the error. I'm attaching the traceback below. Using Airflow: 2.3.4, Python 3.7.3, LocalExecutor.
   ```
   Traceback (most recent call last):
     File "/home/ec2-user/venv/bin/airflow", line 8, in <module>
       sys.exit(main())
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/__main__.py", line 38, in main
       args.func(args)
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 51, in command
       return func(*args, **kwargs)
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
       return f(*args, **kwargs)
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 76, in scheduler
       _run_scheduler_job(args=args)
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/cli/commands/scheduler_command.py", line 46, in _run_scheduler_job
       job.run()
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 244, in run
       self._execute()
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 743, in _execute
       self.executor.start()
     File "/home/ec2-user/venv/lib/python3.7/site-packages/airflow/executors/local_executor.py", line 348, in start
       self.result_queue = self.manager.Queue()
     File "/usr/lib64/python3.7/multiprocessing/managers.py", line 724, in temp
       token, exp = self._create(typeid, *args, **kwds)
     File "/usr/lib64/python3.7/multiprocessing/managers.py", line 607, in _create
       conn = self._Client(self._address, authkey=self._authkey)
     File "/usr/lib64/python3.7/multiprocessing/connection.py", line 498, in Client
       answer_challenge(c, authkey)
     File "/usr/lib64/python3.7/multiprocessing/connection.py", line 742, in answer_challenge
       message = connection.recv_bytes(256)         # reject large message
     File "/usr/lib64/python3.7/multiprocessing/connection.py", line 216, in recv_bytes
       buf = self._recv_bytes(maxlength)
     File "/usr/lib64/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
       buf = self._recv(4)
     File "/usr/lib64/python3.7/multiprocessing/connection.py", line 379, in _recv
       chunk = read(handle, remaining)
   BlockingIOError: [Errno 11] Resource temporarily unavailable
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org