You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/06 12:00:08 UTC
[GitHub] [airflow] gastonlau opened a new issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
gastonlau opened a new issue #17120:
URL: https://github.com/apache/airflow/issues/17120
Hi Airflow Team,
I am running the Airflow in an EC2 instance which is installed by conda-forge during the Code Deploy.
After upgrading the Airflow version from 2.0.2 to >=2.1.0, I am facing an error every time when I try to start the scheduler in daemon mode using this command: ```airflow scheduler --daemon```
I took a look at a similar issue #11456 and try to fix it with Python 3.8.10 and `python-daemon` 2.3.0 but still doesn't work.
The webserver is working fine, but it can't detect the scheduler.
```
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
```
Relevant package version:
`sqlalchemy`=1.3.23
`psycopg2`=2.8.6
`python-daemon`=2.3.0
`apache-airflow-providers-http`=2.0.0
`apache-airflow-providers-elasticsearch`=2.0.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893148745
@potiuk Without webserver and scheduler together, airflow is incomplete. When scheduler service starts when we start webserver. We can have a separate module also to restart the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-930554979
There is no fix, because we do not know what the issue is. Actually your case shows an interesting problem and maybe the root cause for the issue:
OSError: [Errno 98] Address already in use
It looks like this is a deployment issue where you try to run two airflow components on the same server - schedulers and workers for example and they are all serving the same logs. The problem is most likely with your deployment. I am converting this to a discussion as this is likely a deployment issue.
Can you please describe @agentdanger what is the deployment you are using and how it hapened that you run several airflow components on the same server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] txicos commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
txicos commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893024484
Hi,
Hi I also applied suggested correction and raised an Exception just after entering method `create_scheduler_job`, which indeed occurred when executing `airflow scheduler -D`.
However, I am also getting the error whose log is added below:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/serve_logs.py", line 85, in serve_logs
app.run(host='0.0.0.0', port=worker_log_server_port)
File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow//.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 63, in scheduler
job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gastonlau commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
gastonlau commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-884594951
> Curious - does it work when you run airflow scheduler without the --daemon flag?
I think it works as the webserver can detect the scheduler is running and I can't see the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893148745
@potiuk Without webserver and scheduler together, airflow is incomplete. When webserver service starts, we can start scheduler service as a integral part. We can have a separate module also to restart the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-889114171
@potiuk is this fix in production? I am having postgres as database.
I am using below version packages.
Airflow - 2.1.2
sqlalchemy=1.3.24
psycopg2=2.9.1
Getting below error though i started the scheduler as "airflow scheduler -D".
"he scheduler does not appear to be running. Last heartbeat was received 3 hours ago.
The DAGs list may not update, and new tasks will not be scheduled."
When checked airflow.scheduler.err got below error.
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/scheduler_command.py", line 58, in scheduler
job.run()
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
I will be highly obliged if you can let me know how to fix this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-884352953
Curious - does it work when you run airflow without the --deamon flag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-883822012
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-894210585
As a workaround, you do not need to run airflow in Daemon mode.
There is no need to run daemon mode in most cases. You can run airflow in a background using whatever mechanisms you have to manage your deploymens (upstart/systemd/etc. ) and there is no need to use daemon mode at all..
Usually I recommend using Airlfow via docker images - where each component is a separate container - with Ariflow Helm Chart if you run on Kubernetes https://airflow.apache.org/docs/helm-chart/stable/index.html or with Docker Compose, or other deployment mechanisms you use. I am not sure why you are getting the error if you confirmed the problem is not fixed. I could not reproduce it so maybe it's also some problem with your environment or you have some customizations that inject sql-alchemy initialization befre the fork happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] agentdanger commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
agentdanger commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-929580803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17120:
URL: https://github.com/apache/airflow/issues/17120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-889114171
@potiuk is this fix in production? I am having postgres as database.
I am using below version packages.
Airflow - 2.1.2
sqlalchemy=1.3.24
psycopg2=2.9.1
Getting below error though i started the scheduler as "airflow scheduler -D".
"he scheduler does not appear to be running. Last heartbeat was received 3 hours ago.
The DAGs list may not update, and new tasks will not be scheduled."
When checked airflow.scheduler.err got below error.
```
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/scheduler_command.py", line 58, in scheduler
job.run()
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
```
I will be highly obliged if you can let me know how to fix this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-884352953
Curious - does it work when you run airflow scheduler without the --daemon flag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-889797457
> File "/usr/local/lib/python3.8/dist-packages/airflow/cli/cli_parser.py", line 48, in command
There might be multiple reasons why the change might be not taken into account when you just replace the files (.pyc code not recompiled, different location of airflow installation)
You can check the actual location by running this in your python interpreter (but it can also be installed in virtualenv, so you need to make sure you have that virtualenv activated).
```
import airflow.cli
airflow.cli.__file__
```
Usually - to be 100% sure, i change such code and raise some hard exception there directly and see if *really* the modified code is used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893148745
@potiuk Without webserver and scheduler together, airflow is incomplete. When scheduler service starts when we start webserver. We can have a separate module also to restart the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-930554979
There is no fix, because we do not know what the issue is. Actually your case shows an interesting problem and maybe the root cause for the issue:
OSError: [Errno 98] Address already in use
It looks like this is a deployment issue where you try to run two airflow components on the same server - schedulers and workers for example and they are all serving the same logs. The problem is most likely with your deployment. I am converting this to a discussion as this is likely a deployment issue.
Can you please describe @agentdanger what is the deployment you are using and how it hapened that you run several airflow components on the same server?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] txicos edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
txicos edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893024484
Hi,
I also applied suggested correction and raised an Exception just after entering method `create_scheduler_job`, which indeed occurred when executing `airflow scheduler -D`.
However, I am also getting the error whose log is added below:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/serve_logs.py", line 85, in serve_logs
app.run(host='0.0.0.0', port=worker_log_server_port)
File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow//.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 63, in scheduler
job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893148745
@potiuk Without webserver and scheduler together, airflow is incomplete. When webserver service starts, we can start scheduler service as a integral part. We can have a separate module also to restart the scheduler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-884753926
I think I know the reason. In Scheduler, the `SchedulerJob` is instantiated before demon context is activated. SchedulerJob is a database ORM object from SQL Alchemy and it opens the connection to Postgres:
```
job = SchedulerJob(
subdir=process_subdir(args.subdir),
num_runs=args.num_runs,
do_pickle=args.do_pickle,
)
```
When you activate daemon context, what happens under the hood is forking the process, and while some of the opened sockets are passed to the forks (stdin and stderr but also the opened log file handle), the established socket for DB connection is not passed:
```
handle = setup_logging(log_file)
with open(stdout, 'w+') as stdout_handle, open(stderr, 'w+') as stderr_handle:
ctx = daemon.DaemonContext(
pidfile=TimeoutPIDLockFile(pid, -1),
files_preserve=[handle],
stdout=stdout_handle,
stderr=stderr_handle,
)
```
I will add a fix for that in a moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-889600308
Thank you @potiuk for quick reply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17120:
URL: https://github.com/apache/airflow/issues/17120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-884352953
Curious - does it work when you run airflow scheduler without the --deamon flag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-889310401
You have to wait until it is released. The PR has just been approved few hours ago and it's going to be released with one of the next releases of airflow (depending if we manage to cherry-pick it before relase 2.1.3 or whether it comes in 2.2, this might be in a week more or less or a month. The release needs to be announced, voted and published..
You can also manually apply to your version by cherry-picking this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] txicos edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
txicos edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893024484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17120:
URL: https://github.com/apache/airflow/issues/17120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] txicos commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
txicos commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893024484
Hi,
Hi I also applied suggested correction and raised an Exception just after entering method `create_scheduler_job`, which indeed occurred when executing `airflow scheduler -D`.
However, I am also getting the error whose log is added below:
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/serve_logs.py", line 85, in serve_logs
app.run(host='0.0.0.0', port=worker_log_server_port)
File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow//.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 63, in scheduler
job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudipto9 commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
sudipto9 commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-889663324
@potiuk I have copied the code in the external Libraries's dist-packages/airflow/cli/commands folder. After that Iran below commands to start airflow again.
airflow db init
airflow webserver -D
airflow scheduler -D
but the problem still exists. Please do find the err log below.
raceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/cli/commands/scheduler_command.py", line 54, in scheduler
job.run()
File "/usr/local/lib/python3.8/dist-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Also when I use SUDO airflow webserver -D, it is not starting the webserver though getting webserver start msg.
2021-07-30 10:40:45,893] {backend.py:193} INFO - Loading KWallet
[2021-07-30 10:40:45,900] {backend.py:193} INFO - Loading SecretService
[2021-07-30 10:40:45,902] {backend.py:193} INFO - Loading Windows
[2021-07-30 10:40:45,903] {backend.py:193} INFO - Loading chainer
[2021-07-30 10:40:45,903] {backend.py:193} INFO - Loading macOS
[2021-07-30 10:40:47,022] {arrow_result.pyx:0} INFO - Failed to import optional packages, pyarrow
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-30 10:40:47,087] {dagbag.py:496} INFO - Filling up the DagBag from /dev/null
[2021-07-30 10:40:47,155] {manager.py:788} WARNING - No user yet created, use flask fab command to do it.
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
Access Logformat:
On the contrary when using scheduler -D, not start the scheduler.
I am not sure if I am making any mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] txicos edited a comment on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
txicos edited a comment on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-893024484
Hi,
I also applied suggested correction and raised an Exception just after entering method `create_scheduler_job`, which indeed occurred when executing `airflow scheduler -D`.
However, I am also getting the error whose log is added below:
```
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/serve_logs.py", line 85, in serve_logs
app.run(host='0.0.0.0', port=worker_log_server_port)
File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 990, in run
run_simple(host, port, self, **options)
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 1052, in run_simple
inner()
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 996, in inner
srv = make_server(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 847, in make_server
return ThreadedWSGIServer(
File "/home/airflow/.local/lib/python3.8/site-packages/werkzeug/serving.py", line 740, in __init__
HTTPServer.__init__(self, server_address, handler)
File "/usr/lib/python3.8/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/lib/python3.8/http/server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.8/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow//.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 63, in scheduler
job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 241, in run
session.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1046, in commit
self.transaction.commit()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2540, in flush
self._flush(objects)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2682, in _flush
transaction.rollback(_capture_exception=True)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2642, in _flush
flush_context.execute()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 586, in execute
persistence.save_obj(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 205, in save_obj
for (
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 373, in _organize_states_for_save
for state, dict_, mapper, connection in _connections_for_states(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1602, in _connections_for_states
connection = uowtransaction.transaction.connection(base_mapper)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 314, in connection
return self._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 421, in _connection_for_bind
conn = self._parent._connection_for_bind(bind, execution_options)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
conn = bind._contextual_connect()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect
Connection._handle_dbapi_exception_noconnection(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exception_noconnection
util.raise_(
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
return fn()
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect
return _ConnectionFairy._checkout(self)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 809, in _checkout
result = pool._dialect.do_ping(fairy.connection)
File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 575, in do_ping
cursor.execute(self._dialect_specific_select_one)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: Socket operation on non-socket
(Background on this error at: http://sqlalche.me/e/13/e3q8)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17120: [Scheduler error] psycopg2.OperationalError: SSL SYSCALL error: Socket operation on non-socket
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17120:
URL: https://github.com/apache/airflow/issues/17120#issuecomment-894210585
As a workaround, you do not need to run airflow in Daemon mode.
There is no need to run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org