You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/10 20:40:09 UTC
[GitHub] [airflow] dstandish opened a new issue #12999: scheduler error (sqlite3.OperationalError) database is locked in DagFileProcessorManager
dstandish opened a new issue #12999:
URL: https://github.com/apache/airflow/issues/12999
**Apache Airflow version**: 2.0.0rc1
**Environment**:
- **Cloud provider or hardware configuration**: mac
- **OS** (e.g. from /etc/os-release): macOS catalina
- **Kernel** (e.g. `uname -a`): Darwin TFGELSOS3516375 19.6.0 Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64 x86_64 i386 MacBookPro16,1 Darwin
- **Install tools**: pip
- **Others**: SequentialExecutor / sqlite
**What happened**:
Let some test dags run for a while in the background, seemed to be running fine, then got this error.
I may have started webserver during this time -- it was close in time but i don't know if was concurrent.
**What you expected to happen**:
no error
**How to reproduce it**:
I just let it run for a while using mssql operator and odbc hook. Not sure why it is calling "sudo"
**Other note**
After this error, i left scheduler running, and it seems to be doing fine -- have not seen it again yet.
When searching for issues i found this one https://github.com/apache/airflow/issues/12290 but that was 1.10.12.
**Log**
```
[2020-12-10 12:30:11,277] {scheduler_job.py:1200} INFO - Executor reports execution of test_dag.mssqltask execution_date=2020-12-10 20:25:00+00:00 exited with status success for try_number 1
[2020-12-10 12:30:12,390] {dagrun.py:444} INFO - Marking run <DagRun test_dag @ 2020-12-10 20:25:00+00:00: scheduled__2020-12-10T20:25:00+00:00, externally triggered: False> successful
Process ForkProcess-1:
Traceback (most recent call last):
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dstandish/.pyenv/versions/3.7.5/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Users/dstandish/.pyenv/versions/3.7.5/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 365, in _run_processor_manager
processor_manager.start()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 596, in start
return self._run_parsing_loop()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 661, in _run_parsing_loop
self._refresh_dag_dir()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/dag_processing.py", line 744, in _refresh_dag_dir
DagCode.remove_deleted_code(self._file_paths)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/session.py", line 65, in wrapper
return func(*args, session=session, **kwargs)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/models/dagcode.py", line 139, in remove_deleted_code
).delete(synchronize_session='fetch')
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3926, in delete
delete_op.exec_()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1697, in exec_
self._do_exec()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1928, in _do_exec
self._execute_stmt(delete_stmt)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1702, in _execute_stmt
self.result = self.query._execute_crud(stmt, self.mapper)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3568, in _execute_crud
return conn.execute(stmt, self._params)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: DELETE FROM dag_code WHERE dag_code.fileloc_hash NOT IN (?) AND dag_code.fileloc NOT IN (?)]
[parameters: (65669301801004096, '/Users/dstandish/code/abctest/dags/testing.py')]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
[2020-12-10 12:30:17,433] {base_job.py:222} ERROR - SchedulerJob heartbeat got an exception
Traceback (most recent call last):
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
self.engine.dialect.do_commit(self.connection)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 546, in do_commit
dbapi_connection.commit()
sqlite3.OperationalError: database is locked
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 195, in heartbeat
previous_heartbeat = self.latest_heartbeat
File "/Users/dstandish/.pyenv/versions/3.7.5/lib/python3.7/contextlib.py", line 119, in __exit__
next(self.gen)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/airflow/utils/session.py", line 32, in create_session
session.commit()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
self.transaction.commit()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 508, in commit
t[1].commit()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1762, in commit
self._do_commit()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1793, in _do_commit
self.connection._commit_impl()
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 773, in _commit_impl
self._handle_dbapi_exception(e, None, None, None, None)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 771, in _commit_impl
self.engine.dialect.do_commit(self.connection)
File "/Users/dstandish/.virtualenvs/abctest/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 546, in do_commit
dbapi_connection.commit()
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
(Background on this error at: http://sqlalche.me/e/13/e3q8)
[2020-12-10 12:30:18,510] {dag_processing.py:399} WARNING - DagFileProcessorManager (PID=5377) exited with exit code 1 - re-launching```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #12999: scheduler error (sqlite3.OperationalError) database is locked in DagFileProcessorManager
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #12999:
URL: https://github.com/apache/airflow/issues/12999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #12999: scheduler error (sqlite3.OperationalError) database is locked in DagFileProcessorManager
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #12999:
URL: https://github.com/apache/airflow/issues/12999#issuecomment-807731041
i will close for now it surfaced during testing but i have not tried sqlite for a while so i can't say if i can still repro
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vikramkoka commented on issue #12999: scheduler error (sqlite3.OperationalError) database is locked in DagFileProcessorManager
Posted by GitBox <gi...@apache.org>.
vikramkoka commented on issue #12999:
URL: https://github.com/apache/airflow/issues/12999#issuecomment-807674941
@dstandish Just trying to clean up some of the older issues. Is this still a problem?
I have run the scheduler / webserver for several hours with a local executor on SQLite without seeing this on 2.0.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org