You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Andrew Sheng (Jira)" <ji...@apache.org> on 2019/12/02 07:23:00 UTC
[jira] [Updated] (AIRFLOW-6144) Improve the log message of airflow
scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Sheng updated AIRFLOW-6144:
----------------------------------
Description:
It is a little difficult to understand the error message of airflow scheduler.
{code:java}
# start the scheduler
airflow scheduler{code}
{code:java}
[2019-12-02 14:33:56,384] {dag_processing.py:760} ERROR - Cannot use more than 1 thread when using sqlite. Setting parallelism to 1{code}
After I initialized the database as Quick Start description, the _max_threads_ was configured to 2 by default. Then I continued running the command _airflow scheduler_ as Quick Start description, but got an error message. It made me a little confusion.
So I have no choice but to confirm the logic in the _dag_processing.py:760_.
{code:java}
self._parallelism = conf.getint('scheduler', 'max_threads')
if 'sqlite' in conf.get('core', 'sql_alchemy_conn') and self._parallelism > 1:
self.log.error("Cannot use more than 1 thread when using sqlite. "
"Setting parallelism to 1")
self._parallelism = 1{code}
And I found that no problems which I have to deal with. It had been solved by itself.
Instead of outputting ERROR message, I think it is better to output WARNING message as below.
{code:java}
[2019-12-02 14:33:56,384] {dag_processing.py:760} WARNING - Because we cannot use more than 1 thread (max_threads = %d) when using sqlite. So we set parallelism to 1.{code}
was:
It is a little difficult to understand the error message of airflow scheduler.
{code:java}
# start the scheduler
airflow scheduler{code}
{code:java}
[2019-12-02 14:33:56,384] {dag_processing.py:760} ERROR - Cannot use more than 1 thread when using sqlite. Setting parallelism to 1{code}
After I initialized the database as Quick Start description, the _max_threads_ was configured to 2 by default. Then I continued running the command _airflow scheduler_ as Quick Start description, but got an error message. It made me a little confusion.
So I have no choice but to confirm the logic in the _dag_processing.py:760_.
{code:java}
self._parallelism = conf.getint('scheduler', 'max_threads')
if 'sqlite' in conf.get('core', 'sql_alchemy_conn') and self._parallelism > 1:
self.log.error("Cannot use more than 1 thread when using sqlite. "
"Setting parallelism to 1")
self._parallelism = 1{code}
And I found that no problems which I have to deal with. It had been solved by itself.
Instead of outputting ERROR message, I think it is better to output WARNING message as below.
{code:java}
[2019-12-02 14:33:56,384] {dag_processing.py:760} WARNING - Because we cannot use more than 1 thread (max_threads = %d) when using sqlite. So we set parallelism to 1.{code}
> Improve the log message of airflow scheduler
> --------------------------------------------
>
> Key: AIRFLOW-6144
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6144
> Project: Apache Airflow
> Issue Type: Improvement
> Components: DagRun
> Affects Versions: 1.10.6
> Reporter: Andrew Sheng
> Priority: Minor
>
> It is a little difficult to understand the error message of airflow scheduler.
> {code:java}
> # start the scheduler
> airflow scheduler{code}
> {code:java}
> [2019-12-02 14:33:56,384] {dag_processing.py:760} ERROR - Cannot use more than 1 thread when using sqlite. Setting parallelism to 1{code}
> After I initialized the database as Quick Start description, the _max_threads_ was configured to 2 by default. Then I continued running the command _airflow scheduler_ as Quick Start description, but got an error message. It made me a little confusion.
> So I have no choice but to confirm the logic in the _dag_processing.py:760_.
> {code:java}
> self._parallelism = conf.getint('scheduler', 'max_threads')
> if 'sqlite' in conf.get('core', 'sql_alchemy_conn') and self._parallelism > 1:
> self.log.error("Cannot use more than 1 thread when using sqlite. "
> "Setting parallelism to 1")
> self._parallelism = 1{code}
> And I found that no problems which I have to deal with. It had been solved by itself.
> Instead of outputting ERROR message, I think it is better to output WARNING message as below.
> {code:java}
> [2019-12-02 14:33:56,384] {dag_processing.py:760} WARNING - Because we cannot use more than 1 thread (max_threads = %d) when using sqlite. So we set parallelism to 1.{code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)