You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Pierre Jeambrun <pi...@gmail.com> on 2023/05/01 13:07:04 UTC
Re: Issue with airflow and postgresDB database
Hello Mich,
Seems like the BaseSensorOperator is not imported correctly in your
plugin. *BaseSensorOperator
*belongs to the `airflow.sensors.base` module:
https://github.com/apache/airflow/blob/2.5.3/airflow/sensors/base.py#L77
A quick search on the project requirements shows that postgres < 11 is not
supported. This might explain the upgrade issue you are experiencing:
https://github.com/apache/airflow/tree/2.5.3#requirements
Best regards,
Pierre
Le lun. 1 mai 2023 à 16:57, Mich Talebzadeh <mi...@gmail.com> a
écrit :
> Hi,
>
> I recently upgraded Python from 3.7 to 3.9.16
>
> My current airflow version is 2.5.3
> My backend DB is PostresDB version 9.2.24
>
> Airflow webserver is failing to start coming back with below error
>
> [2023-05-01 15:50:02,052] {manager.py:824} WARNING - No user yet created,
> use flask fab command to do it.
> [2023-05-01 15:50:02,906] {plugins_manager.py:266} ERROR - Failed to
> import plugin /root/airflow/plugins/my_operators.py
> Traceback (most recent call last):
> File
> "/data1/src/Python-3.9.16/venv/venv3.9/lib/python3.9/site-packages/airflow/plugins_manager.py",
> line 258, in load_plugins_from_plugin_directory
> loader.exec_module(mod)
> File "<frozen importlib._bootstrap_external>", line 850, in exec_module
> File "<frozen importlib._bootstrap>", line 228, in
> _call_with_frames_removed
> File "/root/airflow/plugins/my_operators.py", line 8, in <module>
> from airflow.operators.sensors import BaseSensorOperator
> ModuleNotFoundError: No module named 'airflow.operators.sensors'
> [2023-05-01 15:50:04 +0100] [25126] [INFO] Starting gunicorn 20.1.0
> [2023-05-01 15:50:04 +0100] [25126] [INFO] Listening at:
> http://0.0.0.0:9099 (25126)
> [2023-05-01 15:50:04 +0100] [25126] [INFO] Using worker: sync
>
>
> And when I try to do
>
> airflow db init
>
> I get
>
>
> sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject)
> unrecognized configuration parameter "lock_timeout"
>
> [SQL: SET LOCK_TIMEOUT TO DEFAULT]
>
>
>
> Appreciate any ideas
>
>
> Thanks
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
Re: Issue with airflow and postgresDB database
Posted by Mich Talebzadeh <mi...@gmail.com>.
OK I created a new postgresDB backend on 15.2
So I now have
(PostgreSQL) 15.2
Airflow 2.5.3
Python 3.9.1.16
And with regard to BaseSensorOperator
The following import works in the plugins
from airflow.sensors.base import BaseSensorOperator
HTH
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 1 May 2023 at 18:39, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Thanks Pierre, I will have a look.
>
> Regards
>
> Mich Talebzadeh,
> Lead Solutions Architect/Engineering Lead
> Palantir Technologies Limited
> London
> United Kingdom
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 1 May 2023 at 16:10, Pierre Jeambrun <pi...@gmail.com>
> wrote:
>
>> Indeed *lock_timeout *is new in 9.3, hence the error. You should update
>> your postgres to >= 11 if you want to be able to use the latest airflow
>> releases.
>> https://postgresqlco.nf/doc/en/param/lock_timeout/
>>
>> Le lun. 1 mai 2023 à 15:07, Pierre Jeambrun <pi...@gmail.com> a
>> écrit :
>>
>>> Hello Mich,
>>>
>>> Seems like the BaseSensorOperator is not imported correctly in your
>>> plugin. *BaseSensorOperator *belongs to the `airflow.sensors.base`
>>> module:
>>> https://github.com/apache/airflow/blob/2.5.3/airflow/sensors/base.py#L77
>>>
>>> A quick search on the project requirements shows that postgres < 11 is
>>> not supported. This might explain the upgrade issue you are experiencing:
>>> https://github.com/apache/airflow/tree/2.5.3#requirements
>>>
>>> Best regards,
>>> Pierre
>>>
>>>
>>>
>>> Le lun. 1 mai 2023 à 16:57, Mich Talebzadeh <mi...@gmail.com>
>>> a écrit :
>>>
>>>> Hi,
>>>>
>>>> I recently upgraded Python from 3.7 to 3.9.16
>>>>
>>>> My current airflow version is 2.5.3
>>>> My backend DB is PostresDB version 9.2.24
>>>>
>>>> Airflow webserver is failing to start coming back with below error
>>>>
>>>> [2023-05-01 15:50:02,052] {manager.py:824} WARNING - No user yet
>>>> created, use flask fab command to do it.
>>>> [2023-05-01 15:50:02,906] {plugins_manager.py:266} ERROR - Failed to
>>>> import plugin /root/airflow/plugins/my_operators.py
>>>> Traceback (most recent call last):
>>>> File
>>>> "/data1/src/Python-3.9.16/venv/venv3.9/lib/python3.9/site-packages/airflow/plugins_manager.py",
>>>> line 258, in load_plugins_from_plugin_directory
>>>> loader.exec_module(mod)
>>>> File "<frozen importlib._bootstrap_external>", line 850, in
>>>> exec_module
>>>> File "<frozen importlib._bootstrap>", line 228, in
>>>> _call_with_frames_removed
>>>> File "/root/airflow/plugins/my_operators.py", line 8, in <module>
>>>> from airflow.operators.sensors import BaseSensorOperator
>>>> ModuleNotFoundError: No module named 'airflow.operators.sensors'
>>>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Starting gunicorn 20.1.0
>>>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Listening at:
>>>> http://0.0.0.0:9099 (25126)
>>>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Using worker: sync
>>>>
>>>>
>>>> And when I try to do
>>>>
>>>> airflow db init
>>>>
>>>> I get
>>>>
>>>>
>>>> sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject)
>>>> unrecognized configuration parameter "lock_timeout"
>>>>
>>>> [SQL: SET LOCK_TIMEOUT TO DEFAULT]
>>>>
>>>>
>>>>
>>>> Appreciate any ideas
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>
Re: Issue with airflow and postgresDB database
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Pierre, I will have a look.
Regards
Mich Talebzadeh,
Lead Solutions Architect/Engineering Lead
Palantir Technologies Limited
London
United Kingdom
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Mon, 1 May 2023 at 16:10, Pierre Jeambrun <pi...@gmail.com> wrote:
> Indeed *lock_timeout *is new in 9.3, hence the error. You should update
> your postgres to >= 11 if you want to be able to use the latest airflow
> releases.
> https://postgresqlco.nf/doc/en/param/lock_timeout/
>
> Le lun. 1 mai 2023 à 15:07, Pierre Jeambrun <pi...@gmail.com> a
> écrit :
>
>> Hello Mich,
>>
>> Seems like the BaseSensorOperator is not imported correctly in your
>> plugin. *BaseSensorOperator *belongs to the `airflow.sensors.base`
>> module:
>> https://github.com/apache/airflow/blob/2.5.3/airflow/sensors/base.py#L77
>>
>> A quick search on the project requirements shows that postgres < 11 is
>> not supported. This might explain the upgrade issue you are experiencing:
>> https://github.com/apache/airflow/tree/2.5.3#requirements
>>
>> Best regards,
>> Pierre
>>
>>
>>
>> Le lun. 1 mai 2023 à 16:57, Mich Talebzadeh <mi...@gmail.com>
>> a écrit :
>>
>>> Hi,
>>>
>>> I recently upgraded Python from 3.7 to 3.9.16
>>>
>>> My current airflow version is 2.5.3
>>> My backend DB is PostresDB version 9.2.24
>>>
>>> Airflow webserver is failing to start coming back with below error
>>>
>>> [2023-05-01 15:50:02,052] {manager.py:824} WARNING - No user yet
>>> created, use flask fab command to do it.
>>> [2023-05-01 15:50:02,906] {plugins_manager.py:266} ERROR - Failed to
>>> import plugin /root/airflow/plugins/my_operators.py
>>> Traceback (most recent call last):
>>> File
>>> "/data1/src/Python-3.9.16/venv/venv3.9/lib/python3.9/site-packages/airflow/plugins_manager.py",
>>> line 258, in load_plugins_from_plugin_directory
>>> loader.exec_module(mod)
>>> File "<frozen importlib._bootstrap_external>", line 850, in exec_module
>>> File "<frozen importlib._bootstrap>", line 228, in
>>> _call_with_frames_removed
>>> File "/root/airflow/plugins/my_operators.py", line 8, in <module>
>>> from airflow.operators.sensors import BaseSensorOperator
>>> ModuleNotFoundError: No module named 'airflow.operators.sensors'
>>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Starting gunicorn 20.1.0
>>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Listening at:
>>> http://0.0.0.0:9099 (25126)
>>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Using worker: sync
>>>
>>>
>>> And when I try to do
>>>
>>> airflow db init
>>>
>>> I get
>>>
>>>
>>> sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject)
>>> unrecognized configuration parameter "lock_timeout"
>>>
>>> [SQL: SET LOCK_TIMEOUT TO DEFAULT]
>>>
>>>
>>>
>>> Appreciate any ideas
>>>
>>>
>>> Thanks
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>
Re: Issue with airflow and postgresDB database
Posted by Pierre Jeambrun <pi...@gmail.com>.
Indeed *lock_timeout *is new in 9.3, hence the error. You should update
your postgres to >= 11 if you want to be able to use the latest airflow
releases.
https://postgresqlco.nf/doc/en/param/lock_timeout/
Le lun. 1 mai 2023 à 15:07, Pierre Jeambrun <pi...@gmail.com> a
écrit :
> Hello Mich,
>
> Seems like the BaseSensorOperator is not imported correctly in your
> plugin. *BaseSensorOperator *belongs to the `airflow.sensors.base` module:
> https://github.com/apache/airflow/blob/2.5.3/airflow/sensors/base.py#L77
>
> A quick search on the project requirements shows that postgres < 11 is not
> supported. This might explain the upgrade issue you are experiencing:
> https://github.com/apache/airflow/tree/2.5.3#requirements
>
> Best regards,
> Pierre
>
>
>
> Le lun. 1 mai 2023 à 16:57, Mich Talebzadeh <mi...@gmail.com> a
> écrit :
>
>> Hi,
>>
>> I recently upgraded Python from 3.7 to 3.9.16
>>
>> My current airflow version is 2.5.3
>> My backend DB is PostresDB version 9.2.24
>>
>> Airflow webserver is failing to start coming back with below error
>>
>> [2023-05-01 15:50:02,052] {manager.py:824} WARNING - No user yet created,
>> use flask fab command to do it.
>> [2023-05-01 15:50:02,906] {plugins_manager.py:266} ERROR - Failed to
>> import plugin /root/airflow/plugins/my_operators.py
>> Traceback (most recent call last):
>> File
>> "/data1/src/Python-3.9.16/venv/venv3.9/lib/python3.9/site-packages/airflow/plugins_manager.py",
>> line 258, in load_plugins_from_plugin_directory
>> loader.exec_module(mod)
>> File "<frozen importlib._bootstrap_external>", line 850, in exec_module
>> File "<frozen importlib._bootstrap>", line 228, in
>> _call_with_frames_removed
>> File "/root/airflow/plugins/my_operators.py", line 8, in <module>
>> from airflow.operators.sensors import BaseSensorOperator
>> ModuleNotFoundError: No module named 'airflow.operators.sensors'
>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Starting gunicorn 20.1.0
>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Listening at:
>> http://0.0.0.0:9099 (25126)
>> [2023-05-01 15:50:04 +0100] [25126] [INFO] Using worker: sync
>>
>>
>> And when I try to do
>>
>> airflow db init
>>
>> I get
>>
>>
>> sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject)
>> unrecognized configuration parameter "lock_timeout"
>>
>> [SQL: SET LOCK_TIMEOUT TO DEFAULT]
>>
>>
>>
>> Appreciate any ideas
>>
>>
>> Thanks
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>