You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Reed Villanueva <rv...@ucera.org> on 2019/12/17 21:34:11 UTC
airflow LocalExecutor initdb throws “sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2”
Attempting to set up airflow (v1.10.5) in a centos7 machine to use
LocalExecutor following the article here (
https://www.ryanmerlin.com/2019/07/apache-airflow-installation-on-ubuntu-18-04-18-10/),
so have settings...
[airflow@airflowetl airflow]$ rpm -q postgresql-server postgresql
postgresql-devel
postgresql-server-9.2.24-1.el7_5.x86_64
postgresql-9.2.24-1.el7_5.x86_64
postgresql-devel-9.2.24-1.el7_5.x86_64
[airflow@airflowetl airflow]$ pip3 freeze | grep sqlalchemy
marshmallow-sqlalchemy==0.19.0[airflow@airflowetl airflow]$ pip3
freeze | grep psycopg2
psycopg2==2.8.4
[airflow@airflowetl airflow]$ psql airflow
psql (9.2.24)Type "help" for help.
airflow=> \l
List of databases
Name | Owner | Encoding | Collate | Ctype |
Access privileges-----------+----------+----------+-------------+-------------+-----------------------
airflow | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
=Tc/postgres +
| | | | |
postgres=CTc/postgres+
| | | | |
airflow=CTc/postgres
postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
=c/postgres +
| | | | |
postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
=c/postgres +
| | | | |
postgres=CTc/postgres(4 rows)
[root@airflowetl airflow]# cat /var/lib/pgsql/data/pg_hba.conf....#
TYPE DATABASE USER ADDRESS METHOD
# "local" is for Unix domain socket connections onlylocal all
all peer# IPv4 local
connections:#host all all 127.0.0.1/32
ident
host all all 0.0.0.0/0 trust#
IPv6 local connections:
host all all ::1/128 ident#
Allow replication connections from localhost, by a user with the#
replication privilege.#local replication postgres
peer#host replication postgres
127.0.0.1/32 ident#host replication postgres
::1/128 ident
[root@airflowetl airflow]# cat
/var/lib/pgsql/data/postgresql.conf....# — — — — — — — — — — — — — — —
— — — — — — — — — — — — — — # CONNECTIONS AND AUTHENTICATION# — — — —
— — — — — — — — — — — — — — — — — — — — — — — — — # — Connection
Settings -#listen_addresses = ‘localhost’ # what IP address(es) to
listen on;
listen_addresses = ‘*’ # for Airflow connection
[airflow@airflowetl airflow]$ cat airflow.cfg....[core]....# The
executor class that airflow should use. Choices include#
SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor,
KubernetesExecutor#executor = SequentialExecutor
executor = LocalExecutor
# The SqlAlchemy connection string to the metadata database.#
SqlAlchemy supports many different database engine, more information#
their website#sql_alchemy_conn =
sqlite:////home/airflow/airflow/airflow.db
sql_alchemy_conn =
postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
Yet am seeing
File
"/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 240, in load "Can't load plugin: %s:%s" % (self.group, name)
sqlalchemy.exc.NoSuchModuleError: Can't load plugin:
sqlalchemy.dialects:postgresql.pyscopg2
error when trying to run airflow initdb. Checking if psycopg2 is in fact
installed, seeing...
[airflow@airflowetl airflow]$ python3Python 3.6.8 (default, Aug 7
2019, 17:28:10)[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linuxType
"help", "copyright", "credits" or "license" for more information.>>>
import psycopg2>>> import airflow[2019-12-17 11:07:32,167]
{settings.py:213} INFO - settings.configure_orm(): Using pool
settings. pool_size=5, max_overflow=10, pool_recycle=1800,
pid=73077Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py",
line 44, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py",
line 338, in initialize
configure_orm()
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py",
line 225, in configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py",
line 479, in create_engine
return strategy.create(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py",
line 61, in create
entrypoint = u._get_entrypoint()
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/url.py",
line 172, in _get_entrypoint
cls = registry.load(name)
File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
line 240, in load
"Can't load plugin: %s:%s" % (self.group, name)
sqlalchemy.exc.NoSuchModuleError: Can't load plugin:
sqlalchemy.dialects:postgresql.pyscopg2
>>> exit()
So it appears the module is in fact there and not quite sure what to make
of things at this point.
Anyone with more experience have any more debugging tips of know how to fix?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: airflow LocalExecutor initdb throws “sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2”
Posted by Kaxil Naik <ka...@gmail.com>.
Haha :D
On Tue, Dec 17, 2019 at 10:02 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Wow. Now I have a new name for you - "Kaxil Hawkeye". It took me a minute
> or so looking at it and I could not see the difference between the two...
>
> Human mind plays weird tricks :).
>
> J.
>
> On Tue, Dec 17, 2019 at 10:47 PM Reed Villanueva <rv...@ucera.org>
> wrote:
>
>> wow thanks
>>
>> On Tue, Dec 17, 2019 at 11:45 AM Kaxil Naik <ka...@gmail.com> wrote:
>>
>>> That is because you have a typo :) in your airflow.cfg file
>>>
>>> *Your have following:*
>>>
>>> sql_alchemy_conn = postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
>>>
>>>
>>> *Update it to:*
>>>
>>> sql_alchemy_conn = postgresql+psycopg2://airflow:mypassword@localhost:5432/airflow
>>>
>>>
>>> Note the spelling of *pyscopg2*
>>>
>>> Regards,
>>> Kaxil
>>>
>>>
>>> On Tue, Dec 17, 2019 at 9:37 PM Reed Villanueva <rv...@ucera.org>
>>> wrote:
>>>
>>>> Attempting to set up airflow (v1.10.5) in a centos7 machine to use
>>>> LocalExecutor following the article here (
>>>> https://www.ryanmerlin.com/2019/07/apache-airflow-installation-on-ubuntu-18-04-18-10/),
>>>> so have settings...
>>>>
>>>> [airflow@airflowetl airflow]$ rpm -q postgresql-server postgresql postgresql-devel
>>>> postgresql-server-9.2.24-1.el7_5.x86_64
>>>> postgresql-9.2.24-1.el7_5.x86_64
>>>> postgresql-devel-9.2.24-1.el7_5.x86_64
>>>>
>>>> [airflow@airflowetl airflow]$ pip3 freeze | grep sqlalchemy
>>>> marshmallow-sqlalchemy==0.19.0[airflow@airflowetl airflow]$ pip3 freeze | grep psycopg2
>>>> psycopg2==2.8.4
>>>>
>>>>
>>>> [airflow@airflowetl airflow]$ psql airflow
>>>> psql (9.2.24)Type "help" for help.
>>>>
>>>> airflow=> \l
>>>> List of databases
>>>> Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+-----------------------
>>>> airflow | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
>>>> | | | | | postgres=CTc/postgres+
>>>> | | | | | airflow=CTc/postgres
>>>> postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
>>>> template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
>>>> | | | | | postgres=CTc/postgres
>>>> template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
>>>> | | | | | postgres=CTc/postgres(4 rows)
>>>>
>>>>
>>>> [root@airflowetl airflow]# cat /var/lib/pgsql/data/pg_hba.conf....# TYPE DATABASE USER ADDRESS METHOD
>>>> # "local" is for Unix domain socket connections onlylocal all all peer# IPv4 local connections:#host all all 127.0.0.1/32 ident
>>>> host all all 0.0.0.0/0 trust# IPv6 local connections:
>>>> host all all ::1/128 ident# Allow replication connections from localhost, by a user with the# replication privilege.#local replication postgres peer#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident
>>>>
>>>>
>>>> [root@airflowetl airflow]# cat /var/lib/pgsql/data/postgresql.conf....# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # CONNECTIONS AND AUTHENTICATION# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # — Connection Settings -#listen_addresses = ‘localhost’ # what IP address(es) to listen on;
>>>> listen_addresses = ‘*’ # for Airflow connection
>>>>
>>>>
>>>> [airflow@airflowetl airflow]$ cat airflow.cfg....[core]....# The executor class that airflow should use. Choices include
>>>>
>>>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
Re: airflow LocalExecutor initdb throws “sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2”
Posted by Jarek Potiuk <Ja...@polidea.com>.
Wow. Now I have a new name for you - "Kaxil Hawkeye". It took me a minute
or so looking at it and I could not see the difference between the two...
Human mind plays weird tricks :).
J.
On Tue, Dec 17, 2019 at 10:47 PM Reed Villanueva <rv...@ucera.org>
wrote:
> wow thanks
>
> On Tue, Dec 17, 2019 at 11:45 AM Kaxil Naik <ka...@gmail.com> wrote:
>
>> That is because you have a typo :) in your airflow.cfg file
>>
>> *Your have following:*
>>
>> sql_alchemy_conn = postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
>>
>>
>> *Update it to:*
>>
>> sql_alchemy_conn = postgresql+psycopg2://airflow:mypassword@localhost:5432/airflow
>>
>>
>> Note the spelling of *pyscopg2*
>>
>> Regards,
>> Kaxil
>>
>>
>> On Tue, Dec 17, 2019 at 9:37 PM Reed Villanueva <rv...@ucera.org>
>> wrote:
>>
>>> Attempting to set up airflow (v1.10.5) in a centos7 machine to use
>>> LocalExecutor following the article here (
>>> https://www.ryanmerlin.com/2019/07/apache-airflow-installation-on-ubuntu-18-04-18-10/),
>>> so have settings...
>>>
>>> [airflow@airflowetl airflow]$ rpm -q postgresql-server postgresql postgresql-devel
>>> postgresql-server-9.2.24-1.el7_5.x86_64
>>> postgresql-9.2.24-1.el7_5.x86_64
>>> postgresql-devel-9.2.24-1.el7_5.x86_64
>>>
>>> [airflow@airflowetl airflow]$ pip3 freeze | grep sqlalchemy
>>> marshmallow-sqlalchemy==0.19.0[airflow@airflowetl airflow]$ pip3 freeze | grep psycopg2
>>> psycopg2==2.8.4
>>>
>>>
>>> [airflow@airflowetl airflow]$ psql airflow
>>> psql (9.2.24)Type "help" for help.
>>>
>>> airflow=> \l
>>> List of databases
>>> Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+-----------------------
>>> airflow | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
>>> | | | | | postgres=CTc/postgres+
>>> | | | | | airflow=CTc/postgres
>>> postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
>>> template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
>>> | | | | | postgres=CTc/postgres
>>> template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
>>> | | | | | postgres=CTc/postgres(4 rows)
>>>
>>>
>>> [root@airflowetl airflow]# cat /var/lib/pgsql/data/pg_hba.conf....# TYPE DATABASE USER ADDRESS METHOD
>>> # "local" is for Unix domain socket connections onlylocal all all peer# IPv4 local connections:#host all all 127.0.0.1/32 ident
>>> host all all 0.0.0.0/0 trust# IPv6 local connections:
>>> host all all ::1/128 ident# Allow replication connections from localhost, by a user with the# replication privilege.#local replication postgres peer#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident
>>>
>>>
>>> [root@airflowetl airflow]# cat /var/lib/pgsql/data/postgresql.conf....# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # CONNECTIONS AND AUTHENTICATION# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # — Connection Settings -#listen_addresses = ‘localhost’ # what IP address(es) to listen on;
>>> listen_addresses = ‘*’ # for Airflow connection
>>>
>>>
>>> [airflow@airflowetl airflow]$ cat airflow.cfg....[core]....# The executor class that airflow should use. Choices include
>>>
>>>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: airflow LocalExecutor initdb throws “sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2”
Posted by Reed Villanueva <rv...@ucera.org>.
wow thanks
On Tue, Dec 17, 2019 at 11:45 AM Kaxil Naik <ka...@gmail.com> wrote:
> That is because you have a typo :) in your airflow.cfg file
>
> *Your have following:*
>
> sql_alchemy_conn = postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
>
>
> *Update it to:*
>
> sql_alchemy_conn = postgresql+psycopg2://airflow:mypassword@localhost:5432/airflow
>
>
> Note the spelling of *pyscopg2*
>
> Regards,
> Kaxil
>
>
> On Tue, Dec 17, 2019 at 9:37 PM Reed Villanueva <rv...@ucera.org>
> wrote:
>
>> Attempting to set up airflow (v1.10.5) in a centos7 machine to use
>> LocalExecutor following the article here (
>> https://www.ryanmerlin.com/2019/07/apache-airflow-installation-on-ubuntu-18-04-18-10/),
>> so have settings...
>>
>> [airflow@airflowetl airflow]$ rpm -q postgresql-server postgresql postgresql-devel
>> postgresql-server-9.2.24-1.el7_5.x86_64
>> postgresql-9.2.24-1.el7_5.x86_64
>> postgresql-devel-9.2.24-1.el7_5.x86_64
>>
>> [airflow@airflowetl airflow]$ pip3 freeze | grep sqlalchemy
>> marshmallow-sqlalchemy==0.19.0[airflow@airflowetl airflow]$ pip3 freeze | grep psycopg2
>> psycopg2==2.8.4
>>
>>
>> [airflow@airflowetl airflow]$ psql airflow
>> psql (9.2.24)Type "help" for help.
>>
>> airflow=> \l
>> List of databases
>> Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+-----------------------
>> airflow | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
>> | | | | | postgres=CTc/postgres+
>> | | | | | airflow=CTc/postgres
>> postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
>> template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
>> | | | | | postgres=CTc/postgres
>> template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
>> | | | | | postgres=CTc/postgres(4 rows)
>>
>>
>> [root@airflowetl airflow]# cat /var/lib/pgsql/data/pg_hba.conf....# TYPE DATABASE USER ADDRESS METHOD
>> # "local" is for Unix domain socket connections onlylocal all all peer# IPv4 local connections:#host all all 127.0.0.1/32 ident
>> host all all 0.0.0.0/0 trust# IPv6 local connections:
>> host all all ::1/128 ident# Allow replication connections from localhost, by a user with the# replication privilege.#local replication postgres peer#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident
>>
>>
>> [root@airflowetl airflow]# cat /var/lib/pgsql/data/postgresql.conf....# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # CONNECTIONS AND AUTHENTICATION# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # — Connection Settings -#listen_addresses = ‘localhost’ # what IP address(es) to listen on;
>> listen_addresses = ‘*’ # for Airflow connection
>>
>>
>> [airflow@airflowetl airflow]$ cat airflow.cfg....[core]....# The executor class that airflow should use. Choices include# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor, KubernetesExecutor#executor = SequentialExecutor
>> executor = LocalExecutor
>> # The SqlAlchemy connection string to the metadata database.# SqlAlchemy supports many different database engine, more information# their website#sql_alchemy_conn = sqlite:////home/airflow/airflow/airflow.db
>> sql_alchemy_conn = postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
>>
>> Yet am seeing
>>
>> File
>> "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
>> line 240, in load "Can't load plugin: %s:%s" % (self.group, name)
>> sqlalchemy.exc.NoSuchModuleError: Can't load plugin:
>> sqlalchemy.dialects:postgresql.pyscopg2
>>
>> error when trying to run airflow initdb. Checking if psycopg2 is in fact
>> installed, seeing...
>>
>> [airflow@airflowetl airflow]$ python3Python 3.6.8 (default, Aug 7 2019, 17:28:10)[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import psycopg2>>> import airflow[2019-12-17 11:07:32,167] {settings.py:213} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=73077Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 44, in <module>
>> settings.initialize()
>> File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 338, in initialize
>> configure_orm()
>> File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 225, in configure_orm
>> engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
>> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 479, in create_engine
>> return strategy.create(*args, **kwargs)
>> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 61, in create
>> entrypoint = u._get_entrypoint()
>> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
>> cls = registry.load(name)
>> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 240, in load
>> "Can't load plugin: %s:%s" % (self.group, name)
>> sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2
>> >>> exit()
>>
>> So it appears the module is in fact there and not quite sure what to make
>> of things at this point.
>>
>> Anyone with more experience have any more debugging tips of know how to
>> fix?
>>
>> This electronic message is intended only for the named
>> recipient, and may contain information that is confidential or
>> privileged. If you are not the intended recipient, you are
>> hereby notified that any disclosure, copying, distribution or
>> use of the contents of this message is strictly prohibited. If
>> you have received this message in error or are not the named
>> recipient, please notify us immediately by contacting the
>> sender at the electronic mail address noted above, and delete
>> and destroy all copies of this message. Thank you.
>>
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: airflow LocalExecutor initdb throws “sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2”
Posted by Kaxil Naik <ka...@gmail.com>.
That is because you have a typo :) in your airflow.cfg file
*Your have following:*
sql_alchemy_conn =
postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
*Update it to:*
sql_alchemy_conn =
postgresql+psycopg2://airflow:mypassword@localhost:5432/airflow
Note the spelling of *pyscopg2*
Regards,
Kaxil
On Tue, Dec 17, 2019 at 9:37 PM Reed Villanueva <rv...@ucera.org>
wrote:
> Attempting to set up airflow (v1.10.5) in a centos7 machine to use
> LocalExecutor following the article here (
> https://www.ryanmerlin.com/2019/07/apache-airflow-installation-on-ubuntu-18-04-18-10/),
> so have settings...
>
> [airflow@airflowetl airflow]$ rpm -q postgresql-server postgresql postgresql-devel
> postgresql-server-9.2.24-1.el7_5.x86_64
> postgresql-9.2.24-1.el7_5.x86_64
> postgresql-devel-9.2.24-1.el7_5.x86_64
>
> [airflow@airflowetl airflow]$ pip3 freeze | grep sqlalchemy
> marshmallow-sqlalchemy==0.19.0[airflow@airflowetl airflow]$ pip3 freeze | grep psycopg2
> psycopg2==2.8.4
>
>
> [airflow@airflowetl airflow]$ psql airflow
> psql (9.2.24)Type "help" for help.
>
> airflow=> \l
> List of databases
> Name | Owner | Encoding | Collate | Ctype | Access privileges-----------+----------+----------+-------------+-------------+-----------------------
> airflow | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres +
> | | | | | postgres=CTc/postgres+
> | | | | | airflow=CTc/postgres
> postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
> template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
> | | | | | postgres=CTc/postgres
> template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres +
> | | | | | postgres=CTc/postgres(4 rows)
>
>
> [root@airflowetl airflow]# cat /var/lib/pgsql/data/pg_hba.conf....# TYPE DATABASE USER ADDRESS METHOD
> # "local" is for Unix domain socket connections onlylocal all all peer# IPv4 local connections:#host all all 127.0.0.1/32 ident
> host all all 0.0.0.0/0 trust# IPv6 local connections:
> host all all ::1/128 ident# Allow replication connections from localhost, by a user with the# replication privilege.#local replication postgres peer#host replication postgres 127.0.0.1/32 ident#host replication postgres ::1/128 ident
>
>
> [root@airflowetl airflow]# cat /var/lib/pgsql/data/postgresql.conf....# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # CONNECTIONS AND AUTHENTICATION# — — — — — — — — — — — — — — — — — — — — — — — — — — — — — # — Connection Settings -#listen_addresses = ‘localhost’ # what IP address(es) to listen on;
> listen_addresses = ‘*’ # for Airflow connection
>
>
> [airflow@airflowetl airflow]$ cat airflow.cfg....[core]....# The executor class that airflow should use. Choices include# SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor, KubernetesExecutor#executor = SequentialExecutor
> executor = LocalExecutor
> # The SqlAlchemy connection string to the metadata database.# SqlAlchemy supports many different database engine, more information# their website#sql_alchemy_conn = sqlite:////home/airflow/airflow/airflow.db
> sql_alchemy_conn = postgresql+pyscopg2://airflow:mypassword@localhost:5432/airflow
>
> Yet am seeing
>
> File
> "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py",
> line 240, in load "Can't load plugin: %s:%s" % (self.group, name)
> sqlalchemy.exc.NoSuchModuleError: Can't load plugin:
> sqlalchemy.dialects:postgresql.pyscopg2
>
> error when trying to run airflow initdb. Checking if psycopg2 is in fact
> installed, seeing...
>
> [airflow@airflowetl airflow]$ python3Python 3.6.8 (default, Aug 7 2019, 17:28:10)[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import psycopg2>>> import airflow[2019-12-17 11:07:32,167] {settings.py:213} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=73077Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__init__.py", line 44, in <module>
> settings.initialize()
> File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 338, in initialize
> configure_orm()
> File "/home/airflow/.local/lib/python3.6/site-packages/airflow/settings.py", line 225, in configure_orm
> engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py", line 479, in create_engine
> return strategy.create(*args, **kwargs)
> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 61, in create
> entrypoint = u._get_entrypoint()
> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/engine/url.py", line 172, in _get_entrypoint
> cls = registry.load(name)
> File "/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 240, in load
> "Can't load plugin: %s:%s" % (self.group, name)
> sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:postgresql.pyscopg2
> >>> exit()
>
> So it appears the module is in fact there and not quite sure what to make
> of things at this point.
>
> Anyone with more experience have any more debugging tips of know how to
> fix?
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.
>