You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/08 09:35:13 UTC
[GitHub] [airflow] aneagoe opened a new issue #14659: MongoHook fails to parse MongoDB replica set URIs
aneagoe opened a new issue #14659:
URL: https://github.com/apache/airflow/issues/14659
**Apache Airflow version**: 1.10.12
**Kubernetes version: 1.16
**What happened**:
As per https://issues.apache.org/jira/browse/AIRFLOW-5164, when using mongodb replica set URIs (of form mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl) then an error is thrown as airflow tries to cram everything after `:` as port. (ie `ValueError: invalid literal for int() with base 10: '27017,mongodb1.example.com:27017,mongodb2.example.com:27017`).
**What you expected to happen**:
Airflow should properly interpret the replicaset mongodb URI formats.
**How to reproduce it**:
```
import os
os.environ["AIRFLOW_CONN_MONGO_STANDALONE"] = "mongodb://mongodb0.example.com:27017/admin"
os.environ["AIRFLOW_CONN_MONGO_REPLICASET"] = "mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl"
from airflow.contrib.hooks.mongo_hook import MongoHook
hook = MongoHook('mongo_standalone')
print(hook)
hook = MongoHook('mongo_replicaset')
print(hook)
```
Example:
```
>>> import os
>>> os.environ["AIRFLOW_CONN_MONGO_STANDALONE"] = "mongodb://mongodb0.example.com:27017/admin"
>>> os.environ["AIRFLOW_CONN_MONGO_REPLICASET"] = "mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/admin?replicaSet=myRepl"
>>> from airflow.contrib.hooks.mongo_hook import MongoHook
>>> hook = MongoHook('mongo_standalone')
[2021-03-08 09:34:06,166] {base_hook.py:89} INFO - Using connection to: id: mongo_standalone. Host: mongodb0.example.com, Port: 27017, Schema: admin, Login: None, Password: None, extra: None
>>> print(hook)
<airflow.contrib.hooks.mongo_hook.MongoHook object at 0x7f8dccc0aa50>
>>> hook = MongoHook('mongo_replicaset')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/contrib/hooks/mongo_hook.py", line 44, in __init__
self.connection = self.get_connection(conn_id)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/hooks/base_hook.py", line 87, in get_connection
conn = random.choice(list(cls.get_connections(conn_id)))
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/hooks/base_hook.py", line 83, in get_connections
return secrets.get_connections(conn_id)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/secrets/__init__.py", line 55, in get_connections
conn_list = secrets_backend.get_connections(conn_id=conn_id)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/secrets/base_secrets.py", line 67, in get_connections
conn = Connection(conn_id=conn_id, uri=conn_uri)
File "<string>", line 4, in __init__
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 433, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 430, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 121, in __init__
self.parse_from_uri(uri)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/connection.py", line 146, in parse_from_uri
self.port = uri_parts.port
File "/usr/local/lib/python3.7/urllib/parse.py", line 169, in port
port = int(port, 10)
ValueError: invalid literal for int() with base 10: '27017,mongodb1.example.com:27017,mongodb2.example.com:27017'
>>> print(hook)
<airflow.contrib.hooks.mongo_hook.MongoHook object at 0x7f8dccc0aa50>
>>>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Junnplus commented on issue #14659: MongoHook fails to parse MongoDB replica set URIs
Posted by GitBox <gi...@apache.org>.
Junnplus commented on issue #14659:
URL: https://github.com/apache/airflow/issues/14659#issuecomment-793426242
It seems that mongodb replica set URIs format are not supported for airflow, but you can use DNS seed list connection format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14659: MongoHook fails to parse MongoDB replica set URIs
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14659:
URL: https://github.com/apache/airflow/issues/14659#issuecomment-792621098
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org