You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2024/02/07 05:15:34 UTC
[PR] Adding certificate validation for ssl in mongo hook [airflow]
amoghrajesh opened a new pull request, #37214:
URL: https://github.com/apache/airflow/pull/37214
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Currently, we are skipping tls certificate validation in mongo hook even if ssl is set to true. This needs to be handled better
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480978086
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
# If we are using SSL disable requiring certs from specific hostname
if options.get("ssl", False):
+ allow_insecure = self.extras.get("allow_insecure", True)
if pymongo.__version__ >= "4.0.0":
- # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
- # replaces `ssl_cert_reqs=CERT_NONE`
- options.update({"tlsAllowInvalidCertificates": True})
+ # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
Review Comment:
Yep makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931574781
> > Also tried adding mongo://127.0.0.1, this doesnt work either for uri
>
> I've wondering, did you run Airflow into the docker?
I am running it using breeze, so yes, you might be right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482504765
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+ self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
+
+ if self.ssl_enabled and not self.allow_insecure:
+ # Case: HTTPS
+ self.allow_insecure = False
+ elif self.ssl_enabled and self.allow_insecure:
+ # Case: HTTPS + allow_insecure
+ self.allow_insecure = True
+ self.extras.pop("ssl", None)
+ elif not self.ssl_enabled and "allow_insecure" in self.extras:
+ # Case: HTTP (ssl=False) with allow_insecure specified
Review Comment:
NIT: we can add warning here, since "allow_insecure" + "ssl = False" does not make sense (i.e. "allow_insecure" only makes sense in "ssl" context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482507709
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+ self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
+
+ if self.ssl_enabled and not self.allow_insecure:
+ # Case: HTTPS
+ self.allow_insecure = False
+ elif self.ssl_enabled and self.allow_insecure:
+ # Case: HTTPS + allow_insecure
+ self.allow_insecure = True
+ self.extras.pop("ssl", None)
+ elif not self.ssl_enabled and "allow_insecure" in self.extras:
+ # Case: HTTP (ssl=False) with allow_insecure specified
Review Comment:
(and possibly we should explain it in docstring of the Hook).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931540047
> Also tried adding mongo://127.0.0.1, this doesnt work either for uri
I've wondering, did you run Airflow into the docker?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481199290
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
+ if self.extras.get("ssl", False):
Review Comment:
Should I integrate it here or take it in a followup? Addressing a vulnerability here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482505496
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+ self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
Review Comment:
NIT: As @Taragolis suggested:
```suggestion
self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true" or self.extras.get("tls", "false").lower() == "true"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1932485357
Sure, thanks for your review.
I will take a look at this soon, mostly tomorrow ist morning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481653314
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
Review Comment:
I believe `allow_insecure` only applies when `ssl` is used (plain http is always insecure). But in fact there were 2 states here:
* http (ssl = False)
* htps + allow_insecure (ssl = True)
What we need is 3 states
* http (ssl = False)
* htps + allow_insecure (ssl = True, allow_insecure=True)
* https - (ssl = True, allow_insecure = False) this should be default when ssl
So we need separate `allow_insecure` flag with default = False
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481199290
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
+ if self.extras.get("ssl", False):
Review Comment:
Should I integrate it here or take it in a followup? Addressing a vulnerability here, so planning to keep the solution scoped only to solve the vulnerability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1932296310
I think having "alllow_insecure" while setting "ssl=True" is precisely the vulnerability here, because it is unexpected. So i think when user sets "ssl=True" then allow_insecure should be false by default.
And it's ok to make breaking change while fixing security bug. We want people to find out they have security issue and make them fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482985907
##########
airflow/providers/mongo/CHANGELOG.rst:
##########
@@ -27,6 +27,17 @@
Changelog
---------
+3.7.0
+......
Review Comment:
Ah .. Right 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1483150140
##########
airflow/providers/mongo/CHANGELOG.rst:
##########
@@ -27,6 +27,17 @@
Changelog
---------
+3.7.0
+......
Review Comment:
I will raise a follow up for this. Sorry missed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480909864
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
# If we are using SSL disable requiring certs from specific hostname
if options.get("ssl", False):
+ allow_insecure = self.extras.get("allow_insecure", True)
if pymongo.__version__ >= "4.0.0":
- # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
- # replaces `ssl_cert_reqs=CERT_NONE`
- options.update({"tlsAllowInvalidCertificates": True})
+ # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
+ options["tlsAllowInvalidCertificates"] = allow_insecure
else:
- options.update({"ssl_cert_reqs": CERT_NONE})
+ # For older pymongo versions, use 'ssl_cert_reqs'
+ options["ssl_cert_reqs"] = None if allow_insecure else CERT_NONE
Review Comment:
Okay realised that CERT_REQUIRED is present
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
# If we are using SSL disable requiring certs from specific hostname
if options.get("ssl", False):
+ allow_insecure = self.extras.get("allow_insecure", True)
if pymongo.__version__ >= "4.0.0":
- # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
- # replaces `ssl_cert_reqs=CERT_NONE`
- options.update({"tlsAllowInvalidCertificates": True})
+ # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
+ options["tlsAllowInvalidCertificates"] = allow_insecure
else:
- options.update({"ssl_cert_reqs": CERT_NONE})
+ # For older pymongo versions, use 'ssl_cert_reqs'
+ options["ssl_cert_reqs"] = None if allow_insecure else CERT_NONE
Review Comment:
Okay realised that CERT_REQUIRED is present and provided it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481643804
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
Review Comment:
Didn't completely get you here. The user can always come back to old behaviour by setting the ssl as False, right or not specifying it at all.
The old behaviour was that irrespective of ssl, we skipped tls validation. Should we retain that behaviour?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482980113
##########
airflow/providers/mongo/CHANGELOG.rst:
##########
@@ -27,6 +27,17 @@
Changelog
---------
+3.7.0
+......
Review Comment:
Post merge comment:
I guess the version should be 4.0.0 in case of breaking changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481130143
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
+ if self.extras.get("ssl", False):
Review Comment:
According to the [`MongoClient`](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient) `ssl` is alias to the `tls`, potentially end users might use `tls` instead of `ssl`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1933671125
> Really Nice. Only docs/spellcheck fix and you can merge it.
Just ran it locally and pushed a fix, we should be good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37214:
URL: https://github.com/apache/airflow/pull/37214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931432371
Also tried adding mongo://127.0.0.1, this doesnt work either for uri
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482586148
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+ self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
+
+ if self.ssl_enabled and not self.allow_insecure:
+ # Case: HTTPS
+ self.allow_insecure = False
+ elif self.ssl_enabled and self.allow_insecure:
+ # Case: HTTPS + allow_insecure
+ self.allow_insecure = True
+ self.extras.pop("ssl", None)
+ elif not self.ssl_enabled and "allow_insecure" in self.extras:
+ # Case: HTTP (ssl=False) with allow_insecure specified
Review Comment:
Yeah makes sense. Making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931637366
Thanks @Taragolis !
That tip saved hassle for me, ran it via docker compose and it worked :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931678760
> That tip saved hassle for me, ran it via docker compose and it worked :D
Yeah in breeze only available simple integration, which need to be run with `--integration mongo`, e.g.
```console
❯ breeze start-airflow --backend postgres --postgres-version 15 --integration mongo
```
After that you could communicate with Mongo 3 thought the host `mongo`
```python
from airflow import DAG
from airflow.decorators import task
from airflow.providers.mongo.hooks.mongo import MongoHook
with DAG("pr_37214", schedule=None, tags=["37214", "mongodb"]):
@task
def test_mongo():
# Create a connection into the runtime, might be replaced by connection created into the UI
import os
from airflow.models.connection import Connection
mongo_conn_id = "mongo_test"
conn = Connection(conn_id=mongo_conn_id, conn_type="mongo", host="mongo", port=27017, schema="test")
os.environ[f"AIRFLOW_CONN_{conn.conn_id.upper()}"] = conn.as_json()
# Actual execution
mongo_hook = MongoHook(mongo_conn_id=mongo_conn_id)
mongo_hook.insert_one("users", {"name": "John", "surname": "Wick"}, mongo_db="test")
collection = mongo_hook.get_collection("users", mongo_db="test")
print(collection.count_documents({}))
test_mongo()
```
And it is expected that if you try to communicate in breeze with `localhost` or `127.0.0.1` that is not exists, because there is no mongo in Airflow Breeze container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931686673
> > That tip saved hassle for me, ran it via docker compose and it worked :D
>
> Yeah in breeze only available simple integration, which need to be run with `--integration mongo`, e.g.
>
> ```
> ❯ breeze start-airflow --backend postgres --postgres-version 15 --integration mongo
> ```
>
> After that you could communicate with Mongo 3 thought the host `mongo`
>
> ```python
> from airflow import DAG
> from airflow.decorators import task
> from airflow.providers.mongo.hooks.mongo import MongoHook
>
>
> with DAG("pr_37214", schedule=None, tags=["37214", "mongodb"]):
> @task
> def test_mongo():
> # Create a connection into the runtime, might be replaced by connection created into the UI
> import os
> from airflow.models.connection import Connection
>
> mongo_conn_id = "mongo_test"
> conn = Connection(conn_id=mongo_conn_id, conn_type="mongo", host="mongo", port=27017, schema="test")
> os.environ[f"AIRFLOW_CONN_{conn.conn_id.upper()}"] = conn.as_json()
>
> # Actual execution
> mongo_hook = MongoHook(mongo_conn_id=mongo_conn_id)
> mongo_hook.insert_one("users", {"name": "John", "surname": "Wick"}, mongo_db="test")
>
> collection = mongo_hook.get_collection("users", mongo_db="test")
> print(collection.count_documents({}))
>
> test_mongo()
> ```
>
> And it is expected that if you try to communicate in breeze with `localhost` or `127.0.0.1` that is not exists, because there is no mongo in Airflow Breeze container
Perfect! I wasn't aware of this, thank you!
This is good :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481609101
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
Review Comment:
I think allow_insecure should be a separate extra, to alllow the user to come back to old behaviour. BTW. This is a breaking change so it neds to be explained in CHANGELOG for the provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480908546
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
# If we are using SSL disable requiring certs from specific hostname
if options.get("ssl", False):
+ allow_insecure = self.extras.get("allow_insecure", True)
if pymongo.__version__ >= "4.0.0":
- # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
- # replaces `ssl_cert_reqs=CERT_NONE`
- options.update({"tlsAllowInvalidCertificates": True})
+ # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
+ options["tlsAllowInvalidCertificates"] = allow_insecure
else:
- options.update({"ssl_cert_reqs": CERT_NONE})
+ # For older pymongo versions, use 'ssl_cert_reqs'
+ options["ssl_cert_reqs"] = None if allow_insecure else CERT_NONE
Review Comment:
Is there any field in the connection that gets the certs from the hook? Ideally that should be provided here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931415839
Right now I am testing with the DAG:
```
from airflow import DAG
from airflow.providers.mongo.hooks.mongo import MongoHook
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime(2024, 2, 7),
'catchup': False,
}
def retrieve_users_from_mongodb():
mongo_hook = MongoHook(conn_id='mongo_conn')
query = {}
users_data = mongo_hook.find('users', query)
for user in users_data:
print(user)
with DAG('mongo_dag', default_args=default_args, schedule_interval=None) as dag:
# Task to retrieve users from MongoDB
retrieve_users_task = PythonOperator(
task_id='retrieve_users_from_mongodb',
python_callable=retrieve_users_from_mongodb
)
# Define the task dependencies
retrieve_users_task
```
Connection:

For some reason, the connection is not going through, to mongodb
Error log:
```
e8786b12ac04
*** Found local files:
*** * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T07:07:35.163848+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log
[2024-02-07, 12:37:37 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti=
[2024-02-07, 12:37:37 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti=
[2024-02-07, 12:37:37 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1
[2024-02-07, 12:37:37 IST] {taskinstance.py:2215} INFO - Executing on 2024-02-07 07:07:35.163848+00:00
[2024-02-07, 12:37:37 IST] {standard_task_runner.py:60} INFO - Started process 1243 to run task
[2024-02-07, 12:37:37 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T07:07:35.163848+00:00', '--job-id', '920', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmpkg6apbfb']
[2024-02-07, 12:37:37 IST] {standard_task_runner.py:88} INFO - Job 920: Subtask retrieve_users_from_mongodb
[2024-02-07, 12:37:37 IST] {task_command.py:423} INFO - Running on host e8786b12ac04
[2024-02-07, 12:37:37 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='airflow@example.com' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T07:07:35.163848+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T07:07:35.163848+00:00'
[2024-02-07, 12:37:37 IST] {base.py:83} INFO - Using connection ID 'myconn' for task execution.
[2024-02-07, 12:38:08 IST] {taskinstance.py:2737} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/airflow/models/taskinstance.py", line 446, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/models/taskinstance.py", line 416, in _execute_callable
return execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/operators/python.py", line 199, in execute
return_value = self.execute_callable()
File "/opt/airflow/airflow/operators/python.py", line 216, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/files/dags/mongo_test.py", line 28, in retrieve_users_from_mongodb
for user in users_data:
File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1264, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1823, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 573, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 589, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 259, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>
[2024-02-07, 12:38:08 IST] {taskinstance.py:1152} INFO - Marking task as FAILED. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T070735, start_date=20240207T070737, end_date=20240207T070808
[2024-02-07, 12:38:08 IST] {standard_task_runner.py:107} ERROR - Failed to execute job 920 for task retrieve_users_from_mongodb (127.0.0.1:27017: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>; 1243)
[2024-02-07, 12:38:08 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 1
[2024-02-07, 12:38:08 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
Point to note is that when I use pymongo directly, it works
```
from pymongo import MongoClient
mongo_host = "127.0.0.1"
mongo_port = 27017
mongo_database = "test"
mongo_collection = "users"
client = MongoClient(mongo_host, mongo_port)
db = client[mongo_database]
collection = db[mongo_collection]
query_result = collection.find_one({"name": "John"})
print("Query result:", query_result)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931650831
Awesome. I was able to test it.
Results:
My connections defined:
<img width="867" alt="image" src="https://github.com/apache/airflow/assets/35884252/06a85bbd-579b-436b-9079-ec27499120c2">
Logs for my_mong (ssl is not present):
```
c47a4cad3064
*** Found local files:
*** * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T09:32:50.030512+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log
[2024-02-07, 15:02:53 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti=
[2024-02-07, 15:02:53 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti=
[2024-02-07, 15:02:53 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1
[2024-02-07, 15:02:53 IST] {taskinstance.py:2215} INFO - Executing on 2024-02-07 09:32:50.030512+00:00
[2024-02-07, 15:02:53 IST] {standard_task_runner.py:60} INFO - Started process 1073 to run task
[2024-02-07, 15:02:53 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T09:32:50.030512+00:00', '--job-id', '929', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmp1dqi4wjk']
[2024-02-07, 15:02:53 IST] {standard_task_runner.py:88} INFO - Job 929: Subtask retrieve_users_from_mongodb
[2024-02-07, 15:02:53 IST] {task_command.py:423} INFO - Running on host c47a4cad3064
[2024-02-07, 15:02:53 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='airflow@example.com' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T09:32:50.030512+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T09:32:50.030512+00:00'
[2024-02-07, 15:02:53 IST] {base.py:83} INFO - Using connection ID 'my_mong' for task execution.
[2024-02-07, 15:02:53 IST] {logging_mixin.py:188} INFO - {'_id': ObjectId('65c34e0e7e0c274d333d8922'), 'username': 'user1', 'email': 'user1@example.com'}
[2024-02-07, 15:02:53 IST] {logging_mixin.py:188} INFO - {'_id': ObjectId('65c34e0e7e0c274d333d8923'), 'username': 'user2', 'email': 'user2@example.com'}
[2024-02-07, 15:02:53 IST] {python.py:201} INFO - Done. Returned value was: None
[2024-02-07, 15:02:53 IST] {taskinstance.py:1152} INFO - Marking task as SUCCESS. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T093250, start_date=20240207T093253, end_date=20240207T093253
[2024-02-07, 15:02:53 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 0
[2024-02-07, 15:02:53 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
Logs for my_mong_secure (explicitly passed my ssl: true)
```
c47a4cad3064
*** Found local files:
*** * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T09:36:02.449095+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log
[2024-02-07, 15:06:05 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti=
[2024-02-07, 15:06:05 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti=
[2024-02-07, 15:06:05 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1
[2024-02-07, 15:06:05 IST] {taskinstance.py:2215} INFO - Executing on 2024-02-07 09:36:02.449095+00:00
[2024-02-07, 15:06:05 IST] {standard_task_runner.py:60} INFO - Started process 1397 to run task
[2024-02-07, 15:06:05 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T09:36:02.449095+00:00', '--job-id', '930', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmpn18c8wnm']
[2024-02-07, 15:06:05 IST] {standard_task_runner.py:88} INFO - Job 930: Subtask retrieve_users_from_mongodb
[2024-02-07, 15:06:05 IST] {task_command.py:423} INFO - Running on host c47a4cad3064
[2024-02-07, 15:06:05 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='airflow@example.com' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T09:36:02.449095+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T09:36:02.449095+00:00'
[2024-02-07, 15:06:05 IST] {base.py:83} INFO - Using connection ID 'my_mong_secure' for task execution.
[2024-02-07, 15:06:35 IST] {taskinstance.py:2737} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/airflow/airflow/models/taskinstance.py", line 446, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/models/taskinstance.py", line 416, in _execute_callable
return execute_callable(context=context, **execute_callable_kwargs)
File "/opt/airflow/airflow/operators/python.py", line 199, in execute
return_value = self.execute_callable()
File "/opt/airflow/airflow/operators/python.py", line 216, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/files/dags/mongo_test.py", line 28, in retrieve_users_from_mongodb
for user in users_data:
File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1264, in next
if len(self.__data) or self._refresh():
File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1823, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
self._topology._check_implicit_session_support()
File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 573, in _check_implicit_session_support
self._check_session_support()
File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 589, in _check_session_support
self._select_servers_loop(
File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 259, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: SSL handshake failed: 172.20.0.3:27017: TLS/SSL connection has been closed (EOF) (_ssl.c:1131) (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>
[2024-02-07, 15:06:35 IST] {taskinstance.py:1152} INFO - Marking task as FAILED. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T093602, start_date=20240207T093605, end_date=20240207T093635
[2024-02-07, 15:06:35 IST] {standard_task_runner.py:107} ERROR - Failed to execute job 930 for task retrieve_users_from_mongodb (SSL handshake failed: 172.20.0.3:27017: TLS/SSL connection has been closed (EOF) (_ssl.c:1131) (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>; 1397)
[2024-02-07, 15:06:35 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 1
[2024-02-07, 15:06:35 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check
```
cc @Taragolis @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480958341
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
# If we are using SSL disable requiring certs from specific hostname
if options.get("ssl", False):
+ allow_insecure = self.extras.get("allow_insecure", True)
if pymongo.__version__ >= "4.0.0":
- # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
- # replaces `ssl_cert_reqs=CERT_NONE`
- options.update({"tlsAllowInvalidCertificates": True})
+ # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
Review Comment:
Pymongo 4.0 was released in 2021 maybe we should just support >=4.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1933376396
@potiuk this has been tested out with the following combinations:
```
{
"mongo_http": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": "{\"ssl\": \"false\"}"
},
"mongo_http_copy1": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": "{\"ssl\": \"false\"}"
},
"mongo_https_allow_insecure": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": "{\"ssl\": \"true\", \"allow_insecure\": \"true\"}"
},
"mongo_https_dont_allow_insecure": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": "{\"ssl\": \"true\", \"allow_insecure\": \"false\"}"
},
"old_no_ssl_set": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": ""
},
"old_ssl_false": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": "{\"ssl\": \"false\"}"
},
"old_ssl_true": {
"conn_type": "mongo",
"description": "",
"login": "",
"password": null,
"host": "172.20.0.3",
"port": 27017,
"schema": "test",
"extra": "{\"ssl\": \"true\"}"
}
}
```
It works as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482507876
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+ self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
Review Comment:
(and also add docstring).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1932289134
@potiuk to maintain the older behaviour, I think one thing we can do is:
in init:
```
if "ssl" is in self.extra and if "allow_insecure" is not in self.extras:
self.extras["allow_insecure"] = False
```
and in get_conn:
```
allow_insecure = get allow_insecure from options
if options.get("ssl", False):
if pymongo.__version__ >= "4.0.0":
# For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
options["tlsAllowInvalidCertificates"] = allow_insecure
else:
# For older pymongo versions, use 'ssl_cert_reqs'
options["ssl_cert_reqs"] = CERT_NONE if allow_insecure else CERT_REQUIRED
```
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482592429
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+ self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
Review Comment:
Sure, making the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481653314
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
Review Comment:
I believe `allow_insecure` only applies when `ssl` is used (plain http is always insecure). But in fact there were 2 states here:
* http (ssl = False)
* htps + allow_insecure (ssl = True)
What we need is 3 states
* http (ssl = False)
* htps + allow_insecure (ssl = True, allow_insecure=True)
* https - (ssl = True, allow_insecure = False) this should be default when ssl
So we need separate `allow_insecure` flag with default = False (and this is breaking change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481650748
##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
self.client: MongoClient | None = None
self.uri = self._create_uri()
+ self.allow_insecure = True
Review Comment:
Do you mean to just add a new allow_insecure in extra and keep ssl as-is? Would that add any value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org