You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2024/02/07 05:15:34 UTC

[PR] Adding certificate validation for ssl in mongo hook [airflow]

amoghrajesh opened a new pull request, #37214:
URL: https://github.com/apache/airflow/pull/37214

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   Currently, we are skipping tls certificate validation in mongo hook even if ssl is set to true. This needs to be handled better
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480978086


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
 
         # If we are using SSL disable requiring certs from specific hostname
         if options.get("ssl", False):
+            allow_insecure = self.extras.get("allow_insecure", True)
             if pymongo.__version__ >= "4.0.0":
-                # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
-                # replaces `ssl_cert_reqs=CERT_NONE`
-                options.update({"tlsAllowInvalidCertificates": True})
+                # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'

Review Comment:
   Yep makes sense



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931574781

   > > Also tried adding mongo://127.0.0.1, this doesnt work either for uri
   > 
   > I've wondering, did you run Airflow into the docker?
   
   I am running it using breeze, so yes, you might be right


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482504765


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+        self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
+
+        if self.ssl_enabled and not self.allow_insecure:
+            # Case: HTTPS
+            self.allow_insecure = False
+        elif self.ssl_enabled and self.allow_insecure:
+            # Case: HTTPS + allow_insecure
+            self.allow_insecure = True
+            self.extras.pop("ssl", None)
+        elif not self.ssl_enabled and "allow_insecure" in self.extras:
+            # Case: HTTP (ssl=False) with allow_insecure specified

Review Comment:
   NIT: we can add  warning here, since "allow_insecure" + "ssl = False" does not make sense (i.e. "allow_insecure" only makes sense in "ssl" context.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482507709


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+        self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
+
+        if self.ssl_enabled and not self.allow_insecure:
+            # Case: HTTPS
+            self.allow_insecure = False
+        elif self.ssl_enabled and self.allow_insecure:
+            # Case: HTTPS + allow_insecure
+            self.allow_insecure = True
+            self.extras.pop("ssl", None)
+        elif not self.ssl_enabled and "allow_insecure" in self.extras:
+            # Case: HTTP (ssl=False) with allow_insecure specified

Review Comment:
   (and possibly we should explain it in docstring of the Hook).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931540047

   > Also tried adding mongo://127.0.0.1, this doesnt work either for uri
   
   I've wondering, did you run Airflow into the docker?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481199290


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True
+        if self.extras.get("ssl", False):

Review Comment:
   Should I integrate it here or take it in a followup? Addressing a vulnerability here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482505496


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+        self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"

Review Comment:
   NIT: As @Taragolis suggested:
   ```suggestion
           self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true" or self.extras.get("tls", "false").lower() == "true" 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1932485357

   Sure, thanks for your review.
   
   I will take a look at this soon, mostly tomorrow ist morning


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481653314


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True

Review Comment:
   I believe `allow_insecure` only applies when `ssl` is used (plain http is always insecure). But in fact there were 2 states here:
   
   * http (ssl = False)
   * htps + allow_insecure (ssl = True)
   
   What we need is 3 states
   
   * http (ssl = False)
   * htps + allow_insecure (ssl = True, allow_insecure=True)
   * https  - (ssl = True, allow_insecure = False) this should be default when ssl
   
   So we need separate `allow_insecure` flag with default = False 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481199290


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True
+        if self.extras.get("ssl", False):

Review Comment:
   Should I integrate it here or take it in a followup? Addressing a vulnerability here, so planning to keep the solution scoped only to solve the vulnerability 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1932296310

   I think having "alllow_insecure" while setting "ssl=True"  is precisely the vulnerability here, because it is unexpected. So i think when user sets "ssl=True" then allow_insecure should be false by default. 
   
   And it's ok to make breaking change while fixing security bug. We want people to find out they have security issue and make them fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482985907


##########
airflow/providers/mongo/CHANGELOG.rst:
##########
@@ -27,6 +27,17 @@
 Changelog
 ---------
 
+3.7.0
+......

Review Comment:
   Ah .. Right 🤦 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1483150140


##########
airflow/providers/mongo/CHANGELOG.rst:
##########
@@ -27,6 +27,17 @@
 Changelog
 ---------
 
+3.7.0
+......

Review Comment:
   I will raise a follow up for this. Sorry missed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480909864


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
 
         # If we are using SSL disable requiring certs from specific hostname
         if options.get("ssl", False):
+            allow_insecure = self.extras.get("allow_insecure", True)
             if pymongo.__version__ >= "4.0.0":
-                # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
-                # replaces `ssl_cert_reqs=CERT_NONE`
-                options.update({"tlsAllowInvalidCertificates": True})
+                # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
+                options["tlsAllowInvalidCertificates"] = allow_insecure
             else:
-                options.update({"ssl_cert_reqs": CERT_NONE})
+                # For older pymongo versions, use 'ssl_cert_reqs'
+                options["ssl_cert_reqs"] = None if allow_insecure else CERT_NONE

Review Comment:
   Okay realised that CERT_REQUIRED is present



##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
 
         # If we are using SSL disable requiring certs from specific hostname
         if options.get("ssl", False):
+            allow_insecure = self.extras.get("allow_insecure", True)
             if pymongo.__version__ >= "4.0.0":
-                # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
-                # replaces `ssl_cert_reqs=CERT_NONE`
-                options.update({"tlsAllowInvalidCertificates": True})
+                # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
+                options["tlsAllowInvalidCertificates"] = allow_insecure
             else:
-                options.update({"ssl_cert_reqs": CERT_NONE})
+                # For older pymongo versions, use 'ssl_cert_reqs'
+                options["ssl_cert_reqs"] = None if allow_insecure else CERT_NONE

Review Comment:
   Okay realised that CERT_REQUIRED is present and provided it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481643804


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True

Review Comment:
   Didn't completely get you here. The user can always come back to old behaviour by setting the ssl as False, right or not specifying it at all. 
   
   The old behaviour was that irrespective of ssl, we skipped tls validation. Should we retain that behaviour?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482980113


##########
airflow/providers/mongo/CHANGELOG.rst:
##########
@@ -27,6 +27,17 @@
 Changelog
 ---------
 
+3.7.0
+......

Review Comment:
   Post merge comment:
   
   I guess the version should be 4.0.0 in case of breaking changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481130143


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True
+        if self.extras.get("ssl", False):

Review Comment:
   According to the [`MongoClient`](https://pymongo.readthedocs.io/en/stable/api/pymongo/mongo_client.html#pymongo.mongo_client.MongoClient) `ssl` is alias to the `tls`, potentially end users might use `tls` instead of `ssl`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1933671125

   > Really Nice. Only docs/spellcheck fix and you can merge it.
   
   Just ran it locally and pushed a fix, we should be good!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37214:
URL: https://github.com/apache/airflow/pull/37214


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931432371

   Also tried adding mongo://127.0.0.1, this doesnt work either for uri


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482586148


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+        self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"
+
+        if self.ssl_enabled and not self.allow_insecure:
+            # Case: HTTPS
+            self.allow_insecure = False
+        elif self.ssl_enabled and self.allow_insecure:
+            # Case: HTTPS + allow_insecure
+            self.allow_insecure = True
+            self.extras.pop("ssl", None)
+        elif not self.ssl_enabled and "allow_insecure" in self.extras:
+            # Case: HTTP (ssl=False) with allow_insecure specified

Review Comment:
   Yeah makes sense. Making the changes
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931637366

   Thanks @Taragolis !
   That tip saved hassle for me, ran it via docker compose and it worked :D


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931678760

   > That tip saved hassle for me, ran it via docker compose and it worked :D
   
   Yeah in breeze only available simple integration, which need to be run with `--integration mongo`, e.g.
   
   ```console
   ❯ breeze start-airflow --backend postgres --postgres-version 15 --integration mongo
   ```
   
   After that you could communicate with Mongo 3 thought the host `mongo`
   
   ```python
   from airflow import DAG
   from airflow.decorators import task
   from airflow.providers.mongo.hooks.mongo import MongoHook
   
   
   with DAG("pr_37214", schedule=None, tags=["37214", "mongodb"]):
       @task
       def test_mongo():
           # Create a connection into the runtime, might be replaced by connection created into the UI
           import os
           from airflow.models.connection import Connection
   
           mongo_conn_id = "mongo_test"
           conn = Connection(conn_id=mongo_conn_id, conn_type="mongo", host="mongo", port=27017, schema="test")
           os.environ[f"AIRFLOW_CONN_{conn.conn_id.upper()}"] = conn.as_json()
   
           # Actual execution
           mongo_hook = MongoHook(mongo_conn_id=mongo_conn_id)
           mongo_hook.insert_one("users", {"name": "John", "surname": "Wick"}, mongo_db="test")
   
           collection = mongo_hook.get_collection("users", mongo_db="test")
           print(collection.count_documents({}))
   
       test_mongo()
   ```
   
   And it is expected that if you try to communicate in breeze with `localhost` or `127.0.0.1` that is not exists, because there is no mongo in Airflow Breeze container 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931686673

   > > That tip saved hassle for me, ran it via docker compose and it worked :D
   > 
   > Yeah in breeze only available simple integration, which need to be run with `--integration mongo`, e.g.
   > 
   > ```
   > ❯ breeze start-airflow --backend postgres --postgres-version 15 --integration mongo
   > ```
   > 
   > After that you could communicate with Mongo 3 thought the host `mongo`
   > 
   > ```python
   > from airflow import DAG
   > from airflow.decorators import task
   > from airflow.providers.mongo.hooks.mongo import MongoHook
   > 
   > 
   > with DAG("pr_37214", schedule=None, tags=["37214", "mongodb"]):
   >     @task
   >     def test_mongo():
   >         # Create a connection into the runtime, might be replaced by connection created into the UI
   >         import os
   >         from airflow.models.connection import Connection
   > 
   >         mongo_conn_id = "mongo_test"
   >         conn = Connection(conn_id=mongo_conn_id, conn_type="mongo", host="mongo", port=27017, schema="test")
   >         os.environ[f"AIRFLOW_CONN_{conn.conn_id.upper()}"] = conn.as_json()
   > 
   >         # Actual execution
   >         mongo_hook = MongoHook(mongo_conn_id=mongo_conn_id)
   >         mongo_hook.insert_one("users", {"name": "John", "surname": "Wick"}, mongo_db="test")
   > 
   >         collection = mongo_hook.get_collection("users", mongo_db="test")
   >         print(collection.count_documents({}))
   > 
   >     test_mongo()
   > ```
   > 
   > And it is expected that if you try to communicate in breeze with `localhost` or `127.0.0.1` that is not exists, because there is no mongo in Airflow Breeze container
   
   Perfect! I wasn't aware of this, thank you!
   This is good :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481609101


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True

Review Comment:
   I think allow_insecure should be  a separate extra, to alllow the user to come back to old behaviour. BTW. This is a breaking change so it neds to be explained in CHANGELOG for the provider.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480908546


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
 
         # If we are using SSL disable requiring certs from specific hostname
         if options.get("ssl", False):
+            allow_insecure = self.extras.get("allow_insecure", True)
             if pymongo.__version__ >= "4.0.0":
-                # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
-                # replaces `ssl_cert_reqs=CERT_NONE`
-                options.update({"tlsAllowInvalidCertificates": True})
+                # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
+                options["tlsAllowInvalidCertificates"] = allow_insecure
             else:
-                options.update({"ssl_cert_reqs": CERT_NONE})
+                # For older pymongo versions, use 'ssl_cert_reqs'
+                options["ssl_cert_reqs"] = None if allow_insecure else CERT_NONE

Review Comment:
   Is there any field in the connection that gets the certs from the hook? Ideally that should be provided here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931415839

   Right now I am testing with the DAG:
   ```
   from airflow import DAG
   from airflow.providers.mongo.hooks.mongo import MongoHook
   from airflow.operators.python_operator import PythonOperator
   from datetime import datetime, timedelta
   
   default_args = {
       'owner': 'airflow',
       'depends_on_past': False,
       'start_date': datetime(2024, 2, 7),
       'catchup': False,
   }
   
   
   def retrieve_users_from_mongodb():
       mongo_hook = MongoHook(conn_id='mongo_conn')
       query = {}
       users_data = mongo_hook.find('users', query)
       for user in users_data:
           print(user)
   
   
   with DAG('mongo_dag', default_args=default_args, schedule_interval=None) as dag:
       # Task to retrieve users from MongoDB
       retrieve_users_task = PythonOperator(
           task_id='retrieve_users_from_mongodb',
           python_callable=retrieve_users_from_mongodb
       )
   
       # Define the task dependencies
       retrieve_users_task
   
   ```
   
   Connection:
   ![image](https://github.com/apache/airflow/assets/35884252/df44ba08-5aad-4e69-a586-fdf04c98309f)
   
   
   For some reason, the connection is not going through, to mongodb
   
   Error log:
   ```
   e8786b12ac04
   *** Found local files:
   ***   * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T07:07:35.163848+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log
   [2024-02-07, 12:37:37 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti=
   [2024-02-07, 12:37:37 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti=
   [2024-02-07, 12:37:37 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1
   [2024-02-07, 12:37:37 IST] {taskinstance.py:2215} INFO - Executing  on 2024-02-07 07:07:35.163848+00:00
   [2024-02-07, 12:37:37 IST] {standard_task_runner.py:60} INFO - Started process 1243 to run task
   [2024-02-07, 12:37:37 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T07:07:35.163848+00:00', '--job-id', '920', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmpkg6apbfb']
   [2024-02-07, 12:37:37 IST] {standard_task_runner.py:88} INFO - Job 920: Subtask retrieve_users_from_mongodb
   [2024-02-07, 12:37:37 IST] {task_command.py:423} INFO - Running  on host e8786b12ac04
   [2024-02-07, 12:37:37 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='airflow@example.com' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T07:07:35.163848+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T07:07:35.163848+00:00'
   [2024-02-07, 12:37:37 IST] {base.py:83} INFO - Using connection ID 'myconn' for task execution.
   [2024-02-07, 12:38:08 IST] {taskinstance.py:2737} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/opt/airflow/airflow/models/taskinstance.py", line 446, in _execute_task
       result = _execute_callable(context=context, **execute_callable_kwargs)
     File "/opt/airflow/airflow/models/taskinstance.py", line 416, in _execute_callable
       return execute_callable(context=context, **execute_callable_kwargs)
     File "/opt/airflow/airflow/operators/python.py", line 199, in execute
       return_value = self.execute_callable()
     File "/opt/airflow/airflow/operators/python.py", line 216, in execute_callable
       return self.python_callable(*self.op_args, **self.op_kwargs)
     File "/files/dags/mongo_test.py", line 28, in retrieve_users_from_mongodb
       for user in users_data:
     File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1264, in next
       if len(self.__data) or self._refresh():
     File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh
       self.__session = self.__collection.database.client._ensure_session()
     File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1823, in _ensure_session
       return self.__start_session(True, causal_consistency=False)
     File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
       self._topology._check_implicit_session_support()
     File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 573, in _check_implicit_session_support
       self._check_session_support()
     File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 589, in _check_session_support
       self._select_servers_loop(
     File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 259, in _select_servers_loop
       raise ServerSelectionTimeoutError(
   pymongo.errors.ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>
   [2024-02-07, 12:38:08 IST] {taskinstance.py:1152} INFO - Marking task as FAILED. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T070735, start_date=20240207T070737, end_date=20240207T070808
   [2024-02-07, 12:38:08 IST] {standard_task_runner.py:107} ERROR - Failed to execute job 920 for task retrieve_users_from_mongodb (127.0.0.1:27017: [Errno 111] Connection refused (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>; 1243)
   [2024-02-07, 12:38:08 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 1
   [2024-02-07, 12:38:08 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check
   ```
   
   Point to note is that when I use pymongo directly, it works
   ```
   from pymongo import MongoClient
   
   mongo_host = "127.0.0.1"
   mongo_port = 27017
   mongo_database = "test"
   mongo_collection = "users" 
   
   client = MongoClient(mongo_host, mongo_port)
   
   db = client[mongo_database]
   collection = db[mongo_collection]
   
   query_result = collection.find_one({"name": "John"})
   print("Query result:", query_result)
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1931650831

   Awesome. I was able to test it.
   
   Results:
   
   My connections defined:
   <img width="867" alt="image" src="https://github.com/apache/airflow/assets/35884252/06a85bbd-579b-436b-9079-ec27499120c2">
   
   
   Logs for my_mong (ssl is not present):
   ```
   c47a4cad3064
   *** Found local files:
   ***   * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T09:32:50.030512+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log
   [2024-02-07, 15:02:53 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti=
   [2024-02-07, 15:02:53 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti=
   [2024-02-07, 15:02:53 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1
   [2024-02-07, 15:02:53 IST] {taskinstance.py:2215} INFO - Executing  on 2024-02-07 09:32:50.030512+00:00
   [2024-02-07, 15:02:53 IST] {standard_task_runner.py:60} INFO - Started process 1073 to run task
   [2024-02-07, 15:02:53 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T09:32:50.030512+00:00', '--job-id', '929', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmp1dqi4wjk']
   [2024-02-07, 15:02:53 IST] {standard_task_runner.py:88} INFO - Job 929: Subtask retrieve_users_from_mongodb
   [2024-02-07, 15:02:53 IST] {task_command.py:423} INFO - Running  on host c47a4cad3064
   [2024-02-07, 15:02:53 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='airflow@example.com' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T09:32:50.030512+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T09:32:50.030512+00:00'
   [2024-02-07, 15:02:53 IST] {base.py:83} INFO - Using connection ID 'my_mong' for task execution.
   [2024-02-07, 15:02:53 IST] {logging_mixin.py:188} INFO - {'_id': ObjectId('65c34e0e7e0c274d333d8922'), 'username': 'user1', 'email': 'user1@example.com'}
   [2024-02-07, 15:02:53 IST] {logging_mixin.py:188} INFO - {'_id': ObjectId('65c34e0e7e0c274d333d8923'), 'username': 'user2', 'email': 'user2@example.com'}
   [2024-02-07, 15:02:53 IST] {python.py:201} INFO - Done. Returned value was: None
   [2024-02-07, 15:02:53 IST] {taskinstance.py:1152} INFO - Marking task as SUCCESS. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T093250, start_date=20240207T093253, end_date=20240207T093253
   [2024-02-07, 15:02:53 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 0
   [2024-02-07, 15:02:53 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check
   ```
   
   Logs for my_mong_secure (explicitly passed my ssl: true)
   ```
   c47a4cad3064
   *** Found local files:
   ***   * /root/airflow/logs/dag_id=mongo_dag/run_id=manual__2024-02-07T09:36:02.449095+00:00/task_id=retrieve_users_from_mongodb/attempt=1.log
   [2024-02-07, 15:06:05 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=non-requeueable deps ti=
   [2024-02-07, 15:06:05 IST] {taskinstance.py:1980} INFO - Dependencies all met for dep_context=requeueable deps ti=
   [2024-02-07, 15:06:05 IST] {taskinstance.py:2194} INFO - Starting attempt 1 of 1
   [2024-02-07, 15:06:05 IST] {taskinstance.py:2215} INFO - Executing  on 2024-02-07 09:36:02.449095+00:00
   [2024-02-07, 15:06:05 IST] {standard_task_runner.py:60} INFO - Started process 1397 to run task
   [2024-02-07, 15:06:05 IST] {standard_task_runner.py:87} INFO - Running: ['airflow', 'tasks', 'run', 'mongo_dag', 'retrieve_users_from_mongodb', 'manual__2024-02-07T09:36:02.449095+00:00', '--job-id', '930', '--raw', '--subdir', 'DAGS_FOLDER/mongo_test.py', '--cfg-path', '/tmp/tmpn18c8wnm']
   [2024-02-07, 15:06:05 IST] {standard_task_runner.py:88} INFO - Job 930: Subtask retrieve_users_from_mongodb
   [2024-02-07, 15:06:05 IST] {task_command.py:423} INFO - Running  on host c47a4cad3064
   [2024-02-07, 15:06:05 IST] {taskinstance.py:2515} INFO - Exporting env vars: AIRFLOW_CTX_DAG_EMAIL='airflow@example.com' AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='mongo_dag' AIRFLOW_CTX_TASK_ID='retrieve_users_from_mongodb' AIRFLOW_CTX_EXECUTION_DATE='2024-02-07T09:36:02.449095+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2024-02-07T09:36:02.449095+00:00'
   [2024-02-07, 15:06:05 IST] {base.py:83} INFO - Using connection ID 'my_mong_secure' for task execution.
   [2024-02-07, 15:06:35 IST] {taskinstance.py:2737} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/opt/airflow/airflow/models/taskinstance.py", line 446, in _execute_task
       result = _execute_callable(context=context, **execute_callable_kwargs)
     File "/opt/airflow/airflow/models/taskinstance.py", line 416, in _execute_callable
       return execute_callable(context=context, **execute_callable_kwargs)
     File "/opt/airflow/airflow/operators/python.py", line 199, in execute
       return_value = self.execute_callable()
     File "/opt/airflow/airflow/operators/python.py", line 216, in execute_callable
       return self.python_callable(*self.op_args, **self.op_kwargs)
     File "/files/dags/mongo_test.py", line 28, in retrieve_users_from_mongodb
       for user in users_data:
     File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1264, in next
       if len(self.__data) or self._refresh():
     File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh
       self.__session = self.__collection.database.client._ensure_session()
     File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1823, in _ensure_session
       return self.__start_session(True, causal_consistency=False)
     File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
       self._topology._check_implicit_session_support()
     File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 573, in _check_implicit_session_support
       self._check_session_support()
     File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 589, in _check_session_support
       self._select_servers_loop(
     File "/usr/local/lib/python3.8/site-packages/pymongo/topology.py", line 259, in _select_servers_loop
       raise ServerSelectionTimeoutError(
   pymongo.errors.ServerSelectionTimeoutError: SSL handshake failed: 172.20.0.3:27017: TLS/SSL connection has been closed (EOF) (_ssl.c:1131) (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>
   [2024-02-07, 15:06:35 IST] {taskinstance.py:1152} INFO - Marking task as FAILED. dag_id=mongo_dag, task_id=retrieve_users_from_mongodb, execution_date=20240207T093602, start_date=20240207T093605, end_date=20240207T093635
   [2024-02-07, 15:06:35 IST] {standard_task_runner.py:107} ERROR - Failed to execute job 930 for task retrieve_users_from_mongodb (SSL handshake failed: 172.20.0.3:27017: TLS/SSL connection has been closed (EOF) (_ssl.c:1131) (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 30s, Topology Description: ]>; 1397)
   [2024-02-07, 15:06:35 IST] {local_task_job_runner.py:234} INFO - Task exited with return code 1
   [2024-02-07, 15:06:35 IST] {taskinstance.py:3318} INFO - 0 downstream tasks scheduled from follow-on schedule check
   ```
   
   cc @Taragolis @potiuk 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1480958341


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -98,12 +101,13 @@ def get_conn(self) -> MongoClient:
 
         # If we are using SSL disable requiring certs from specific hostname
         if options.get("ssl", False):
+            allow_insecure = self.extras.get("allow_insecure", True)
             if pymongo.__version__ >= "4.0.0":
-                # In pymongo 4.0.0+ `tlsAllowInvalidCertificates=True`
-                # replaces `ssl_cert_reqs=CERT_NONE`
-                options.update({"tlsAllowInvalidCertificates": True})
+                # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'

Review Comment:
   Pymongo 4.0 was released in 2021 maybe we should just support >=4.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1933376396

   @potiuk this has been tested out with the following combinations:
   ```
   {
     "mongo_http": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": "{\"ssl\": \"false\"}"
     },
     "mongo_http_copy1": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": "{\"ssl\": \"false\"}"
     },
     "mongo_https_allow_insecure": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": "{\"ssl\": \"true\", \"allow_insecure\": \"true\"}"
     },
     "mongo_https_dont_allow_insecure": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": "{\"ssl\": \"true\", \"allow_insecure\": \"false\"}"
     },
     "old_no_ssl_set": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": ""
     },
     "old_ssl_false": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": "{\"ssl\": \"false\"}"
     },
     "old_ssl_true": {
       "conn_type": "mongo",
       "description": "",
       "login": "",
       "password": null,
       "host": "172.20.0.3",
       "port": 27017,
       "schema": "test",
       "extra": "{\"ssl\": \"true\"}"
     }
   }
   ```
   
   It works as expected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482507876


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+        self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"

Review Comment:
   (and also add docstring).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #37214:
URL: https://github.com/apache/airflow/pull/37214#issuecomment-1932289134

   @potiuk to maintain the older behaviour, I think one thing we can do is:
   in init:
   ```
       if "ssl" is in self.extra and if "allow_insecure" is not in self.extras:
           self.extras["allow_insecure"] = False
   ```
   
   and in get_conn:
   ```
       allow_insecure = get allow_insecure from options
   
       if options.get("ssl", False):
           if pymongo.__version__ >= "4.0.0":
               # For pymongo 4.0.0+, use 'tlsAllowInvalidCertificates'
               options["tlsAllowInvalidCertificates"] = allow_insecure
           else:
               # For older pymongo versions, use 'ssl_cert_reqs'
               options["ssl_cert_reqs"] = CERT_NONE if allow_insecure else CERT_REQUIRED
   ```
   Thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1482592429


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,23 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = self.extras.pop("allow_insecure", "false").lower() == "true"
+        self.ssl_enabled = self.extras.get("ssl", "false").lower() == "true"

Review Comment:
   Sure, making the changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481653314


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True

Review Comment:
   I believe `allow_insecure` only applies when `ssl` is used (plain http is always insecure). But in fact there were 2 states here:
   
   * http (ssl = False)
   * htps + allow_insecure (ssl = True)
   
   What we need is 3 states
   
   * http (ssl = False)
   * htps + allow_insecure (ssl = True, allow_insecure=True)
   * https  - (ssl = True, allow_insecure = False) this should be default when ssl
   
   So we need separate `allow_insecure` flag with default = False  (and this is breaking change)
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Adding certificate validation for ssl in mongo hook [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #37214:
URL: https://github.com/apache/airflow/pull/37214#discussion_r1481650748


##########
airflow/providers/mongo/hooks/mongo.py:
##########
@@ -75,6 +74,10 @@ def __init__(self, mongo_conn_id: str = default_conn_name, *args, **kwargs) -> N
         self.client: MongoClient | None = None
         self.uri = self._create_uri()
 
+        self.allow_insecure = True

Review Comment:
   Do you mean to just add a new allow_insecure in extra and keep ssl as-is? Would that add any value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org