You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/12/12 15:44:03 UTC

[PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

potiuk opened a new pull request, #36186:
URL: https://github.com/apache/airflow/pull/36186

   The released providers added support to previous version of the `airflow.io` - where options were not passed to `get_fs` method that provides Fsspec compatible FileSystem. However #35820 added positional "options" parameter when the method is called and it broke already released providers.
   
   This PR dynamically inspects signature of the get_fs method and when one parameter is detected, it will skip passing options to get_fs method call.
   
   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424629695


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   yeah - it's a bit more expensive, but It happens only once at discovery. 
   
   I thought about it too but I decided dointg it more explicitly is better. It is much better to handle (potential) patterns where we add more parameters (which might happen). In this case we will just add `elif len() ==2`  - it would be difficult to distinguish those cases with exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424623825


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   Isn't the typical pythonic way to do this with a try catch? Inspect is expensive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424966653


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   Also, this is pretty common pattern in Airflow already. We use inspect and looking at parameters and comparing signatures of methods in quite a numer of places already. I personally see it as much more explicit. But of course nothing prevents us from changing it in the future if we agree it's a better solution, this part is internal and can be changed any time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy merged PR #36186:
URL: https://github.com/apache/airflow/pull/36186


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin commented on PR #36186:
URL: https://github.com/apache/airflow/pull/36186#issuecomment-1852873466

   Mmm yes I understand. Sorry for this, I mentioned when merging the other it would require a release of the providers and wanted to prevent backwards compat check (which is silly imho as the current set of providers cannot work). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424629695


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   yeah - it's a bit more expensive, but It happens only once at discovery. 
   
   I thought about it too but I decided dointg it more explicitly is better. It is much better to handle (potential) patterns where we add more parameters (which might happen). In this case we will just add `if len() ==2`  - it would be difficult to distinguish those cases with exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424205810


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   Should we add a test for the old signature to verify



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424216835


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   Let me see. It's new code for me :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix compatibility of updated `airflow.io` with released providers [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #36186:
URL: https://github.com/apache/airflow/pull/36186#discussion_r1424225335


##########
airflow/io/__init__.py:
##########
@@ -86,7 +93,13 @@ def get_fs(
         raise ValueError(f"No filesystem registered for scheme {scheme}") from None
 
     options = storage_options or {}
-    return fs(conn_id, options)
+    # MyPy does not recognize dynamic parameters inspection when we call the method, and we have to do
+    # it for compatibility reasons with already released providers, that's why we need to ignore
+    # mypy errors here
+    parameters = inspect.signature(fs).parameters
+    if len(parameters) == 1:
+        return fs(conn_id)  # type: ignore[call-arg]
+    return fs(conn_id, options)  # type: ignore[call-arg]

Review Comment:
   Looks like there are no tests even for retrieval for providers, if time is of an essence, I'd rather leave it for later. created an issue: https://github.com/apache/airflow/issues/36187



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org