You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "RNHTTR (via GitHub)" <gi...@apache.org> on 2023/08/24 03:15:51 UTC

[GitHub] [airflow] RNHTTR opened a new pull request, #33685: Disable openlineage for operator

RNHTTR opened a new pull request, #33685:
URL: https://github.com/apache/airflow/pull/33685

   It might be valuable to disable OpenLineage extraction at the Operator level instead of at the Airflow level. This PR adds an environment variable to check for Operators for which OpenLineage extraction has been disabled.
   
   
   
   <!-- Please keep an empty line above the dashes. -->
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] RNHTTR commented on pull request #33685: Disable openlineage for operator

Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1691590144

   The current config/env var `openlineage_disabled` is a boolean that disables OpenLineage for an entire Airflow instance. This allows users to disable OpenLineage for just a subset of Operators.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] RNHTTR commented on pull request #33685: Disable openlineage for operator

Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1691738968

   > > This allows users to disable OpenLineage for just a subset of Operators.
   > 
   > Is it duplicate of standard configurations Airflow Env Var? ~`AIRFLOW__OPENLINEAGE__OPENLINEAGE_DISABLED_FOR_OPERATORS`~ **upd** `AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS`
   
   I don't think I understand; `AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS` doesn't exist unless there's some wild cosmic coincidence and I named it the same :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1306046598


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean
+        example: "BashOperator,PythonOperatorOperator"

Review Comment:
   I **think** we should change the check and check fully qualifed name instead. The task_type uses __class__.name - and while it is good for display, it's not good to determine if we should disable/enable something. This is a generic function that could by also used to disable custom operators, so what happens if someone has 
   
   * my_package_a.MyOperator
   * my_package_b.MyOperator
   
   and we want to only disable the one from my_package_a ?
   
   With `task_type` it's not possible. Maybe a bit "artifdficial" but i think we should make it possible and non-amibguous. 
   
   I am not even sure if we don't have in our codebase two operators named in the same way (would be worth checking) 
   
   I think this parameter will be rarely used and there is little benefit of having it "easier" to add. I think those who will be adding the exclusion will value the "explicitness" and `non-ambiguity" over "brevity".
   
   We already have quite a few configurations where we enter fully qualified names of classes we refer.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1304403832


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -71,7 +78,7 @@ config:
           Comma-separated paths to custom OpenLineage extractors.
         type: string
         example: full.path.to.ExtractorClass;full.path.to.AnotherExtractorClass
-        default: ''
+        default: ""

Review Comment:
   Not sure if we should change those :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #33685: Allow to disable openlineage at operator level

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #33685:
URL: https://github.com/apache/airflow/pull/33685


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33685: Allow to disable openlineage at operator level

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1696375975

   This is an interesting/subtle issue you see here in the tests.... 
   
   I believe the problem is that at the time of openlineage import in tests,  ProvidersManager have not yet extracted the configuration for openlineage provider. (it is intialized by pytest automated fixture when the test is run).
   
    That's why - I think - the default value is missing and you are getting [openlineage/disabyled_for_operators] not found in config"
   
   The solution to that is to  extract this one:
   
   ```python
       _openlineage_disabled_for_operators = conf.get("openlineage", "disabled_for_operators")
       openlineage_disabled_for_operators = set(
           operator.strip() for operator in _openlineage_disabled_for_operators.split(";")
       )
   ```
   
   to become a cached_property returning the list of disabled operators:
   
   ```python
   @cached_property
   def disabled_operators() -> set[str]:
       _openlineage_disabled_for_operators = conf.get("openlineage", "disabled_for_operators")
       return set(
           operator.strip() for operator in _openlineage_disabled_for_operators.split(";")
       )
   ```
   
   then  change:
   
   ```
   if fully_qualified_class_name in self.openlineage_disabled_for_operators:
   ```
   
   into:
   
   ```
   if fully_qualified_class_name in self.disabled_operators
   ```
   
   This will change the list to be retrieved only after provider manager already loaded defaults for provider configuration.
   
   That should solve the test failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on pull request #33685: Disable openlineage for operator

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1690930581

   How is this different from setting this in config?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on pull request #33685: Disable openlineage for operator

Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1691667199

   > This allows users to disable OpenLineage for just a subset of Operators.
   
   Is it duplicate of standard configurations Airflow Env Var?
   `AIRFLOW__OPENLINEAGE__OPENLINEAGE_DISABLED_FOR_OPERATORS`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1304400070


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean
+        example: "BashOperator,PythonOperatorOperator"

Review Comment:
   I think those examples are wrong BTW. There should be fully qualified class names including the modules.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1306695081


##########
airflow/providers/openlineage/extractors/base.py:
##########
@@ -44,6 +46,13 @@ class BaseExtractor(ABC, LoggingMixin):
     """
 
     _allowed_query_params: list[str] = []
+    _openlineage_disabled_for_operators = os.getenv(
+        "OPENLINEAGE_DISABLED_FOR_OPERATORS",

Review Comment:
   How about the env var?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1691663498

   > My local shows the precommit for mypy for providers passed...
   > 
   > <img alt="image" width="630" src="https://user-images.githubusercontent.com/25823361/262976772-60742b20-176b-4a44-867f-d89cee27b35a.png">
   > 🤔
   
   Happens Mypy uses heuristics that "guesses" type of the variables. And sometimes it gets it wrong when it does not run on "all files". Local version of pre-commit runs only on the files that are affected by commit and MyPy often will pull the other files and import them to feed it's heuristcs. But not always. So sometimes mypy is not able to guess correctly the type if it is not given explicitly all files. 
   
   Also sometimes the result depend if you use the same breeze image or whether you have an old one - because you might have an older version of providers.
   
   Also sometimes it depends on the cache you have locally built. Sometimes that cache is just not invalidated properly.
   
   And to add up, all this is succintly explained  **JUST** below the error you see in CI. Including precise instructions on what you should do to reproduce the failure you see locally.
   
   ![image](https://github.com/apache/airflow/assets/595491/679d135f-e323-4f98-b9b6-0d42f8f0fce0)
   
   
   I wonder (maybe you have an idea @RNHTTR ) how to better communicate it  than the yellow message above :) ? Any ideas? 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1304401179


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean

Review Comment:
   Type is also wrong. Should be string. And setting the default to "", is a good idea as then you do not have to add fallback (UPDATE: default is already there I see. cool).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1304401179


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean

Review Comment:
   Type is also wrong. Should be string. And setting the default to "", is a good idea as then you do not have to add fallback (default is already there I see. cool).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1304401179


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean

Review Comment:
   Type is also wrong. Should be string. And setting the default to "", is a good idea as then you will not have to add fallback.



##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean

Review Comment:
   Type is also wrong. Should be string. And setting the default to "", is a good idea as then you do not have to add fallback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] RNHTTR commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1306036623


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean
+        example: "BashOperator,PythonOperatorOperator"

Review Comment:
   I don't _think_ so, given that the check uses `<operator>.task_type`
   
   ```python
   >>> from airflow.operators.python import PythonOperator
   >>> f = lambda x: print(x)
   >>> p = PythonOperator(task_id="test", python_callable=f)
   >>> p.task_type
   'PythonOperator'
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] RNHTTR commented on pull request #33685: Disable openlineage for operator

Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1691611089

   <img width="630" alt="image" src="https://github.com/apache/airflow/assets/25823361/60742b20-176b-4a44-867f-d89cee27b35a">
   
   🤔 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1691752784

   > > > This allows users to disable OpenLineage for just a subset of Operators.
   > > 
   > > 
   > > Is it duplicate of standard configurations Airflow Env Var? ~`AIRFLOW__OPENLINEAGE__OPENLINEAGE_DISABLED_FOR_OPERATORS`~ **upd** `AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS`
   > 
   > I don't think I understand; `AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS` doesn't exist unless there's some wild cosmic coincidence and I named it the same :)
   
   @RHATTR: When yo do this (note I removed the "openlineage_" from your code to avoid double-openlineage in variable):
   
   ```
   conf.get("openlineage", "disabled_for_operators", fallback=""),
   ```
   
   Then conf.get will automatically read stuff from `AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS` - like for all the otehr config values.
   
   What @Taragolis is saying is that:
   
   ```
      _openlineage_disabled_for_operators = os.getenv(
           "OPENLINEAGE_DISABLED_FOR_OPERATORS",
           conf.get("openlineage", "openlineage_disabled_for_operators", fallback=""),
       )
   ```
   
   can be replaced with:
   
   ```
      _openlineage_disabled_for_operators = conf.get("openlineage", "disabled_for_operators"),
   ```
   
   And it will:
   
   * automatically use the defaults defined in `provider.yaml` (so no need for a fallback) 
   * automaticallly read configuraiton from `AIRFLOW__OPENLINEAGE__DISABLED_FOR_OPERATORS` env var
   
   No need to do all the `os.getenv` stuff.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1304402959


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -59,6 +59,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: boolean
+        example: "BashOperator,PythonOperatorOperator"
+        default: ""
+        version_added: ~

Review Comment:
   Here we should add open-lineage version in which it will be added. Which I think should be 1.1.0 as this is a new feature.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #33685: Disable openlineage for operator

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #33685:
URL: https://github.com/apache/airflow/pull/33685#discussion_r1307258071


##########
airflow/providers/openlineage/provider.yaml:
##########
@@ -60,6 +60,13 @@ config:
         example: ~
         default: "False"
         version_added: ~
+      disabled_for_operators:
+        description: |
+          Comma separated string of Airflow Operator names to disable
+        type: string
+        example: "airflow.operators.bash.BashOperator,airflow.operators.python.PythonOperator"

Review Comment:
   Maybe `;` separated to keep it consistent with `extractors` setting?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org