[GitHub] [airflow] potiuk commented on pull request #33685: Allow to disable openlineage at operator level

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on PR #33685:
URL: https://github.com/apache/airflow/pull/33685#issuecomment-1696375975

   This is an interesting/subtle issue you see here in the tests.... 
   
   I believe the problem is that at the time of openlineage import in tests,  ProvidersManager have not yet extracted the configuration for openlineage provider. (it is intialized by pytest automated fixture when the test is run).
   
    That's why - I think - the default value is missing and you are getting [openlineage/disabyled_for_operators] not found in config"
   
   The solution to that is to  extract this one:
   
   ```python
       _openlineage_disabled_for_operators = conf.get("openlineage", "disabled_for_operators")
       openlineage_disabled_for_operators = set(
           operator.strip() for operator in _openlineage_disabled_for_operators.split(";")
       )
   ```
   
   to become a cached_property returning the list of disabled operators:
   
   ```python
   @cached_property
   def disabled_operators() -> set[str]:
       _openlineage_disabled_for_operators = conf.get("openlineage", "disabled_for_operators")
       return set(
           operator.strip() for operator in _openlineage_disabled_for_operators.split(";")
       )
   ```
   
   then  change:
   
   ```
   if fully_qualified_class_name in self.openlineage_disabled_for_operators:
   ```
   
   into:
   
   ```
   if fully_qualified_class_name in self.disabled_operators
   ```
   
   This will change the list to be retrieved only after provider manager already loaded defaults for provider configuration.
   
   That should solve the test failure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org