You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2023/04/18 14:44:47 UTC

[GitHub] [airflow] mobuchowski opened a new pull request, #30713: openlineage: add extractors for python and bash operators

mobuchowski opened a new pull request, #30713:
URL: https://github.com/apache/airflow/pull/30713

   Add extractor support for `PythonOperator` and `BashOperator`. It's a basic support, adding OL facets to them. It does not yet support advanced features that we plan in the future, like extracting data from taskflow-style jobs.
   
   Additionally, slightly refactor registering extractors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1180257900


##########
airflow/providers/openlineage/extractors/python.py:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import inspect
+import os
+from typing import Callable
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class PythonExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what particular task does by extracting
+    executed source code and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["PythonOperator"]
+
+    def extract(self) -> OperatorLineage | None:
+        # TODO Airflow config
+        collect_source = os.environ.get("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE", "True").lower() not in (
+            "true",
+            "1",
+            "t",
+        )
+
+        source_code = self.get_source_code(self.operator.python_callable)
+        job_facet: dict = {}
+        if collect_source and source_code:
+            job_facet = {
+                "sourceCode": SourceCodeJobFacet(
+                    language="python",
+                    # We're on worker and should have access to DAG files
+                    source=source_code,
+                )
+            }
+        return OperatorLineage(
+            job_facets=job_facet,
+            run_facets={
+                # The BashOperator is recorded as an "unknownSource" even though we have an
+                # extractor, as the <i>data lineage</i> cannot be determined from the operator
+                # directly.
+                "unknownSourceAttribute": UnknownOperatorAttributeRunFacet(
+                    unknownItems=[
+                        UnknownOperatorInstance(
+                            name="PythonOperator",
+                            properties={attr: value for attr, value in self.operator.__dict__.items()},
+                        )
+                    ]
+                )
+            },
+        )
+
+    def get_source_code(self, callable: Callable) -> str | None:
+        try:
+            return inspect.getsource(callable)
+        except TypeError:
+            # Trying to extract source code of builtin_function_or_method
+            return str(callable)
+        except OSError:
+            self.log.exception(f"Can't get source code facet of PythonOperator {self.operator.task_id}")

Review Comment:
   @josh-fell done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1545897613

   @ashb Okay, let's merge parts 1 and 2 of the implementation plan together. We'll create new PRs with OL support directly in operators. 
   
   For this one it won't work - since it requires support in core Airflow, and that would probably require waiting for 2.7 release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1189780454


##########
airflow/providers/openlineage/extractors/bash.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from airflow.providers.openlineage.utils.utils import is_source_enabled
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class BashExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what bash task does by extracting
+    executed bash command and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["BashOperator"]

Review Comment:
   Could be, but we'd have to keep `get_operator_classnames` on base extractor for backwards compatibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1525219818

   @josh-fell generally, those wouldn't be used by users of Airflow - you don't need to manually specify those paths anywhere, unless you want to add your own custom extractor, so I personally don't think it matters. I like shorter paths though and I'll change it. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1520031994

   Ideally I'd add `:meta private: to whole `airflow.providers.openlineage.extractors` package.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] josh-fell commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1524377423

   Just curious, should this provider be [AIP-21](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths) compliant? If so, maybe we have file paths of `airflow/providers/openlineage/extractors/bash.py` and `airflow/providers/openlineage/extractors/python.py` instead?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1520408787

   > > I think we should also (discussed it with @eladkal - add a mechanism - similar to suspended but only for releases) to exclude open-lineage from releasing until it is ready.
   > 
   > yeah the manual removal of the provider from the release is not so nice but I guess this is really an edge case... it's unlikely we will encounter another case of merged new provider that we don't want to release so I'm not sure if worth the effort setting this option
   
   Indeed. That's why I just HARDCODED It for now : https://github.com/apache/airflow/pull/30841


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1541788412

   @potiuk @josh-fell @pierrejeambrun can you merge?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1520339091

   I think we should also (discussed it with @eladkal - add a mechanism - similar to suspended but only for releases) to exclude open-lineage from releasing until it is ready.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1192488194


##########
tests/providers/openlineage/extractors/test_bash_extractor.py:
##########


Review Comment:
   @ashb added more tests with `@conf_vars`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb merged pull request #30713: openlineage: add extractors for python and bash operators

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb merged PR #30713:
URL: https://github.com/apache/airflow/pull/30713


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1523470659

   @potiuk @eladkal anything still needed to merge it? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1520337173

   > Ideally I'd add `:meta private:` to whole `airflow.providers.openlineage.extractors` package.
   
   yep


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1520382230

   > I think we should also (discussed it with @eladkal - add a mechanism - similar to suspended but only for releases) to exclude open-lineage from releasing until it is ready.
   
   yeah the manual removal of the provider from the release is not so nice but I guess this is really an edge case... it's unlikely we will encounter another case of merged new provider that we don't want to release so I'm not sure if worth the effort setting this option


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1187448772


##########
airflow/providers/openlineage/extractors/manager.py:
##########
@@ -25,6 +27,26 @@
     UnknownOperatorInstance,
 )
 from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.utils.module_loading import import_string
+
+if TYPE_CHECKING:
+    from airflow.models import Operator
+
+
+def try_import_from_string(string):
+    with suppress(ImportError):
+        return import_string(string)
+
+
+_extractors: list[BaseExtractor] = list(
+    filter(
+        lambda t: t is not None,
+        [
+            try_import_from_string("airflow.providers.openlineage.extractors.python.PythonExtractor"),
+            try_import_from_string("airflow.providers.openlineage.extractors.bash.BashExtractor"),

Review Comment:
   Yeah - moved those to regular imports.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1184722744


##########
airflow/providers/openlineage/extractors/bash.py:
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import os
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class BashExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what bash task does by extracting
+    executed bash command and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["BashOperator"]
+
+    def extract(self) -> OperatorLineage | None:
+        collect_source = os.environ.get("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE", "true").lower() not in (

Review Comment:
   Should be from airflow conf now?



##########
airflow/providers/openlineage/extractors/manager.py:
##########
@@ -25,6 +27,26 @@
     UnknownOperatorInstance,
 )
 from airflow.utils.log.logging_mixin import LoggingMixin
+from airflow.utils.module_loading import import_string
+
+if TYPE_CHECKING:
+    from airflow.models import Operator
+
+
+def try_import_from_string(string):
+    with suppress(ImportError):
+        return import_string(string)
+
+
+_extractors: list[BaseExtractor] = list(
+    filter(
+        lambda t: t is not None,
+        [
+            try_import_from_string("airflow.providers.openlineage.extractors.python.PythonExtractor"),
+            try_import_from_string("airflow.providers.openlineage.extractors.bash.BashExtractor"),

Review Comment:
   Given these are modules from the same provider I don't see how these could fail?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1189709939


##########
airflow/providers/openlineage/extractors/bash.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from airflow.providers.openlineage.utils.utils import is_source_enabled
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class BashExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what bash task does by extracting
+    executed bash command and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["BashOperator"]
+
+    def extract(self) -> OperatorLineage | None:
+        job_facet: dict = {}
+        if is_source_enabled():
+            job_facet = {
+                "sourceCode": SourceCodeJobFacet(
+                    language="bash",
+                    # We're on worker and should have access to DAG files
+                    source=self.operator.bash_command,
+                )
+            }
+
+        return OperatorLineage(
+            job_facets=job_facet,
+            run_facets={
+                # The BashOperator is recorded as an "unknownSource" even though we have an
+                # extractor, as the <i>data lineage</i> cannot be determined from the operator
+                # directly.
+                "unknownSourceAttribute": UnknownOperatorAttributeRunFacet(
+                    unknownItems=[
+                        UnknownOperatorInstance(
+                            name="BashOperator",
+                            properties={attr: value for attr, value in self.operator.__dict__.items()},

Review Comment:
   This is going to include the whole dag. (since `self.operator.dag` is a property.) Do you want that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1541912525

   I've just noticed something. Wasn't one of the key points of AIP-53 that we would have the facet extraction logic _on_ the operator directly?
   
   > As part of this new provider, we define a new optional contract for operators defined in the documentation here: [get_openlineage_facets*](https://openlineage.io/docs/integrations/airflow/operator/). Each


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] josh-fell commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1179853067


##########
airflow/providers/openlineage/extractors/python.py:
##########
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import inspect
+import os
+from typing import Callable
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class PythonExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what particular task does by extracting
+    executed source code and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["PythonOperator"]
+
+    def extract(self) -> OperatorLineage | None:
+        # TODO Airflow config
+        collect_source = os.environ.get("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE", "True").lower() not in (
+            "true",
+            "1",
+            "t",
+        )
+
+        source_code = self.get_source_code(self.operator.python_callable)
+        job_facet: dict = {}
+        if collect_source and source_code:
+            job_facet = {
+                "sourceCode": SourceCodeJobFacet(
+                    language="python",
+                    # We're on worker and should have access to DAG files
+                    source=source_code,
+                )
+            }
+        return OperatorLineage(
+            job_facets=job_facet,
+            run_facets={
+                # The BashOperator is recorded as an "unknownSource" even though we have an
+                # extractor, as the <i>data lineage</i> cannot be determined from the operator
+                # directly.
+                "unknownSourceAttribute": UnknownOperatorAttributeRunFacet(
+                    unknownItems=[
+                        UnknownOperatorInstance(
+                            name="PythonOperator",
+                            properties={attr: value for attr, value in self.operator.__dict__.items()},
+                        )
+                    ]
+                )
+            },
+        )
+
+    def get_source_code(self, callable: Callable) -> str | None:
+        try:
+            return inspect.getsource(callable)
+        except TypeError:
+            # Trying to extract source code of builtin_function_or_method
+            return str(callable)
+        except OSError:
+            self.log.exception(f"Can't get source code facet of PythonOperator {self.operator.task_id}")

Review Comment:
   ```suggestion
               self.log.exception("Can't get source code facet of PythonOperator %s", self.operator.task_id)
   ```
   To keep with the consistency of using %-formatting for logging.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1519752088

   Hey @potiuk - as I wrote [in the implementation doc](https://docs.google.com/document/d/1YQMg4xePhetyjXCBHXwU7jbJpi5TEcqhwN8YbQ32P4Q) we want to migrate them, as we already have them it in OL. Then move to gradually reimplementing this in-operator. This would allow us to release `apache-airflow-provider-openlineage` with parity to `openlineage-airflow`, so that users moving to Airflow 2.6 could migrate seamlessly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1520475352

   Thanks @eladkal @potiuk - that makes sense


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] josh-fell commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "josh-fell (via GitHub)" <gi...@apache.org>.
josh-fell commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1526856829

   > @josh-fell generally, those wouldn't be used by users of Airflow - you don't need to manually specify those paths anywhere, unless you want to add your own custom extractor, so I personally don't think it matters. I like shorter paths though and I'll change it.
   
   Of course. This was more of a philosophical question since this might not be considered a "traditional" provider so should the AIP apply. I'm a fan because it at least removes some schematic redundancy in the import path, but figured I'd pose the question.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1189713867


##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -395,3 +397,10 @@ def wrapper(*args, **kwargs):
             log.exception(e)
 
     return wrapper
+
+
+def is_source_enabled():

Review Comment:
   `from airflow.compat.functools import cache` and then 
   ```suggestion
   @cache
   def is_source_enabled():
   ```



##########
tests/providers/openlineage/extractors/test_bash_extractor.py:
##########


Review Comment:
   I don't see any tests that set the airflow config, just the "fallback" env. `@conf_vars` can help.



##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -395,3 +397,10 @@ def wrapper(*args, **kwargs):
             log.exception(e)
 
     return wrapper
+
+
+def is_source_enabled():
+    source_var = conf.get(
+        "openlinege", "disable_source_code", fallback=os.getenv("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE")
+    )
+    return isinstance(source_var, str) and source_var.lower() not in ("true", "1", "t")

Review Comment:
   ```suggestion
       return conf.getboolean(
           "openlinege",
           "disable_source_code",
           fallback=os.getenv("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE", "").lower() not in ("true", "1", "t")
       )
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1187488746


##########
airflow/providers/openlineage/extractors/bash.py:
##########
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+import os
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class BashExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what bash task does by extracting
+    executed bash command and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["BashOperator"]
+
+    def extract(self) -> OperatorLineage | None:
+        collect_source = os.environ.get("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE", "true").lower() not in (

Review Comment:
   Rebased, moved to Airflow conf with env fallback.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1189707840


##########
airflow/providers/openlineage/extractors/bash.py:
##########
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from airflow.providers.openlineage.extractors.base import BaseExtractor, OperatorLineage
+from airflow.providers.openlineage.plugins.facets import (
+    UnknownOperatorAttributeRunFacet,
+    UnknownOperatorInstance,
+)
+from airflow.providers.openlineage.utils.utils import is_source_enabled
+from openlineage.client.facet import SourceCodeJobFacet
+
+"""
+:meta private:
+"""
+
+
+class BashExtractor(BaseExtractor):
+    """
+    This extractor provides visibility on what bash task does by extracting
+    executed bash command and putting it into SourceCodeJobFacet. It does not extract
+    datasets.
+
+    :meta private:
+    """
+
+    @classmethod
+    def get_operator_classnames(cls) -> list[str]:
+        return ["BashOperator"]

Review Comment:
   This is maybe a bigger question, but is there a reason this couldn't be just a property?
   
   ```suggestion
       operator_classnames: list[str] = ["BashOperator"]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mobuchowski commented on a diff in pull request #30713: openlineage: add extractors for python and bash operators

Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #30713:
URL: https://github.com/apache/airflow/pull/30713#discussion_r1192489192


##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -395,3 +397,10 @@ def wrapper(*args, **kwargs):
             log.exception(e)
 
     return wrapper
+
+
+def is_source_enabled():
+    source_var = conf.get(
+        "openlinege", "disable_source_code", fallback=os.getenv("OPENLINEAGE_AIRFLOW_DISABLE_SOURCE_CODE")
+    )
+    return isinstance(source_var, str) and source_var.lower() not in ("true", "1", "t")

Review Comment:
   Can't do this this way unfortunately, since `conf.getboolean` needs default value, and default value prevents fallback from working, since it's taken before fallback.



##########
airflow/providers/openlineage/utils/utils.py:
##########
@@ -395,3 +397,10 @@ def wrapper(*args, **kwargs):
             log.exception(e)
 
     return wrapper
+
+
+def is_source_enabled():

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #30713: openlineage: add extractors for python and bash operators

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #30713:
URL: https://github.com/apache/airflow/pull/30713#issuecomment-1519772308

   > Hey @potiuk - as I wrote [in the implementation doc](https://docs.google.com/document/d/1YQMg4xePhetyjXCBHXwU7jbJpi5TEcqhwN8YbQ32P4Q) we want to migrate them, as we already have them it in OL. Then move to gradually reimplementing this in-operator. This would allow us to release `apache-airflow-provider-openlineage` with parity to `openlineage-airflow`, so that users moving to Airflow 2.6 could migrate seamlessly.
   
   Ah. OK makes sense. Can we mark all of those classes as :meta private: (not sure if this is the right way to mark the whole modules as private) and add description in those modules/classes so that it is clear those are private parts and not belonging to the API of the provider ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org