You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mobuchowski (via GitHub)" <gi...@apache.org> on 2023/06/05 16:40:38 UTC
[GitHub] [airflow] mobuchowski opened a new pull request, #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
mobuchowski opened a new pull request, #31717:
URL: https://github.com/apache/airflow/pull/31717
The idea for system tests is as follows:
- there is `VariableTransport` that will push OpenLineage events to `Variable` with key `<DAG_ID>.<TASK_ID>.event.<EVENT_TYPE>`.
- System tests DAGs will be configured to use this transport
- then, newly added `OpenLineageTestOperator` will compare the `expected` events with the ones that are stored in Variables.
This PR adds those components, as well as OL support for `BigQueryGetDataOperator` and OL support for system test utilizing it.
PR is based on previous PR: https://github.com/apache/airflow/pull/31293 - only second commit is relevant to this PR.
Closes: #29676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mobuchowski commented on a diff in pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #31717:
URL: https://github.com/apache/airflow/pull/31717#discussion_r1228010282
##########
tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py:
##########
@@ -235,11 +236,21 @@
trigger_rule=TriggerRule.ALL_DONE,
)
+ openlineage_test = OpenlineageTestOperator(
Review Comment:
I believe it's advantageous to have OL system tests as an addition to regular ones, because they are just an addition of one operator, and actively use rest of the dag.
Those aren't only "OpenLineage system tests" in vacuum, but test OL integration of this particular provider.
Having separate dags for system tests would result in fragmentation - we'd have to maintain the same dag, or at least subset of it, with just the addition of a single `OpenLineageTestOperator`.
However, I believe option to disable OpenLineage and make this special operator always pass is a very valid need, and will add such option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mobuchowski closed pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski closed pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
URL: https://github.com/apache/airflow/pull/31717
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sunank200 commented on a diff in pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "sunank200 (via GitHub)" <gi...@apache.org>.
sunank200 commented on code in PR #31717:
URL: https://github.com/apache/airflow/pull/31717#discussion_r1227886086
##########
airflow/providers/openlineage/extractors/base.py:
##########
@@ -84,6 +84,12 @@ def get_operator_classnames(cls) -> list[str]:
def extract(self) -> OperatorLineage | None:
try:
return self._get_openlineage_facets(self.operator.get_openlineage_facets_on_start) # type: ignore
+ except ImportError:
+ self.log.error(
+ "OpenLineage provider method failed to import OpenLineage integration. "
+ "This should not happen."
Review Comment:
should we not add what user should do when this error occurs?
##########
tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py:
##########
@@ -235,11 +236,21 @@
trigger_rule=TriggerRule.ALL_DONE,
)
+ openlineage_test = OpenlineageTestOperator(
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mobuchowski commented on a diff in pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on code in PR #31717:
URL: https://github.com/apache/airflow/pull/31717#discussion_r1271481452
##########
airflow/providers/openlineage/extractors/base.py:
##########
@@ -84,6 +84,12 @@ def get_operator_classnames(cls) -> list[str]:
def extract(self) -> OperatorLineage | None:
try:
return self._get_openlineage_facets(self.operator.get_openlineage_facets_on_start) # type: ignore
+ except ImportError:
+ self.log.error(
+ "OpenLineage provider method failed to import OpenLineage integration. "
+ "This should not happen."
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #31717:
URL: https://github.com/apache/airflow/pull/31717#discussion_r1227775807
##########
tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py:
##########
@@ -235,11 +236,21 @@
trigger_rule=TriggerRule.ALL_DONE,
)
+ openlineage_test = OpenlineageTestOperator(
Review Comment:
I am wondering if this is the right place to keep OpenLineage system tests or should we have it in its own home `tests/system/providers/openlineage/...`? WDYT?
Wondering if some config disables OpenLineage or uses a different lineage backend then would having this here affect the system test run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajkoti commented on a diff in pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on code in PR #31717:
URL: https://github.com/apache/airflow/pull/31717#discussion_r1227775807
##########
tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py:
##########
@@ -235,11 +236,21 @@
trigger_rule=TriggerRule.ALL_DONE,
)
+ openlineage_test = OpenlineageTestOperator(
Review Comment:
I am wondering if this is the right place to keep OpenLineage system tests or should we have it in its own home system/providers/openlineage? WDYT?
Wondering if some config disables OpenLineage or uses a different lineage backend then would having this here affect the system test run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mobuchowski commented on pull request #31717: openlineage: add support for system tests, add support for example_bigquery_queries system test
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski commented on PR #31717:
URL: https://github.com/apache/airflow/pull/31717#issuecomment-1702905647
Will reopen later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org