You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/11 11:41:59 UTC
[GitHub] [airflow] kotaaaa opened a new issue, #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator
kotaaaa opened a new issue, #22907:
URL: https://github.com/apache/airflow/issues/22907
### Description
In [SQLCheckOperator](https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/operators/sql/index.html#airflow.operators.sql.SQLCheckOperator),
I propose to add a feature of the check option that makes it optionally skip instead of occurring error if the condition meets.
### Use case/motivation
I was looking for a feature similar to [SQLCheckOperator](https://airflow.apache.org/docs/apache-airflow/1.10.12/_api/airflow/operators/sql/index.html#airflow.operators.sql.SQLCheckOperator).
Actually, in SQLCheckOperator, if it does not match the condition, it throws error. However, for my case, I wanted to skip rather than throwing error.
I asked for advise at [slack](https://apache-airflow.slack.com/archives/CSS36QQS1/p1648534212464789) about how I can implement this, then I followed @dstandish's advise so that I could develop for just my case. However, I want to contribute to community by adding a new feature to solve this.
### Related issues
NONE
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22907:
URL: https://github.com/apache/airflow/issues/22907#issuecomment-1094948322
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #22907:
URL: https://github.com/apache/airflow/issues/22907#issuecomment-1094958083
This feels more like a custom use case that you should solve locally in your deployment.
However, should this be something needed then this should be solved in `BaseOprator`.
What are the use cases for this request?
We have similar of `soft_fail` in sensors:
https://github.com/apache/airflow/blob/a3dd8473e4c5bbea214ebc8d5545b75281166428/airflow/sensors/base.py#L83
which allows all sensors to set to skip on timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kotaaaa commented on issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator
Posted by GitBox <gi...@apache.org>.
kotaaaa commented on issue #22907:
URL: https://github.com/apache/airflow/issues/22907#issuecomment-1094989998
@eladkal, thank you for your reply!
For my case, process is folked by the result of `select count(*) from {bigquery's table}`
so, I created class like below.
In not only my case, but also general usecase, it might happen that the result of the query to bigquery's table will decide whether it should skip the process or proceed, I thought.
(below is actually Airflow v1)
```
from airflow.exceptions import AirflowException
from airflow.exceptions import AirflowSkipException
from airflow.models import BaseOperator
from airflow.hooks.base_hook import BaseHook
from airflow.contrib.hooks.bigquery_hook import BigQueryHook
from airflow.utils.decorators import apply_defaults
class SQLCheckSkipOperator(BaseOperator):
...
def execute(self, context=None):
self.log.info("Executing SQL check: %s", self.sql)
records = self.get_db_hook().get_first(self.sql)
self.log.info("Record: %s", records)
if not records:
raise AirflowSkipException("The query returned None") # skip the upstream tasks
elif not all([bool(r) for r in records]):
raise AirflowSkipException(
"Test failed.\nQuery:\n{query}\nResults:\n{records!s}".format(
query=self.sql, records=records
)
)
self.log.info("Success.")
...
class BigQueryCheckSkipOperator(SQLCheckSkipOperator):
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add SKIP process in addition to occurring ERROR in SQLCheckOperator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #22907: Add SKIP process in addition to occurring ERROR in SQLCheckOperator
URL: https://github.com/apache/airflow/issues/22907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org