You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "patricker (via GitHub)" <gi...@apache.org> on 2023/06/09 21:21:31 UTC
[GitHub] [airflow] patricker opened a new issue, #31823: Sensor with Deferrable not set (default is False) is Deferring
patricker opened a new issue, #31823:
URL: https://github.com/apache/airflow/issues/31823
### Apache Airflow version
2.6.1
### What happened
A BigQueryTableExistenceSensor without deferrable set to True is deferring. Since we don't have a triggerer, the status of these sensors stays in 'deferred'.
### What you think should happen instead
Sensor has not been marked as deferred, so it should run normally on the worker.
### How to reproduce
Add the sensor to a DAG and run it in an environment without a triggerer service running.
export_freshness = BigQueryTableExistenceSensor(
project_id=_project_id,
dataset_id=dataset_id,
table_id=table_id,
dag=dag,
task_id="daily_export_sensor",
gcp_conn_id="google_cloud_default",
poke_interval=60 * 5,
)
### Operating System
Debian
### Versions of Apache Airflow Providers
apache-airflow-providers-google==10.0.0
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
We just upgraded 2.6.1, and have only seen the issue since upgrading.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1585169066
I believe that this issue was resolved by #30946 which was released in 10.1.0 of the google provider.
Can you check what is the version you are using and upgrade to the latest version if it is not already the case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "patricker (via GitHub)" <gi...@apache.org>.
patricker commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1588109064
@potiuk , Sorry, a slight miscommunication on my part, let me expand on my earlier statements.
First, the emphasis I was trying to make was not on "repeatable installation", but on "known-to-be-working". Yes, I want a repeatable installation, but I also know that lots of people put time and effort into testing each release, and I don't want to just switch up package versions and possibly introduce issues when there is a "known-to-be-working" version readily available.
To clarify our install process of updated providers: occasionally new versions of providers are released _before_ a new version of Airflow is released. In these instances we sometimes need a bug fix that is in that new provider version (such as in this case where we have Airflow 2.6.1, but the updated provider is already available). Our assumption has been that if a new version of Airflow has not been released yet, that we should be able to update the version number on the affected provider inside our local copy of the constraints file (we keep a local copy for just this use case), especially if it's a minor version like this, without issue.
When we install an updated version of a provider we don't run `pip install apache-airflow-providers-google=...` at all. We update the version number in our local copy of the constraints file, and then setup our Python env with this new constraints file.
We only assume that there won't be a version issue if there has not been a new version of Airflow released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1588138441
I will make a PR with "how to generate your own constraints" chapter to our docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "patricker (via GitHub)" <gi...@apache.org>.
patricker commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1587744066
@hussein-awala We tried to test 10.1.0, but it's not compatible with the pip-constraints for Aifrlow 2.6.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1588137947
OK. So glad that I asked for details and clarifications.
That explains it. Replacing **just** provider in this case is wrong (or at least it won't always work). Simply because - precisely - you can have other dependencies conflicting. This is pretty much expected.
The right way of generating your own constraint is different (and now I see what is missing in our docs). I explained that in my recent presentation "Mastering custom dependencies" in London:
https://docs.google.com/presentation/d/1JSWM-pJMeSl2GUG8M1jw-2p46kLOcO9p8XMCx9OLvO8/edit (slides 45-47)
The right way of generating your new constraints is:
```
pip install apache-airflow[your extras here]==2.6.1 --constraints=...........
pip install apache-airflow==2.6.1 apache-airflow-providers-google==10.1.1
pip freeze > my-custom-constraints.txt
```
This will:
* install base version of airlfow with everything you need
* upgrade google provider making sure that pip will not accidentally upgrade airflow
* generate constraints from what you just installed
BTW. I will have an improved and shortened version of this presentation at Airlfow Summit - this time it will be recorded and published.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1587795733
BTW. I REALLY would love to know where the impression you should always use constraints is from. The reason why we are using constraints not requirements is precisely because they allow you to upgrade things independently, while keeping repeatable airflow installation. So by trying to upgrade something AND keeping the constraints you are effectively preventing the upgrade from being possible. ... So I really, really wonder where it came from.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "patricker (via GitHub)" <gi...@apache.org>.
patricker commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1587898031
Also... the presence of the providers themselves in the pip-constraints file leads in that direction too.
We have been doing all our provider upgrades by just updating the pip-constraints...
Example:
```
apache-airflow-providers-github==2.2.1
apache-airflow-providers-google==10.0.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1590140748
PR here: https://github.com/apache/airflow/pull/31882/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #31823: Sensor with Deferrable not set (default is False) is Deferring
URL: https://github.com/apache/airflow/issues/31823
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1588086695
Ok. Few more questions then (I want to see how we can update the description so that you can get it right.
> To have repeatable installation, however, we keep a set of "known-to-be-working" constraint files in the orphan constraints-main and constraints-2-0 branches. We keep those "known-to-be-working" constraints files separately per major/minor Python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and Python versions in the URL.
Could you please explain what "repeatable installation" means for you?
In other words when you are upgrading a provider on top of airflow, do you have expectations to have the same (i.e. repeatable) installation of airflow as with the original provider? Or do you want to have different, (non repeatble) installation with new provider?
> Also... the presence of the providers themselves in the pip-constraints file leads in that direction too.
We have been doing all our provider upgrades by just updating the pip-constraints...
So you already know that provider is in the constraints. And yet you are suprised that that when you are trying to install different version, than the one that you have in constraints. Is this what you are trying to tell?
In other words you expect this `pip install apache-airflow-providers-google=10.0.1 --constraints = <file>` where the file contains ` apache-airflow-providers-google=10.0.0` to work ?
What is your expectation to happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] patricker commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "patricker (via GitHub)" <gi...@apache.org>.
patricker commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1587893531
@potiuk I got that impression from the PyPi install docs.
> To have repeatable installation, however, we keep a set of "known-to-be-working" constraint files in the orphan constraints-main and constraints-2-0 branches. We keep those "known-to-be-working" constraints files separately per major/minor Python version. You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify correct Airflow tag/version/branch and Python versions in the URL.
The impression is that installing without constraints will give us an environment that is _not_ `"known-to-be-working" constraint`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1587778102
> @hussein-awala We tried to test 10.1.0, but it's not compatible with the pip-constraints for Airflow 2.6.1
Where do you get the impression @patricker that google provider should be installed with constraint if it gets updated. Our documentation is quite clear you should NOT use the constraints if you update providers separately from Airflow core and even (as of recently) explains why https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installing-upgrading-downgrading-providers-separately-from-airflow-core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31823: Sensor with Deferrable not set (default is False) is Deferring
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31823:
URL: https://github.com/apache/airflow/issues/31823#issuecomment-1587785037
Closing the issue provisionally unless we have information it is still not fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org