You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/01/11 07:38:31 UTC
[GitHub] [airflow] stevenzhang-support opened a new issue #20799: Does that Smart Sensor support the ExternalTaskSensor ?
stevenzhang-support opened a new issue #20799:
URL: https://github.com/apache/airflow/issues/20799
### Description
As far as I know, Smart Sensor is a new feature added by Airflow v2.x.x. Whether we use Google Cloud Composer 1 or Composer 2, the corresponding Airflow version is v2.1.4. What I suspect now is that Smart Sensor does not support the ExternalTaskSensor we want to use, because I found others have the same confusion [1].
Also, at first I thought it was related to the Composer version, so in our another Project I created a new Composer 2 and performed the same test, but the error was [2] reported. I believe this is only a feature issue within scope of Airflow without impact from Google Cloud as only a hosting vessel.
[1]: https://github.com/apache/airflow/discussions/13949
[2]: [2021-12-30 09:29:56,382] {smart_sensor.py:589} ERROR - Handling poke exception: Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/smart_sensor.py", line 641, in _execute_sensor_work
if self.poke(sensor_work):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/smart_sensor.py", line 700, in poke
operator_class = import_string(sensor_work.op_classpath)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/opt/python3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'unusual_prefix_81c243b038434094df429af5b5dfb379ff5c67aa_test_smart_sensor'
### Use case/motivation
The approach is to customize a Sensor, inherit the ExternalTaskSensor, and then rewrite the get_poke_context method. According to the log ('Task successfully registered in smart sensor !') It shows that the customized Sensor has been registered successfully. However, an exception was reported later. Only one issue from here found to be related: [1]
[1]:https://github.com/apache/airflow/discussions/13949
### Related issues
https://github.com/apache/airflow/discussions/13949
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stevenzhang-support commented on issue #20799: Does that Smart Sensor support the ExternalTaskSensor ?
Posted by GitBox <gi...@apache.org>.
stevenzhang-support commented on issue #20799:
URL: https://github.com/apache/airflow/issues/20799#issuecomment-1009677900
I have followed the issue template and thanks for confirming the below
1) Does that Smart Sensor support the ExternalTaskSensor
2) If Smart Sensor support the ExternalTaskSensor can be implemented?
BR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20799: Does that Smart Sensor support the ExternalTaskSensor ?
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20799:
URL: https://github.com/apache/airflow/issues/20799#issuecomment-1009922651
As mentioned in #13949 smart sensors are on their way to be deprecated. Please use Deferrable Operators instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #20799: Does that Smart Sensor support the ExternalTaskSensor ?
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #20799:
URL: https://github.com/apache/airflow/issues/20799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #20799: Does that Smart Sensor support the ExternalTaskSensor ?
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #20799:
URL: https://github.com/apache/airflow/issues/20799#issuecomment-1009673000
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org