You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "fpopic (via GitHub)" <gi...@apache.org> on 2024/01/06 10:11:49 UTC
[I] SFTPSensor.newer_than not working with jinja logical date expression [airflow]
fpopic opened a new issue, #36629:
URL: https://github.com/apache/airflow/issues/36629
### Apache Airflow Provider(s)
sftp
### Versions of Apache Airflow Providers
apache-airflow-providers-sftp==4.4.0
### Apache Airflow version
2.6.3
### Operating System
Ubuntu 20.04.6 LTS
### Deployment
Virtualenv installation
### Deployment details
_No response_
### What happened
I tried to use parameter `newer_than` of type `datetime` with jinja expression for logical execution date `{{ ds }}` or timestamp `{{ ts }}` but couldn't find a single way how to please the `SFTPSensor.newer_than` checks.
```py
import datetime
import pendulum
from airflow import models
from airflow.providers.sftp.sensors.sftp import SFTPSensor
from airflow.operators.empty import EmptyOperator
with models.DAG(
"dag_example_sftp_sensor_newer_than_example",
schedule_interval="@once",
start_date=datetime.datetime(2024, 1, 1)
) as dag:
start_dag = EmptyOperator(task_id="start_dag")
end_dag = EmptyOperator(task_id="end_dag")
wait_for_sftp_file = SFTPSensor(
task_id=f"wait_for_sftp_file",
sftp_conn_id="sftp_conn_id",
path=f"some-other-jinja-expression-depending-on-airflow-ds-{{ds}}",
newer_than=pendulum.from_format('{{ ds }}', fmt="YYYY-MM-DD")
)
start_dag >> wait_for_sftp_file >> end_dag
```
I get
```
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/dagbag.py", line 346, in parse
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/airflow/gcs/dags/dag_example_sftp_sensor_newer_than_example.py", line XXX, in <module>
newer_than=pendulum.from_format('{{ ds }}', fmt="YYYY-MM-DD")
File "/opt/python3.8/lib/python3.8/site-packages/pendulum/__init__.py", line 259, in from_format
parts = _formatter.parse(string, fmt, now(), locale=locale)
File "/opt/python3.8/lib/python3.8/site-packages/pendulum/formatting/formatter.py", line 413, in parse
raise ValueError("String does not match format {}".format(fmt))
ValueError: String does not match format YYYY-MM-DD
```
In case I hard-code the value (not using jinja) to something like
```py
newer_than=pendulum.from_format('2024-01-01', fmt="YYYY-MM-DD")
```
everything works.
### What you think should happen instead
Parameter `newer_than` should be working with jinja templates {{ts}} or {{ds}}.
### How to reproduce
```shell
airflow tasks test dag_example_sftp_sensor_newer_than_example wait_for_sftp_file 2024-01-01
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1882637394
@AdamPaslawski @hussein-awala might you be able to help with
```py
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 39, in is_localized
return value.utcoffset() is not None
AttributeError: 'str' object has no attribute 'utcoffset'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1882612654
and seems that not all pendulum method are imported. Substract doesn't work. Add works.
```py
newer_than='{{ dag_run.logical_date.substract(days=1, hours=-2) }}',
```
```py
[2024-01-09T08:26:14.525+0000] {taskinstance.py:1826} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1408, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1533, in _execute_task_with_callbacks
task_orig = self.render_templates(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2182, in render_templates
original_task.render_template_fields(context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1266, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/session.py", line 76, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/abstractoperator.py", line 586, in _do_render_template_fields
rendered_content = self.render_template(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/template/templater.py", line 156, in render_template
return self._render(template, context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/abstractoperator.py", line 540, in _render
return super()._render(template, context, dag=dag)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/template/templater.py", line 113, in _render
return render_template_to_string(template, context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
return render_template(template, cast(MutableMapping[str, Any], context), native=False)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/helpers.py", line 283, in render_template
return "".join(nodes)
File "<template>", line 12, in root
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/sandbox.py", line 391, in call
if not __self.is_safe_callable(__obj):
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/sandbox.py", line 275, in is_safe_callable
getattr(obj, "unsafe_callable", False) or getattr(obj, "alters_data", False)
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/runtime.py", line 859, in __getattr__
return self._fail_with_undefined_error()
File "/opt/python3.8/lib/python3.8/site-packages/jinja2/runtime.py", line 852, in _fail_with_undefined_error
raise self._undefined_exception(self._undefined_message)
jinja2.exceptions.UndefinedError: 'pendulum.datetime.DateTime object' has no attribute 'substract'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical date expression [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1879626523
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [WIP] SFTPSensor.newer_than not working with jinja logical date expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1880199167
> @fpopic instead of using `pendulum.from_format('{{ ds }}', fmt="YYYY-MM-DD") `can you try `newer_than='{{ ds }}'` as defauft format is `YYYY-MM-DD` only
@vatsrahul1001 this works.
Moreover I found a way how to make working expressions based on airflow logical full datetime (date+time).
Was referring to
- https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/timezone.html
- https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html
and tried only options with parsing `ts` into datetime object.
However, found a simpler way using pendulum https://pendulum.eustace.io/docs/#addition-and-subtraction.
```py
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "grrolland (via GitHub)" <gi...@apache.org>.
grrolland commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-2058626348
Hi,
I try to fixe this issue here : https://github.com/apache/airflow/pull/39056
The newer_than parameter is actually not interpreted as a str when the sensor poke().
Hope it helps ! Works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #36629: SFTPSensor.newer_than not working with jinja logical ds/ts expression
URL: https://github.com/apache/airflow/issues/36629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1920184928
I will take a look tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [WIP] SFTPSensor.newer_than not working with jinja logical date expression [airflow]
Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1879651002
@fpopic instead of using `pendulum.from_format('{{ ds }}', fmt="YYYY-MM-DD") `can you try `newer_than='{{ ds }}'` as defauft format is `YYYY-MM-DD` only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1882609788
@RNHTTR @vatsrahul1001 Sorry for closing, but was too early.
My pattern was testing for non-existing file and it worked, but once the file exists on SFTP server, I am getting
```
[2024-01-09T08:18:29.763+0000] {sftp.py:90} INFO - Found File XXXX_YYY_KW2351_ZZZZ.WWW last modified: 20231221050445
[2024-01-09 08:18:29,771] {taskinstance.py:1826} ERROR - Task failed with exception\nTraceback (most recent call last):\n File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/base.py", line 225, in execute\n raise e\n File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/base.py", line 212, in execute\n poke_return = self.poke(context)\n File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/sftp/sensors/sftp.py", line 97, in poke\n _newer_than = convert_to_utc(self.newer_than)\n File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 104, in convert_to_utc\n if not is_localized(value):\n File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 39, in is_localized\n return value.utcoffset() is not None\nAttributeError: 'str' object has no attribute 'utcoffset'
[2024-01-09T08:18:29.771+0000] {taskinstance.py:1826} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/base.py", line 225, in execute
raise e
File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/base.py", line 212, in execute
poke_return = self.poke(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/sftp/sensors/sftp.py", line 97, in poke
_newer_than = convert_to_utc(self.newer_than)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 104, in convert_to_utc
if not is_localized(value):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 39, in is_localized
return value.utcoffset() is not None
AttributeError: 'str' object has no attribute 'utcoffset'
[2024-01-09 08:18:29,783] {taskinstance.py:1346} INFO - Marking task as FAILED. dag_id=dag_XXX, task_id=wait_for_sftp_file, execution_date=20231222T000000, start_date=, end_date=20240109T081829
[2024-01-09T08:18:29.783+0000] {taskinstance.py:1346} INFO - Marking task as FAILED. dag_id=dag_XXX, task_id=wait_for_sftp_file, execution_date=20231222T000000, start_date=, end_date=20240109T081829
Traceback (most recent call last):
File "/opt/python3.8/bin/airflow", line 8, in <module>
sys.exit(main())
File "/opt/python3.8/lib/python3.8/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/cli/cli_config.py", line 52, in command
return func(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/cli.py", line 112, in wrapper
return f(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 618, in task_test
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/session.py", line 76, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1723, in run
self._run_raw_task(
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/session.py", line 73, in wrapper
return func(*args, **kwargs)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1408, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1560, in _execute_task_with_callbacks
result = self._execute_task(context, task_orig)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1630, in _execute_task
result = execute_callable(context=context)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/base.py", line 225, in execute
raise e
File "/opt/python3.8/lib/python3.8/site-packages/airflow/sensors/base.py", line 212, in execute
poke_return = self.poke(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/sftp/sensors/sftp.py", line 97, in poke
_newer_than = convert_to_utc(self.newer_than)
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 104, in convert_to_utc
if not is_localized(value):
File "/opt/python3.8/lib/python3.8/site-packages/airflow/utils/timezone.py", line 39, in is_localized
return value.utcoffset() is not None
AttributeError: 'str' object has no attribute 'utcoffset'
```
this was with:
```python
wait_for_sftp_file = SFTPSensor(
task_id=f"wait_for_sftp_file",
sftp_conn_id="sftp_musictrace",
path=f"XXXX_YYY_KW{{{{ macros.ds_format(ds, '%Y-%m-%d', '%y%U') }}}}_ZZZZ.WWW",
newer_than='newer_than='{{ dag_run.logical_date.add(days=-1, hours=-2) }}',
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [WIP] SFTPSensor.newer_than not working with jinja logical date expression [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1880070155
For what it's worth, [the `SFTPSensor` does support templating `newer_than`](https://github.com/apache/airflow/blob/providers-sftp/4.4.0/airflow/providers/sftp/sensors/sftp.py#L47), so to @vatsrahul1001 point, I wonder if it's something to do with when the DateTime is parsed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1907128227
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1920135731
How can I request help here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SFTPSensor.newer_than not working with jinja logical ds/ts expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on issue #36629:
URL: https://github.com/apache/airflow/issues/36629#issuecomment-1883120505
I think I recreated here the same problem in unit test
https://github.com/apache/airflow/actions/runs/7459580855/job/20296049813#step:5:781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [WIP] SFTPSensor.newer_than not working with jinja logical date expression [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic closed issue #36629: [WIP] SFTPSensor.newer_than not working with jinja logical date expression
URL: https://github.com/apache/airflow/issues/36629
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org