You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/05 15:57:03 UTC
[GitHub] [airflow] ashb opened a new pull request #17451: Add date format filters to Jinja environment
ashb opened a new pull request #17451:
URL: https://github.com/apache/airflow/pull/17451
On it's own this doesn't add all that much, but this is preparatory work
to be combined with the new `data_interval_start` variables we are
adding, without having to add the ds/ts/no-dash etc permutations of all
of them.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#issuecomment-896819797
I proposed this to Ash when he was drafting this, but we decided some of the old variables are used to often (e.g. `ds` and `ds_nodash`) and should be kept. Some of the `_ds` and `_nodash` variables will be deprecated as a part of AIP-39 though, and we will not add new variables of such names to replace them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r684096786
##########
File path: airflow/templates.py
##########
@@ -30,3 +35,32 @@ def is_safe_attribute(self, obj, attr, value):
``_``) whilst still blocking internal or truely private attributes (``__`` prefixed ones).
"""
return not jinja2.sandbox.is_internal_attribute(obj, attr)
+
+
+def ds_filter(value):
+ return value.strftime('%Y-%m-%d')
+
+
+def ds_nodash_filter(value):
+ return value.strftime('%Y%m%d')
+
+
+def ts_filter(value):
+ return value.isoformat()
+
+
+def ts_nodash_filter(value):
+ return value.strftime('%Y%m%dT%H%M%S')
+
+
+def ts_nodash_with_tz_filter(value):
+ return value.isoformat().replace('-', '').replace(':', '')
+
+
+FILTERS = {
+ 'ds': ds_filter,
+ 'ds_no_dash': ds_nodash_filter,
+ 'ts': ts_filter,
+ 'ts_no_dash': ts_nodash_filter,
+ 'ts_no_dash_with_tz': ts_nodash_with_tz_filter,
Review comment:
Oh yeah, that was not an intentional change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r684097826
##########
File path: docs/apache-airflow/templates-ref.rst
##########
@@ -105,14 +105,35 @@ For example, you could use expressions in your templates like ``{{ conn.my_conn_
Just like with ``var`` it's possible to fetch a connection by string (e.g. ``{{ conn.get('my_conn_id_'+index).host }}``
) or provide defaults (e.g ``{{ conn.get('my_conn_id', {"host": "host1", "login": "user1"}).host }}``)
+Filters
+-------
+
+Airflow defines the some Jinja filters that can be used to format values.
+
+For example, using ``{{ execution_date | ds }}`` will output the execution_date in the ``YYYY-MM-DD`` format.
+
+===================== ============ ==================================================================
+Filter Operates on Description
+===================== ============ ==================================================================
+``ds`` datetime Format the datetime as ``YYYY-MM-DD``
+``ds_no_dash`` datetime Format the datetime as ``YYYYMMDD``
+``ts`` datetime Same as ``.isoformat()``, Example: ``2018-01-01T00:00:00+00:00``
+``ts_no_dash`` datetime Same as ``ts`` filter without ``-``, ``:`` or TimeZone info.
+ Example: ``20180101T000000``
+``ts_nodash_with_tz`` datetime As ``ts`` filter without ``-`` or ``:``. Example
+ ``20180101T000000+0000``
+===================== ============ ==================================================================
Review comment:
```suggestion
====================== ============ ==================================================================
Filter Operates on Description
====================== ============ ==================================================================
``ds`` datetime Format the datetime as ``YYYY-MM-DD``
``ds_no_dash`` datetime Format the datetime as ``YYYYMMDD``
``ts`` datetime Same as ``.isoformat()``, Example: ``2018-01-01T00:00:00+00:00``
``ts_no_dash`` datetime Same as ``ts`` filter without ``-``, ``:`` or TimeZone info.
Example: ``20180101T000000``
``ts_no_dash_with_tz`` datetime As ``ts`` filter without ``-`` or ``:``. Example
``20180101T000000+0000``
====================== ============ ==================================================================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r683589210
##########
File path: docs/apache-airflow/index.rst
##########
@@ -110,7 +110,7 @@ unit of work and continuity.
Operators and hooks <operators-and-hooks-ref>
CLI <cli-and-env-variables-ref>
- Macros <macros-ref>
+ Templates <templates-ref>
Review comment:
I decided to rename the page, as it was already talking about more than just the macros, but by adding filters there too it is even more true.
##########
File path: docs/apache-airflow/redirects.txt
##########
@@ -44,6 +44,7 @@ start.rst start/index.rst
cli-ref.rst cli-and-env-variables-ref.rst
_api/index.rst python-api-ref.rst
rest-api-ref.rst deprecated-rest-api-ref.rst
+macros-ref.rst templates-ref.rst
Review comment:
Old url is redirected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BasPH commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
BasPH commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r683673558
##########
File path: docs/apache-airflow/templates-ref.rst
##########
@@ -105,14 +105,35 @@ For example, you could use expressions in your templates like ``{{ conn.my_conn_
Just like with ``var`` it's possible to fetch a connection by string (e.g. ``{{ conn.get('my_conn_id_'+index).host }}``
) or provide defaults (e.g ``{{ conn.get('my_conn_id', {"host": "host1", "login": "user1"}).host }}``)
+Filters
+-------
+
+Airflow defines the some Jinja filters that can be used to format values.
+
+For example, using ``{{ execution_date | ds }}`` will output the execution_date in the ``YYYY-MM-DD`` format.
+
+===================== ============ ==================================================================
+Filter Operates on Description
+===================== ============ ==================================================================
+``ds`` datetime Format the datetime as ``YYYY-MM-DD``
+``ds_no_dash`` datetime Format the datetime as ``YYYYMMDD``
+``ts`` datetime Same as ``.isoformat()``, Example: ``2018-01-01T00:00:00+00:00``
+``ts_no_dash`` datetime Same as ``ts`` filter without ``-``, ``:`` or TimeZone info.
+ Example: ``20180101T000000``
+``ts_nodash_with_tz`` datetime As ``ts`` filter without ``-`` or ``:``. Example
Review comment:
Doesn't align with the filter name
```suggestion
``ts_no_dash_with_tz`` datetime As ``ts`` filter without ``-`` or ``:``. Example
```
##########
File path: airflow/templates.py
##########
@@ -30,3 +35,32 @@ def is_safe_attribute(self, obj, attr, value):
``_``) whilst still blocking internal or truely private attributes (``__`` prefixed ones).
"""
return not jinja2.sandbox.is_internal_attribute(obj, attr)
+
+
+def ds_filter(value):
+ return value.strftime('%Y-%m-%d')
+
+
+def ds_nodash_filter(value):
+ return value.strftime('%Y%m%d')
+
+
+def ts_filter(value):
+ return value.isoformat()
+
+
+def ts_nodash_filter(value):
+ return value.strftime('%Y%m%dT%H%M%S')
+
+
+def ts_nodash_with_tz_filter(value):
+ return value.isoformat().replace('-', '').replace(':', '')
+
+
+FILTERS = {
+ 'ds': ds_filter,
+ 'ds_no_dash': ds_nodash_filter,
+ 'ts': ts_filter,
+ 'ts_no_dash': ts_nodash_filter,
+ 'ts_no_dash_with_tz': ts_nodash_with_tz_filter,
Review comment:
Bit nitpicky, but I think keeping the "no_dash" naming equal avoids any confusion.
```suggestion
def ds_no_dash_filter(value):
return value.strftime('%Y%m%d')
def ts_filter(value):
return value.isoformat()
def ts_no_dash_filter(value):
return value.strftime('%Y%m%dT%H%M%S')
def ts_no_dash_with_tz_filter(value):
return value.isoformat().replace('-', '').replace(':', '')
FILTERS = {
'ds': ds_filter,
'ds_no_dash': ds_no_dash_filter,
'ts': ts_filter,
'ts_no_dash': ts_no_dash_filter,
'ts_no_dash_with_tz': ts_no_dash_with_tz_filter,
```
##########
File path: docs/apache-airflow/index.rst
##########
@@ -110,7 +110,7 @@ unit of work and continuity.
Operators and hooks <operators-and-hooks-ref>
CLI <cli-and-env-variables-ref>
- Macros <macros-ref>
+ Templates <templates-ref>
Review comment:
Why not name it something like "context" or "task context"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#issuecomment-894141873
:facepalm: Went the wrong way. Yes, will change to `nodash`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#issuecomment-896812175
Should we deprecate but not delete old variables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr merged pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
uranusjr merged pull request #17451:
URL: https://github.com/apache/airflow/pull/17451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#issuecomment-894334379
Hmmm all these failures _seem_ unrelated but I'm worried by just how many of them there are
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r684097826
##########
File path: docs/apache-airflow/templates-ref.rst
##########
@@ -105,14 +105,35 @@ For example, you could use expressions in your templates like ``{{ conn.my_conn_
Just like with ``var`` it's possible to fetch a connection by string (e.g. ``{{ conn.get('my_conn_id_'+index).host }}``
) or provide defaults (e.g ``{{ conn.get('my_conn_id', {"host": "host1", "login": "user1"}).host }}``)
+Filters
+-------
+
+Airflow defines the some Jinja filters that can be used to format values.
+
+For example, using ``{{ execution_date | ds }}`` will output the execution_date in the ``YYYY-MM-DD`` format.
+
+===================== ============ ==================================================================
+Filter Operates on Description
+===================== ============ ==================================================================
+``ds`` datetime Format the datetime as ``YYYY-MM-DD``
+``ds_no_dash`` datetime Format the datetime as ``YYYYMMDD``
+``ts`` datetime Same as ``.isoformat()``, Example: ``2018-01-01T00:00:00+00:00``
+``ts_no_dash`` datetime Same as ``ts`` filter without ``-``, ``:`` or TimeZone info.
+ Example: ``20180101T000000``
+``ts_nodash_with_tz`` datetime As ``ts`` filter without ``-`` or ``:``. Example
+ ``20180101T000000+0000``
+===================== ============ ==================================================================
Review comment:
```suggestion
====================== ============ ==================================================================
Filter Operates on Description
====================== ============ ==================================================================
``ds`` datetime Format the datetime as ``YYYY-MM-DD``
``ds_no_dash`` datetime Format the datetime as ``YYYYMMDD``
``ts`` datetime Same as ``.isoformat()``, Example: ``2018-01-01T00:00:00+00:00``
``ts_no_dash`` datetime Same as ``ts`` filter without ``-``, ``:`` or TimeZone info.
Example: ``20180101T000000``
``ts_nod|ash_with_tz`` datetime As ``ts`` filter without ``-`` or ``:``. Example
``20180101T000000+0000``
====================== ============ ==================================================================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#issuecomment-896819797
I proposed this to Ash when he was drafting this, but we decided some of the old variables are used to often (e.g. ds) and should be kept. Some of the `_ds` and `_nodash` variables will be deprecated as a part of AIP-39 though, and we will not add new variables of such names to replace them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r684097240
##########
File path: docs/apache-airflow/index.rst
##########
@@ -110,7 +110,7 @@ unit of work and continuity.
Operators and hooks <operators-and-hooks-ref>
CLI <cli-and-env-variables-ref>
- Macros <macros-ref>
+ Templates <templates-ref>
Review comment:
Because macros and filters are not anything to do with task context - only the variables are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#discussion_r683586341
##########
File path: airflow/macros/__init__.py
##########
@@ -15,7 +15,6 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-"""Macros."""
Review comment:
This showed up as `Macros` in the docs output, and now we don't have pylint this should be fine :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17451: Add date format filters to Jinja environment
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17451:
URL: https://github.com/apache/airflow/pull/17451#issuecomment-894145912
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org