You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Ori Popowski <or...@gmail.com> on 2020/04/29 10:39:13 UTC
Consider supporting .jinja2 extensions across the project
At the moment, various operators support reading templates from given
filenames (for example, `bash_command` of `BashOperator`). However, all of
these operators specify extensions such as .sh, .json etc., so it means
users need to write Jinja2 templates inside .sh, .json files. This is
problematic for 3 reasons:
1. No semantic support when writing templates (major IDEs support this
extension, including Github itself)
2. It's not aligned to widespread convention of naming template files with
.jinja2 extension
3. It's harder to distinguish between template files and non-template files
A better way to do it would be supporting the .jinja2 extension.
The proposition at the moment is to globally accept .jinja2 extension in
`BaseOperator` whenever `template_ext` is defined. It means that all
inheriting operators will now automatically support their already defined
extensions but also with .jinja2 appended to them (i.e. if you have
`template_ext = ('.json', '.yml')` then also .json.jinja2 and .yml.jinja2
will be added)).
Since this affects a vast part of the codebase, this suggestion is open to
discussion. WDYT?
For more info, see https://github.com/apache/airflow/issues/8603
Re: Consider supporting .jinja2 extensions across the project
Posted by Kaxil Naik <ka...@gmail.com>.
Yea I like that plan. Curious to hear what other users think?
On Wed, Apr 29, 2020 at 12:33 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> We could do this globally for 1.10, but for 2.0 I think it might be nice
> to change this as follows:
>
> Rename template_ext to file_exts, and it looks for files with those
> extensions, but doesn't render them.
>
> And then make it only render files with, for example, .sh.jinja2.
>
> Right now it's impossible to have scripts/files external to the DAG that
> aren't rendered, so you have to be careful to escape {{ in them, which
> can be annoying.
>
> WDYT?
>
> -a
>
> On Apr 29 2020, at 12:17 pm, Ori Popowski <or...@gmail.com> wrote:
>
> > No. Any extension mentioned in `template_ext` will still be templated.
> But
> > in addition, also those ending with `.jinja2`, as long it's preceded
> > by one
> > of those extensions (meaning that `file.jinja2` won't be templated)
> >
> > On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> >> Are you also proposing that, for example .sh would not be templated, but
> >> that .sh.jinja2 would be templated?
> >>
> >> Ash
> >>
> >> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
> >> >At the moment, various operators support reading templates from given
> >> >filenames (for example, `bash_command` of `BashOperator`). However, all
> >> >of
> >> >these operators specify extensions such as .sh, .json etc., so it means
> >> >users need to write Jinja2 templates inside .sh, .json files. This is
> >> >problematic for 3 reasons:
> >> >
> >> >1. No semantic support when writing templates (major IDEs support this
> >> >extension, including Github itself)
> >> >2. It's not aligned to widespread convention of naming template files
> >> >with
> >> >.jinja2 extension
> >> >3. It's harder to distinguish between template files and non-template
> >> >files
> >> >
> >> >A better way to do it would be supporting the .jinja2 extension.
> >> >
> >> >The proposition at the moment is to globally accept .jinja2 extension
> >> >in
> >> >`BaseOperator` whenever `template_ext` is defined. It means that all
> >> >inheriting operators will now automatically support their already
> >> >defined
> >> >extensions but also with .jinja2 appended to them (i.e. if you have
> >> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> >> >.yml.jinja2
> >> >will be added)).
> >> >
> >> >Since this affects a vast part of the codebase, this suggestion is open
> >> >to
> >> >discussion. WDYT?
> >> >
> >> >For more info, see https://github.com/apache/airflow/issues/8603
> >>
> >
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Kaxil Naik <ka...@gmail.com>.
Yea it can (and should) be a configurable value with `.jinja2` & `.j2` as
the default.
On Wed, Apr 29, 2020 at 5:06 PM Ori Popowski <or...@gmail.com> wrote:
> I built the code so it can easily accommodate other Jinja extensions if
> needed.
> When this issue consolidates they can be added to the source code and it
> will work automatically (although at the moment it's only .jinja2)
>
> On Wed, Apr 29, 2020 at 7:01 PM Xinbin Huang <bi...@gmail.com>
> wrote:
>
> > I like the idea of rendering files with .jinja2 extensions, and we can
> > extend it a bit to some variations that are generally recognized by
> > Plugins, e.g. .jinja, .j2.
> >
> > For VSCode, the plugin I use is Better Jinja
> > <
> >
> https://marketplace.visualstudio.com/items?itemName=samuelcolvin.jinjahtml
> > >
> > .
> >
> > Best
> > Bin
> >
> > On Wed, Apr 29, 2020 at 4:40 AM Ori Popowski <or...@gmail.com> wrote:
> >
> > > @ash I agree that in 2.0 it will be much better to _not_ render files
> not
> > > ending with .jinja2, and to render only .jinja2 files.
> > > It's much more intuitive, since it can be quite confusing for a dev to
> > see
> > > regular files being rendered.
> > >
> > > On Wed, Apr 29, 2020 at 2:33 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > > > We could do this globally for 1.10, but for 2.0 I think it might be
> > nice
> > > > to change this as follows:
> > > >
> > > > Rename template_ext to file_exts, and it looks for files with those
> > > > extensions, but doesn't render them.
> > > >
> > > > And then make it only render files with, for example, .sh.jinja2.
> > > >
> > > > Right now it's impossible to have scripts/files external to the DAG
> > that
> > > > aren't rendered, so you have to be careful to escape {{ in them,
> which
> > > > can be annoying.
> > > >
> > > > WDYT?
> > > >
> > > > -a
> > > >
> > > > On Apr 29 2020, at 12:17 pm, Ori Popowski <or...@gmail.com> wrote:
> > > >
> > > > > No. Any extension mentioned in `template_ext` will still be
> > templated.
> > > > But
> > > > > in addition, also those ending with `.jinja2`, as long it's
> preceded
> > > > > by one
> > > > > of those extensions (meaning that `file.jinja2` won't be templated)
> > > > >
> > > > > On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org>
> > > > wrote:
> > > > >
> > > > >> Are you also proposing that, for example .sh would not be
> templated,
> > > but
> > > > >> that .sh.jinja2 would be templated?
> > > > >>
> > > > >> Ash
> > > > >>
> > > > >> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com>
> > > wrote:
> > > > >> >At the moment, various operators support reading templates from
> > given
> > > > >> >filenames (for example, `bash_command` of `BashOperator`).
> However,
> > > all
> > > > >> >of
> > > > >> >these operators specify extensions such as .sh, .json etc., so it
> > > means
> > > > >> >users need to write Jinja2 templates inside .sh, .json files.
> This
> > is
> > > > >> >problematic for 3 reasons:
> > > > >> >
> > > > >> >1. No semantic support when writing templates (major IDEs support
> > > this
> > > > >> >extension, including Github itself)
> > > > >> >2. It's not aligned to widespread convention of naming template
> > files
> > > > >> >with
> > > > >> >.jinja2 extension
> > > > >> >3. It's harder to distinguish between template files and
> > non-template
> > > > >> >files
> > > > >> >
> > > > >> >A better way to do it would be supporting the .jinja2 extension.
> > > > >> >
> > > > >> >The proposition at the moment is to globally accept .jinja2
> > extension
> > > > >> >in
> > > > >> >`BaseOperator` whenever `template_ext` is defined. It means that
> > all
> > > > >> >inheriting operators will now automatically support their already
> > > > >> >defined
> > > > >> >extensions but also with .jinja2 appended to them (i.e. if you
> have
> > > > >> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> > > > >> >.yml.jinja2
> > > > >> >will be added)).
> > > > >> >
> > > > >> >Since this affects a vast part of the codebase, this suggestion
> is
> > > open
> > > > >> >to
> > > > >> >discussion. WDYT?
> > > > >> >
> > > > >> >For more info, see https://github.com/apache/airflow/issues/8603
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ori Popowski <or...@gmail.com>.
I built the code so it can easily accommodate other Jinja extensions if
needed.
When this issue consolidates they can be added to the source code and it
will work automatically (although at the moment it's only .jinja2)
On Wed, Apr 29, 2020 at 7:01 PM Xinbin Huang <bi...@gmail.com> wrote:
> I like the idea of rendering files with .jinja2 extensions, and we can
> extend it a bit to some variations that are generally recognized by
> Plugins, e.g. .jinja, .j2.
>
> For VSCode, the plugin I use is Better Jinja
> <
> https://marketplace.visualstudio.com/items?itemName=samuelcolvin.jinjahtml
> >
> .
>
> Best
> Bin
>
> On Wed, Apr 29, 2020 at 4:40 AM Ori Popowski <or...@gmail.com> wrote:
>
> > @ash I agree that in 2.0 it will be much better to _not_ render files not
> > ending with .jinja2, and to render only .jinja2 files.
> > It's much more intuitive, since it can be quite confusing for a dev to
> see
> > regular files being rendered.
> >
> > On Wed, Apr 29, 2020 at 2:33 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> > > We could do this globally for 1.10, but for 2.0 I think it might be
> nice
> > > to change this as follows:
> > >
> > > Rename template_ext to file_exts, and it looks for files with those
> > > extensions, but doesn't render them.
> > >
> > > And then make it only render files with, for example, .sh.jinja2.
> > >
> > > Right now it's impossible to have scripts/files external to the DAG
> that
> > > aren't rendered, so you have to be careful to escape {{ in them, which
> > > can be annoying.
> > >
> > > WDYT?
> > >
> > > -a
> > >
> > > On Apr 29 2020, at 12:17 pm, Ori Popowski <or...@gmail.com> wrote:
> > >
> > > > No. Any extension mentioned in `template_ext` will still be
> templated.
> > > But
> > > > in addition, also those ending with `.jinja2`, as long it's preceded
> > > > by one
> > > > of those extensions (meaning that `file.jinja2` won't be templated)
> > > >
> > > > On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org>
> > > wrote:
> > > >
> > > >> Are you also proposing that, for example .sh would not be templated,
> > but
> > > >> that .sh.jinja2 would be templated?
> > > >>
> > > >> Ash
> > > >>
> > > >> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com>
> > wrote:
> > > >> >At the moment, various operators support reading templates from
> given
> > > >> >filenames (for example, `bash_command` of `BashOperator`). However,
> > all
> > > >> >of
> > > >> >these operators specify extensions such as .sh, .json etc., so it
> > means
> > > >> >users need to write Jinja2 templates inside .sh, .json files. This
> is
> > > >> >problematic for 3 reasons:
> > > >> >
> > > >> >1. No semantic support when writing templates (major IDEs support
> > this
> > > >> >extension, including Github itself)
> > > >> >2. It's not aligned to widespread convention of naming template
> files
> > > >> >with
> > > >> >.jinja2 extension
> > > >> >3. It's harder to distinguish between template files and
> non-template
> > > >> >files
> > > >> >
> > > >> >A better way to do it would be supporting the .jinja2 extension.
> > > >> >
> > > >> >The proposition at the moment is to globally accept .jinja2
> extension
> > > >> >in
> > > >> >`BaseOperator` whenever `template_ext` is defined. It means that
> all
> > > >> >inheriting operators will now automatically support their already
> > > >> >defined
> > > >> >extensions but also with .jinja2 appended to them (i.e. if you have
> > > >> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> > > >> >.yml.jinja2
> > > >> >will be added)).
> > > >> >
> > > >> >Since this affects a vast part of the codebase, this suggestion is
> > open
> > > >> >to
> > > >> >discussion. WDYT?
> > > >> >
> > > >> >For more info, see https://github.com/apache/airflow/issues/8603
> > > >>
> > > >
> > >
> >
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Xinbin Huang <bi...@gmail.com>.
I like the idea of rendering files with .jinja2 extensions, and we can
extend it a bit to some variations that are generally recognized by
Plugins, e.g. .jinja, .j2.
For VSCode, the plugin I use is Better Jinja
<https://marketplace.visualstudio.com/items?itemName=samuelcolvin.jinjahtml>
.
Best
Bin
On Wed, Apr 29, 2020 at 4:40 AM Ori Popowski <or...@gmail.com> wrote:
> @ash I agree that in 2.0 it will be much better to _not_ render files not
> ending with .jinja2, and to render only .jinja2 files.
> It's much more intuitive, since it can be quite confusing for a dev to see
> regular files being rendered.
>
> On Wed, Apr 29, 2020 at 2:33 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > We could do this globally for 1.10, but for 2.0 I think it might be nice
> > to change this as follows:
> >
> > Rename template_ext to file_exts, and it looks for files with those
> > extensions, but doesn't render them.
> >
> > And then make it only render files with, for example, .sh.jinja2.
> >
> > Right now it's impossible to have scripts/files external to the DAG that
> > aren't rendered, so you have to be careful to escape {{ in them, which
> > can be annoying.
> >
> > WDYT?
> >
> > -a
> >
> > On Apr 29 2020, at 12:17 pm, Ori Popowski <or...@gmail.com> wrote:
> >
> > > No. Any extension mentioned in `template_ext` will still be templated.
> > But
> > > in addition, also those ending with `.jinja2`, as long it's preceded
> > > by one
> > > of those extensions (meaning that `file.jinja2` won't be templated)
> > >
> > > On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org>
> > wrote:
> > >
> > >> Are you also proposing that, for example .sh would not be templated,
> but
> > >> that .sh.jinja2 would be templated?
> > >>
> > >> Ash
> > >>
> > >> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com>
> wrote:
> > >> >At the moment, various operators support reading templates from given
> > >> >filenames (for example, `bash_command` of `BashOperator`). However,
> all
> > >> >of
> > >> >these operators specify extensions such as .sh, .json etc., so it
> means
> > >> >users need to write Jinja2 templates inside .sh, .json files. This is
> > >> >problematic for 3 reasons:
> > >> >
> > >> >1. No semantic support when writing templates (major IDEs support
> this
> > >> >extension, including Github itself)
> > >> >2. It's not aligned to widespread convention of naming template files
> > >> >with
> > >> >.jinja2 extension
> > >> >3. It's harder to distinguish between template files and non-template
> > >> >files
> > >> >
> > >> >A better way to do it would be supporting the .jinja2 extension.
> > >> >
> > >> >The proposition at the moment is to globally accept .jinja2 extension
> > >> >in
> > >> >`BaseOperator` whenever `template_ext` is defined. It means that all
> > >> >inheriting operators will now automatically support their already
> > >> >defined
> > >> >extensions but also with .jinja2 appended to them (i.e. if you have
> > >> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> > >> >.yml.jinja2
> > >> >will be added)).
> > >> >
> > >> >Since this affects a vast part of the codebase, this suggestion is
> open
> > >> >to
> > >> >discussion. WDYT?
> > >> >
> > >> >For more info, see https://github.com/apache/airflow/issues/8603
> > >>
> > >
> >
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ori Popowski <or...@gmail.com>.
@ash I agree that in 2.0 it will be much better to _not_ render files not
ending with .jinja2, and to render only .jinja2 files.
It's much more intuitive, since it can be quite confusing for a dev to see
regular files being rendered.
On Wed, Apr 29, 2020 at 2:33 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> We could do this globally for 1.10, but for 2.0 I think it might be nice
> to change this as follows:
>
> Rename template_ext to file_exts, and it looks for files with those
> extensions, but doesn't render them.
>
> And then make it only render files with, for example, .sh.jinja2.
>
> Right now it's impossible to have scripts/files external to the DAG that
> aren't rendered, so you have to be careful to escape {{ in them, which
> can be annoying.
>
> WDYT?
>
> -a
>
> On Apr 29 2020, at 12:17 pm, Ori Popowski <or...@gmail.com> wrote:
>
> > No. Any extension mentioned in `template_ext` will still be templated.
> But
> > in addition, also those ending with `.jinja2`, as long it's preceded
> > by one
> > of those extensions (meaning that `file.jinja2` won't be templated)
> >
> > On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org>
> wrote:
> >
> >> Are you also proposing that, for example .sh would not be templated, but
> >> that .sh.jinja2 would be templated?
> >>
> >> Ash
> >>
> >> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
> >> >At the moment, various operators support reading templates from given
> >> >filenames (for example, `bash_command` of `BashOperator`). However, all
> >> >of
> >> >these operators specify extensions such as .sh, .json etc., so it means
> >> >users need to write Jinja2 templates inside .sh, .json files. This is
> >> >problematic for 3 reasons:
> >> >
> >> >1. No semantic support when writing templates (major IDEs support this
> >> >extension, including Github itself)
> >> >2. It's not aligned to widespread convention of naming template files
> >> >with
> >> >.jinja2 extension
> >> >3. It's harder to distinguish between template files and non-template
> >> >files
> >> >
> >> >A better way to do it would be supporting the .jinja2 extension.
> >> >
> >> >The proposition at the moment is to globally accept .jinja2 extension
> >> >in
> >> >`BaseOperator` whenever `template_ext` is defined. It means that all
> >> >inheriting operators will now automatically support their already
> >> >defined
> >> >extensions but also with .jinja2 appended to them (i.e. if you have
> >> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> >> >.yml.jinja2
> >> >will be added)).
> >> >
> >> >Since this affects a vast part of the codebase, this suggestion is open
> >> >to
> >> >discussion. WDYT?
> >> >
> >> >For more info, see https://github.com/apache/airflow/issues/8603
> >>
> >
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ash Berlin-Taylor <as...@apache.org>.
We could do this globally for 1.10, but for 2.0 I think it might be nice
to change this as follows:
Rename template_ext to file_exts, and it looks for files with those
extensions, but doesn't render them.
And then make it only render files with, for example, .sh.jinja2.
Right now it's impossible to have scripts/files external to the DAG that
aren't rendered, so you have to be careful to escape {{ in them, which
can be annoying.
WDYT?
-a
On Apr 29 2020, at 12:17 pm, Ori Popowski <or...@gmail.com> wrote:
> No. Any extension mentioned in `template_ext` will still be templated. But
> in addition, also those ending with `.jinja2`, as long it's preceded
> by one
> of those extensions (meaning that `file.jinja2` won't be templated)
>
> On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
>> Are you also proposing that, for example .sh would not be templated, but
>> that .sh.jinja2 would be templated?
>>
>> Ash
>>
>> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
>> >At the moment, various operators support reading templates from given
>> >filenames (for example, `bash_command` of `BashOperator`). However, all
>> >of
>> >these operators specify extensions such as .sh, .json etc., so it means
>> >users need to write Jinja2 templates inside .sh, .json files. This is
>> >problematic for 3 reasons:
>> >
>> >1. No semantic support when writing templates (major IDEs support this
>> >extension, including Github itself)
>> >2. It's not aligned to widespread convention of naming template files
>> >with
>> >.jinja2 extension
>> >3. It's harder to distinguish between template files and non-template
>> >files
>> >
>> >A better way to do it would be supporting the .jinja2 extension.
>> >
>> >The proposition at the moment is to globally accept .jinja2 extension
>> >in
>> >`BaseOperator` whenever `template_ext` is defined. It means that all
>> >inheriting operators will now automatically support their already
>> >defined
>> >extensions but also with .jinja2 appended to them (i.e. if you have
>> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
>> >.yml.jinja2
>> >will be added)).
>> >
>> >Since this affects a vast part of the codebase, this suggestion is open
>> >to
>> >discussion. WDYT?
>> >
>> >For more info, see https://github.com/apache/airflow/issues/8603
>>
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ori Popowski <or...@gmail.com>.
In my PyCharm version, it came out-of-the-box with the syntax highlighting
On Wed, Apr 29, 2020 at 2:20 PM Kaxil Naik <ka...@gmail.com> wrote:
> Somewhat off-topic: What's the plugin for Pycharm/Intellij that supports
> Syntax highlighting for Jinja2 templates ?
>
> On Wed, Apr 29, 2020 at 12:17 PM Ori Popowski <or...@gmail.com> wrote:
>
>> No. Any extension mentioned in `template_ext` will still be templated. But
>> in addition, also those ending with `.jinja2`, as long it's preceded by
>> one
>> of those extensions (meaning that `file.jinja2` won't be templated)
>>
>> On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> > Are you also proposing that, for example .sh would not be templated, but
>> > that .sh.jinja2 would be templated?
>> >
>> > Ash
>> >
>> > On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
>> > >At the moment, various operators support reading templates from given
>> > >filenames (for example, `bash_command` of `BashOperator`). However, all
>> > >of
>> > >these operators specify extensions such as .sh, .json etc., so it means
>> > >users need to write Jinja2 templates inside .sh, .json files. This is
>> > >problematic for 3 reasons:
>> > >
>> > >1. No semantic support when writing templates (major IDEs support this
>> > >extension, including Github itself)
>> > >2. It's not aligned to widespread convention of naming template files
>> > >with
>> > >.jinja2 extension
>> > >3. It's harder to distinguish between template files and non-template
>> > >files
>> > >
>> > >A better way to do it would be supporting the .jinja2 extension.
>> > >
>> > >The proposition at the moment is to globally accept .jinja2 extension
>> > >in
>> > >`BaseOperator` whenever `template_ext` is defined. It means that all
>> > >inheriting operators will now automatically support their already
>> > >defined
>> > >extensions but also with .jinja2 appended to them (i.e. if you have
>> > >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
>> > >.yml.jinja2
>> > >will be added)).
>> > >
>> > >Since this affects a vast part of the codebase, this suggestion is open
>> > >to
>> > >discussion. WDYT?
>> > >
>> > >For more info, see https://github.com/apache/airflow/issues/8603
>> >
>>
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Kaxil Naik <ka...@gmail.com>.
Looks like it is only available in Pycharm Professional Edition :)
Thanks
Regards,
Kaxil
On Wed, Apr 29, 2020 at 12:30 PM Ori Popowski <or...@gmail.com> wrote:
> I'm trying to attach a screenshot
>
> [image: image.png]
>
>
> On Wed, Apr 29, 2020 at 2:20 PM Kaxil Naik <ka...@gmail.com> wrote:
>
>> Somewhat off-topic: What's the plugin for Pycharm/Intellij that supports
>> Syntax highlighting for Jinja2 templates ?
>>
>> On Wed, Apr 29, 2020 at 12:17 PM Ori Popowski <or...@gmail.com> wrote:
>>
>>> No. Any extension mentioned in `template_ext` will still be templated.
>>> But
>>> in addition, also those ending with `.jinja2`, as long it's preceded by
>>> one
>>> of those extensions (meaning that `file.jinja2` won't be templated)
>>>
>>> On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org>
>>> wrote:
>>>
>>> > Are you also proposing that, for example .sh would not be templated,
>>> but
>>> > that .sh.jinja2 would be templated?
>>> >
>>> > Ash
>>> >
>>> > On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
>>> > >At the moment, various operators support reading templates from given
>>> > >filenames (for example, `bash_command` of `BashOperator`). However,
>>> all
>>> > >of
>>> > >these operators specify extensions such as .sh, .json etc., so it
>>> means
>>> > >users need to write Jinja2 templates inside .sh, .json files. This is
>>> > >problematic for 3 reasons:
>>> > >
>>> > >1. No semantic support when writing templates (major IDEs support this
>>> > >extension, including Github itself)
>>> > >2. It's not aligned to widespread convention of naming template files
>>> > >with
>>> > >.jinja2 extension
>>> > >3. It's harder to distinguish between template files and non-template
>>> > >files
>>> > >
>>> > >A better way to do it would be supporting the .jinja2 extension.
>>> > >
>>> > >The proposition at the moment is to globally accept .jinja2 extension
>>> > >in
>>> > >`BaseOperator` whenever `template_ext` is defined. It means that all
>>> > >inheriting operators will now automatically support their already
>>> > >defined
>>> > >extensions but also with .jinja2 appended to them (i.e. if you have
>>> > >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
>>> > >.yml.jinja2
>>> > >will be added)).
>>> > >
>>> > >Since this affects a vast part of the codebase, this suggestion is
>>> open
>>> > >to
>>> > >discussion. WDYT?
>>> > >
>>> > >For more info, see https://github.com/apache/airflow/issues/8603
>>> >
>>>
>>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ori Popowski <or...@gmail.com>.
I'm trying to attach a screenshot
[image: image.png]
On Wed, Apr 29, 2020 at 2:20 PM Kaxil Naik <ka...@gmail.com> wrote:
> Somewhat off-topic: What's the plugin for Pycharm/Intellij that supports
> Syntax highlighting for Jinja2 templates ?
>
> On Wed, Apr 29, 2020 at 12:17 PM Ori Popowski <or...@gmail.com> wrote:
>
>> No. Any extension mentioned in `template_ext` will still be templated. But
>> in addition, also those ending with `.jinja2`, as long it's preceded by
>> one
>> of those extensions (meaning that `file.jinja2` won't be templated)
>>
>> On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>>
>> > Are you also proposing that, for example .sh would not be templated, but
>> > that .sh.jinja2 would be templated?
>> >
>> > Ash
>> >
>> > On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
>> > >At the moment, various operators support reading templates from given
>> > >filenames (for example, `bash_command` of `BashOperator`). However, all
>> > >of
>> > >these operators specify extensions such as .sh, .json etc., so it means
>> > >users need to write Jinja2 templates inside .sh, .json files. This is
>> > >problematic for 3 reasons:
>> > >
>> > >1. No semantic support when writing templates (major IDEs support this
>> > >extension, including Github itself)
>> > >2. It's not aligned to widespread convention of naming template files
>> > >with
>> > >.jinja2 extension
>> > >3. It's harder to distinguish between template files and non-template
>> > >files
>> > >
>> > >A better way to do it would be supporting the .jinja2 extension.
>> > >
>> > >The proposition at the moment is to globally accept .jinja2 extension
>> > >in
>> > >`BaseOperator` whenever `template_ext` is defined. It means that all
>> > >inheriting operators will now automatically support their already
>> > >defined
>> > >extensions but also with .jinja2 appended to them (i.e. if you have
>> > >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
>> > >.yml.jinja2
>> > >will be added)).
>> > >
>> > >Since this affects a vast part of the codebase, this suggestion is open
>> > >to
>> > >discussion. WDYT?
>> > >
>> > >For more info, see https://github.com/apache/airflow/issues/8603
>> >
>>
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Kaxil Naik <ka...@gmail.com>.
Somewhat off-topic: What's the plugin for Pycharm/Intellij that supports
Syntax highlighting for Jinja2 templates ?
On Wed, Apr 29, 2020 at 12:17 PM Ori Popowski <or...@gmail.com> wrote:
> No. Any extension mentioned in `template_ext` will still be templated. But
> in addition, also those ending with `.jinja2`, as long it's preceded by one
> of those extensions (meaning that `file.jinja2` won't be templated)
>
> On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org> wrote:
>
> > Are you also proposing that, for example .sh would not be templated, but
> > that .sh.jinja2 would be templated?
> >
> > Ash
> >
> > On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
> > >At the moment, various operators support reading templates from given
> > >filenames (for example, `bash_command` of `BashOperator`). However, all
> > >of
> > >these operators specify extensions such as .sh, .json etc., so it means
> > >users need to write Jinja2 templates inside .sh, .json files. This is
> > >problematic for 3 reasons:
> > >
> > >1. No semantic support when writing templates (major IDEs support this
> > >extension, including Github itself)
> > >2. It's not aligned to widespread convention of naming template files
> > >with
> > >.jinja2 extension
> > >3. It's harder to distinguish between template files and non-template
> > >files
> > >
> > >A better way to do it would be supporting the .jinja2 extension.
> > >
> > >The proposition at the moment is to globally accept .jinja2 extension
> > >in
> > >`BaseOperator` whenever `template_ext` is defined. It means that all
> > >inheriting operators will now automatically support their already
> > >defined
> > >extensions but also with .jinja2 appended to them (i.e. if you have
> > >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> > >.yml.jinja2
> > >will be added)).
> > >
> > >Since this affects a vast part of the codebase, this suggestion is open
> > >to
> > >discussion. WDYT?
> > >
> > >For more info, see https://github.com/apache/airflow/issues/8603
> >
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ori Popowski <or...@gmail.com>.
No. Any extension mentioned in `template_ext` will still be templated. But
in addition, also those ending with `.jinja2`, as long it's preceded by one
of those extensions (meaning that `file.jinja2` won't be templated)
On Wed, Apr 29, 2020 at 1:41 PM Ash Berlin-Taylor <as...@apache.org> wrote:
> Are you also proposing that, for example .sh would not be templated, but
> that .sh.jinja2 would be templated?
>
> Ash
>
> On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
> >At the moment, various operators support reading templates from given
> >filenames (for example, `bash_command` of `BashOperator`). However, all
> >of
> >these operators specify extensions such as .sh, .json etc., so it means
> >users need to write Jinja2 templates inside .sh, .json files. This is
> >problematic for 3 reasons:
> >
> >1. No semantic support when writing templates (major IDEs support this
> >extension, including Github itself)
> >2. It's not aligned to widespread convention of naming template files
> >with
> >.jinja2 extension
> >3. It's harder to distinguish between template files and non-template
> >files
> >
> >A better way to do it would be supporting the .jinja2 extension.
> >
> >The proposition at the moment is to globally accept .jinja2 extension
> >in
> >`BaseOperator` whenever `template_ext` is defined. It means that all
> >inheriting operators will now automatically support their already
> >defined
> >extensions but also with .jinja2 appended to them (i.e. if you have
> >`template_ext = ('.json', '.yml')` then also .json.jinja2 and
> >.yml.jinja2
> >will be added)).
> >
> >Since this affects a vast part of the codebase, this suggestion is open
> >to
> >discussion. WDYT?
> >
> >For more info, see https://github.com/apache/airflow/issues/8603
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Ash Berlin-Taylor <as...@apache.org>.
Are you also proposing that, for example .sh would not be templated, but that .sh.jinja2 would be templated?
Ash
On 29 April 2020 11:39:13 BST, Ori Popowski <or...@gmail.com> wrote:
>At the moment, various operators support reading templates from given
>filenames (for example, `bash_command` of `BashOperator`). However, all
>of
>these operators specify extensions such as .sh, .json etc., so it means
>users need to write Jinja2 templates inside .sh, .json files. This is
>problematic for 3 reasons:
>
>1. No semantic support when writing templates (major IDEs support this
>extension, including Github itself)
>2. It's not aligned to widespread convention of naming template files
>with
>.jinja2 extension
>3. It's harder to distinguish between template files and non-template
>files
>
>A better way to do it would be supporting the .jinja2 extension.
>
>The proposition at the moment is to globally accept .jinja2 extension
>in
>`BaseOperator` whenever `template_ext` is defined. It means that all
>inheriting operators will now automatically support their already
>defined
>extensions but also with .jinja2 appended to them (i.e. if you have
>`template_ext = ('.json', '.yml')` then also .json.jinja2 and
>.yml.jinja2
>will be added)).
>
>Since this affects a vast part of the codebase, this suggestion is open
>to
>discussion. WDYT?
>
>For more info, see https://github.com/apache/airflow/issues/8603
Re: Consider supporting .jinja2 extensions across the project
Posted by Ash Berlin-Taylor <as...@apache.org>.
We're now walking about making it read but NOT render .sh, and only
render .sh.jinja2 etc.
(See the other messages in this thread)
-ash
On Apr 30 2020, at 12:16 am, Edgar Ramírez <ed...@spoton.com> wrote:
> +1 I like the idea. Setting `template_ext=['.sh']` in an operator class
> would mean rendering both `.sh` and `.sh.jinja2` (and possibly `.j2`) files.
>
> On Wed, Apr 29, 2020 at 5:39 AM Ori Popowski <or...@gmail.com> wrote:
>
>> At the moment, various operators support reading templates from given
>> filenames (for example, `bash_command` of `BashOperator`). However,
>> all of
>> these operators specify extensions such as .sh, .json etc., so it means
>> users need to write Jinja2 templates inside .sh, .json files. This is
>> problematic for 3 reasons:
>>
>> 1. No semantic support when writing templates (major IDEs support this
>> extension, including Github itself)
>> 2. It's not aligned to widespread convention of naming template files with
>> .jinja2 extension
>> 3. It's harder to distinguish between template files and non-template files
>>
>> A better way to do it would be supporting the .jinja2 extension.
>>
>> The proposition at the moment is to globally accept .jinja2 extension in
>> `BaseOperator` whenever `template_ext` is defined. It means that all
>> inheriting operators will now automatically support their already defined
>> extensions but also with .jinja2 appended to them (i.e. if you have
>> `template_ext = ('.json', '.yml')` then also .json.jinja2 and .yml.jinja2
>> will be added)).
>>
>> Since this affects a vast part of the codebase, this suggestion is
>> open to
>> discussion. WDYT?
>>
>> For more info, see
>> https://urldefense.com/v3/__https://github.com/apache/airflow/issues/8603__;!!MeEMYEo!E7idudrJJUmicBplTDGxBMNwCy32aOW_GG0uhxVXbUqWxtes_iRwL8JpBcpz$
>>
>
>
> --
> Edgar Ramírez Mondragón
> Python Developer
> 55 3723 0202
>
Re: Consider supporting .jinja2 extensions across the project
Posted by Edgar Ramírez <ed...@spoton.com>.
+1 I like the idea. Setting `template_ext=['.sh']` in an operator class
would mean rendering both `.sh` and `.sh.jinja2` (and possibly `.j2`) files.
On Wed, Apr 29, 2020 at 5:39 AM Ori Popowski <or...@gmail.com> wrote:
> At the moment, various operators support reading templates from given
> filenames (for example, `bash_command` of `BashOperator`). However, all of
> these operators specify extensions such as .sh, .json etc., so it means
> users need to write Jinja2 templates inside .sh, .json files. This is
> problematic for 3 reasons:
>
> 1. No semantic support when writing templates (major IDEs support this
> extension, including Github itself)
> 2. It's not aligned to widespread convention of naming template files with
> .jinja2 extension
> 3. It's harder to distinguish between template files and non-template files
>
> A better way to do it would be supporting the .jinja2 extension.
>
> The proposition at the moment is to globally accept .jinja2 extension in
> `BaseOperator` whenever `template_ext` is defined. It means that all
> inheriting operators will now automatically support their already defined
> extensions but also with .jinja2 appended to them (i.e. if you have
> `template_ext = ('.json', '.yml')` then also .json.jinja2 and .yml.jinja2
> will be added)).
>
> Since this affects a vast part of the codebase, this suggestion is open to
> discussion. WDYT?
>
> For more info, see
> https://urldefense.com/v3/__https://github.com/apache/airflow/issues/8603__;!!MeEMYEo!E7idudrJJUmicBplTDGxBMNwCy32aOW_GG0uhxVXbUqWxtes_iRwL8JpBcpz$
>
--
Edgar Ramírez Mondragón
Python Developer
55 3723 0202