You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2025/12/27 07:07:25 UTC
Re: [LAZY CONSENSUS] Not exposing sensitive data over API
Hello here,
I created an issue for it - splitting it into smaller sub-tasks:
https://github.com/apache/airflow/issues/59838
One of those is already worked on (
https://github.com/apache/airflow/pull/58659). All the issues are marked as
"good first issue" and they are up for grabs.
Those are a really good set of issues to implement as early contributions
:).
J.
On Fri, Nov 28, 2025 at 3:00 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> No - not yet, but work is on-going already - and I pointed it out in a few
> issues that target it :). But I will do some soon
>
> On Fri, Nov 28, 2025 at 6:53 AM Amogh Desai <am...@apache.org> wrote:
>
>> Nice, thanks!
>>
>> Have we already created issues / work items for this?
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Fri, Nov 21, 2025 at 4:55 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > Lay consensus has been reached.
>> >
>> > On Mon, Nov 17, 2025 at 11:07 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> > >
>> > > This is calling for a consensus on the way we treat exposing sensitive
>> > > data over API (in short "NO SENSITIVE DATA EXPOSED").
>> > >
>> > > Discussion here:
>> > >
>> > > https://lists.apache.org/thread/c79668yh42m5g7f7xck3oh6vft0z2kb6
>> > >
>> > > The consensus will be reached (unless someone objects) on Thursday
>> > > 20th of November, 2025, 23:30 CET.
>> > >
>> > > Summary:
>> > >
>> > > 1) we want to make it crystal clear that no APIs ever expose sensitive
>> > data
>> > >
>> > > 2) we should remove export (import can stay) via UI - and leave a
>> > > comment that export is only available via local CLI
>> > >
>> > > 3) the "sensitive data not exposed over API" is also present in
>> > > airflow-ctl - this means that airflow-ctl should never expose
>> > > sensitive data (including connections, variables, config, export)
>> > >
>> > > 4) the "expose config" [5] - will only accept "false" and
>> > > "non-sensitive-only". The "true" will be rejected.
>> > >
>> > > There is also an impact to local CLI, even if local CLI user has
>> > > access to all data anyway:
>> > >
>> > > 5) local CLI * list (connections, variables, config) only by default
>> > > returns "keys" - and it will only return values when `--show-values`
>> > > is passed as command line option (with clear comment in help that this
>> > > option **might** show sensitive data, also when we do `* list` command
>> > > without `--show-values` we emit stderr output explaining that
>> > > potentially sensitive data is hidden and you need to specify
>> > > `--show-values` to see them
>> > >
>> > > 6) the local CLI * get commands are unaffected (those are more likely
>> > > already used as CLI API
>> > >
>> > > 7) we remove connections list --conn-id as it is equivalent to
>> > connections get
>> > >
>> > > Again:he consensus will be reached (unless someone objects) on
>> > > Thursday 20th of November, 2025, 23:30 CET.
>> > >
>> > > J.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > For additional commands, e-mail: dev-help@airflow.apache.org
>> >
>> >
>>
>
Re: [LAZY CONSENSUS] Not exposing sensitive data over API
Posted by Amogh Desai <am...@apache.org>.
Well said.
Thanks & Regards,
Amogh Desai
On Sun, Dec 28, 2025 at 7:44 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > It's impressive how quickly the issues have been assigned and some have
> PRs as well!
>
> Indeed. Not unexpected though - I hypothesised some time ago that we will
> have more and more people interested in contributing to Open Source, people
> who will be eager to learn (and use AI to do so of course) - and that in
> general Open Source will draw more and more attention from people who seek
> mentorship and guidance in becoming more experienced engineers - and It
> somehow nicely fits that pattern.
>
> I personally think we - as maintainers - should shift more from coding and
> doing PRs ourselves, to mentoring and guiding people. Not exclusively, but
> the shift is quite inevitable IMHO.
>
> J.
>
>
> On Sun, Dec 28, 2025 at 3:04 PM Amogh Desai <am...@apache.org> wrote:
>
> > Thanks Jarek.
> >
> > It's impressive how quickly the issues have been assigned and some have
> PRs
> > as well!
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Sat, Dec 27, 2025 at 12:37 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Hello here,
> > >
> > > I created an issue for it - splitting it into smaller sub-tasks:
> > >
> > > https://github.com/apache/airflow/issues/59838
> > >
> > > One of those is already worked on (
> > > https://github.com/apache/airflow/pull/58659). All the issues are
> marked
> > > as
> > > "good first issue" and they are up for grabs.
> > >
> > > Those are a really good set of issues to implement as early
> contributions
> > > :).
> > >
> > > J.
> > >
> > >
> > >
> > > On Fri, Nov 28, 2025 at 3:00 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > > > No - not yet, but work is on-going already - and I pointed it out in
> a
> > > few
> > > > issues that target it :). But I will do some soon
> > > >
> > > > On Fri, Nov 28, 2025 at 6:53 AM Amogh Desai <am...@apache.org>
> > > wrote:
> > > >
> > > >> Nice, thanks!
> > > >>
> > > >> Have we already created issues / work items for this?
> > > >>
> > > >> Thanks & Regards,
> > > >> Amogh Desai
> > > >>
> > > >>
> > > >> On Fri, Nov 21, 2025 at 4:55 AM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >>
> > > >> > Lay consensus has been reached.
> > > >> >
> > > >> > On Mon, Nov 17, 2025 at 11:07 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > >> > >
> > > >> > > This is calling for a consensus on the way we treat exposing
> > > sensitive
> > > >> > > data over API (in short "NO SENSITIVE DATA EXPOSED").
> > > >> > >
> > > >> > > Discussion here:
> > > >> > >
> > > >> > >
> https://lists.apache.org/thread/c79668yh42m5g7f7xck3oh6vft0z2kb6
> > > >> > >
> > > >> > > The consensus will be reached (unless someone objects) on
> Thursday
> > > >> > > 20th of November, 2025, 23:30 CET.
> > > >> > >
> > > >> > > Summary:
> > > >> > >
> > > >> > > 1) we want to make it crystal clear that no APIs ever expose
> > > sensitive
> > > >> > data
> > > >> > >
> > > >> > > 2) we should remove export (import can stay) via UI - and leave
> a
> > > >> > > comment that export is only available via local CLI
> > > >> > >
> > > >> > > 3) the "sensitive data not exposed over API" is also present in
> > > >> > > airflow-ctl - this means that airflow-ctl should never expose
> > > >> > > sensitive data (including connections, variables, config,
> export)
> > > >> > >
> > > >> > > 4) the "expose config" [5] - will only accept "false" and
> > > >> > > "non-sensitive-only". The "true" will be rejected.
> > > >> > >
> > > >> > > There is also an impact to local CLI, even if local CLI user has
> > > >> > > access to all data anyway:
> > > >> > >
> > > >> > > 5) local CLI * list (connections, variables, config) only by
> > > default
> > > >> > > returns "keys" - and it will only return values when
> > `--show-values`
> > > >> > > is passed as command line option (with clear comment in help
> that
> > > this
> > > >> > > option **might** show sensitive data, also when we do `* list`
> > > command
> > > >> > > without `--show-values` we emit stderr output explaining that
> > > >> > > potentially sensitive data is hidden and you need to specify
> > > >> > > `--show-values` to see them
> > > >> > >
> > > >> > > 6) the local CLI * get commands are unaffected (those are more
> > > likely
> > > >> > > already used as CLI API
> > > >> > >
> > > >> > > 7) we remove connections list --conn-id as it is equivalent to
> > > >> > connections get
> > > >> > >
> > > >> > > Again:he consensus will be reached (unless someone objects) on
> > > >> > > Thursday 20th of November, 2025, 23:30 CET.
> > > >> > >
> > > >> > > J.
> > > >> >
> > > >> >
> > ---------------------------------------------------------------------
> > > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [LAZY CONSENSUS] Not exposing sensitive data over API
Posted by Jarek Potiuk <ja...@potiuk.com>.
> It's impressive how quickly the issues have been assigned and some have
PRs as well!
Indeed. Not unexpected though - I hypothesised some time ago that we will
have more and more people interested in contributing to Open Source, people
who will be eager to learn (and use AI to do so of course) - and that in
general Open Source will draw more and more attention from people who seek
mentorship and guidance in becoming more experienced engineers - and It
somehow nicely fits that pattern.
I personally think we - as maintainers - should shift more from coding and
doing PRs ourselves, to mentoring and guiding people. Not exclusively, but
the shift is quite inevitable IMHO.
J.
On Sun, Dec 28, 2025 at 3:04 PM Amogh Desai <am...@apache.org> wrote:
> Thanks Jarek.
>
> It's impressive how quickly the issues have been assigned and some have PRs
> as well!
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Sat, Dec 27, 2025 at 12:37 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hello here,
> >
> > I created an issue for it - splitting it into smaller sub-tasks:
> >
> > https://github.com/apache/airflow/issues/59838
> >
> > One of those is already worked on (
> > https://github.com/apache/airflow/pull/58659). All the issues are marked
> > as
> > "good first issue" and they are up for grabs.
> >
> > Those are a really good set of issues to implement as early contributions
> > :).
> >
> > J.
> >
> >
> >
> > On Fri, Nov 28, 2025 at 3:00 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > No - not yet, but work is on-going already - and I pointed it out in a
> > few
> > > issues that target it :). But I will do some soon
> > >
> > > On Fri, Nov 28, 2025 at 6:53 AM Amogh Desai <am...@apache.org>
> > wrote:
> > >
> > >> Nice, thanks!
> > >>
> > >> Have we already created issues / work items for this?
> > >>
> > >> Thanks & Regards,
> > >> Amogh Desai
> > >>
> > >>
> > >> On Fri, Nov 21, 2025 at 4:55 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >> > Lay consensus has been reached.
> > >> >
> > >> > On Mon, Nov 17, 2025 at 11:07 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > >> > >
> > >> > > This is calling for a consensus on the way we treat exposing
> > sensitive
> > >> > > data over API (in short "NO SENSITIVE DATA EXPOSED").
> > >> > >
> > >> > > Discussion here:
> > >> > >
> > >> > > https://lists.apache.org/thread/c79668yh42m5g7f7xck3oh6vft0z2kb6
> > >> > >
> > >> > > The consensus will be reached (unless someone objects) on Thursday
> > >> > > 20th of November, 2025, 23:30 CET.
> > >> > >
> > >> > > Summary:
> > >> > >
> > >> > > 1) we want to make it crystal clear that no APIs ever expose
> > sensitive
> > >> > data
> > >> > >
> > >> > > 2) we should remove export (import can stay) via UI - and leave a
> > >> > > comment that export is only available via local CLI
> > >> > >
> > >> > > 3) the "sensitive data not exposed over API" is also present in
> > >> > > airflow-ctl - this means that airflow-ctl should never expose
> > >> > > sensitive data (including connections, variables, config, export)
> > >> > >
> > >> > > 4) the "expose config" [5] - will only accept "false" and
> > >> > > "non-sensitive-only". The "true" will be rejected.
> > >> > >
> > >> > > There is also an impact to local CLI, even if local CLI user has
> > >> > > access to all data anyway:
> > >> > >
> > >> > > 5) local CLI * list (connections, variables, config) only by
> > default
> > >> > > returns "keys" - and it will only return values when
> `--show-values`
> > >> > > is passed as command line option (with clear comment in help that
> > this
> > >> > > option **might** show sensitive data, also when we do `* list`
> > command
> > >> > > without `--show-values` we emit stderr output explaining that
> > >> > > potentially sensitive data is hidden and you need to specify
> > >> > > `--show-values` to see them
> > >> > >
> > >> > > 6) the local CLI * get commands are unaffected (those are more
> > likely
> > >> > > already used as CLI API
> > >> > >
> > >> > > 7) we remove connections list --conn-id as it is equivalent to
> > >> > connections get
> > >> > >
> > >> > > Again:he consensus will be reached (unless someone objects) on
> > >> > > Thursday 20th of November, 2025, 23:30 CET.
> > >> > >
> > >> > > J.
> > >> >
> > >> >
> ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: [LAZY CONSENSUS] Not exposing sensitive data over API
Posted by Amogh Desai <am...@apache.org>.
Thanks Jarek.
It's impressive how quickly the issues have been assigned and some have PRs
as well!
Thanks & Regards,
Amogh Desai
On Sat, Dec 27, 2025 at 12:37 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello here,
>
> I created an issue for it - splitting it into smaller sub-tasks:
>
> https://github.com/apache/airflow/issues/59838
>
> One of those is already worked on (
> https://github.com/apache/airflow/pull/58659). All the issues are marked
> as
> "good first issue" and they are up for grabs.
>
> Those are a really good set of issues to implement as early contributions
> :).
>
> J.
>
>
>
> On Fri, Nov 28, 2025 at 3:00 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > No - not yet, but work is on-going already - and I pointed it out in a
> few
> > issues that target it :). But I will do some soon
> >
> > On Fri, Nov 28, 2025 at 6:53 AM Amogh Desai <am...@apache.org>
> wrote:
> >
> >> Nice, thanks!
> >>
> >> Have we already created issues / work items for this?
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >> On Fri, Nov 21, 2025 at 4:55 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > Lay consensus has been reached.
> >> >
> >> > On Mon, Nov 17, 2025 at 11:07 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> > >
> >> > > This is calling for a consensus on the way we treat exposing
> sensitive
> >> > > data over API (in short "NO SENSITIVE DATA EXPOSED").
> >> > >
> >> > > Discussion here:
> >> > >
> >> > > https://lists.apache.org/thread/c79668yh42m5g7f7xck3oh6vft0z2kb6
> >> > >
> >> > > The consensus will be reached (unless someone objects) on Thursday
> >> > > 20th of November, 2025, 23:30 CET.
> >> > >
> >> > > Summary:
> >> > >
> >> > > 1) we want to make it crystal clear that no APIs ever expose
> sensitive
> >> > data
> >> > >
> >> > > 2) we should remove export (import can stay) via UI - and leave a
> >> > > comment that export is only available via local CLI
> >> > >
> >> > > 3) the "sensitive data not exposed over API" is also present in
> >> > > airflow-ctl - this means that airflow-ctl should never expose
> >> > > sensitive data (including connections, variables, config, export)
> >> > >
> >> > > 4) the "expose config" [5] - will only accept "false" and
> >> > > "non-sensitive-only". The "true" will be rejected.
> >> > >
> >> > > There is also an impact to local CLI, even if local CLI user has
> >> > > access to all data anyway:
> >> > >
> >> > > 5) local CLI * list (connections, variables, config) only by
> default
> >> > > returns "keys" - and it will only return values when `--show-values`
> >> > > is passed as command line option (with clear comment in help that
> this
> >> > > option **might** show sensitive data, also when we do `* list`
> command
> >> > > without `--show-values` we emit stderr output explaining that
> >> > > potentially sensitive data is hidden and you need to specify
> >> > > `--show-values` to see them
> >> > >
> >> > > 6) the local CLI * get commands are unaffected (those are more
> likely
> >> > > already used as CLI API
> >> > >
> >> > > 7) we remove connections list --conn-id as it is equivalent to
> >> > connections get
> >> > >
> >> > > Again:he consensus will be reached (unless someone objects) on
> >> > > Thursday 20th of November, 2025, 23:30 CET.
> >> > >
> >> > > J.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > For additional commands, e-mail: dev-help@airflow.apache.org
> >> >
> >> >
> >>
> >
>