Re: RFC: Managing task credentials inside KubernetesExecutor

[Feng Lu <fe...@google.com.INVALID>]

Update:

The first version of task-credentials initializer, which supports GCP
service account injection,  is available now. Please feel free to check it
out here
<https://github.com/GoogleCloudPlatform/gke-serviceaccounts-initializer>.
I'll add support for AWS_ACCESS_KEY_ID and SSH private key, kindly let me
know if you have specific requirements on these credential types.

We are also implementing the corresponding KubernetesExecutor side changes
as described in the original design.

On Tue, Sep 12, 2017 at 12:15 PM, Feng Lu <fe...@google.com> wrote:

> Thank you Maxime for the confirmation, good suggestion on the use of
> policy function!
>
> On Mon, Sep 11, 2017 at 9:16 AM, Maxime Beauchemin <
> maximebeauchemin@gmail.com> wrote:
>
>> Hi,
>>
>> The proposal seems rational to me. `BaseOperator.executor_config` seems
>> like a good [new] place to put this. I'd assume that in some environments
>> there would be rules in the policy function
>> <https://airflow.incubator.apache.org/concepts.html#cluster-policy> to
>> force values in certain/all contexts.
>>
>> Max
>>
>> On Thu, Aug 31, 2017 at 10:17 PM, Feng Lu <fe...@google.com.invalid>
>> wrote:
>>
>> > Sounds great, thanks a lot for setting up the meeting and will be there.
>> >
>> > On Thu, Aug 31, 2017 at 4:10 PM, Daniel Imberman <
>> > daniel.imberman@gmail.com>
>> > wrote:
>> >
>> > > Thank you for posting this to the wiki Feng Lu :).
>> > >
>> > > I'm going to propose an overall "airflow + kubernetes update" meeting
>> in
>> > a
>> > > seperate email to discuss with the community at large. Would love it
>> if
>> > you
>> > > could discuss this further at that meeting!
>> > >
>> > > Daniel
>> > >
>> > > On Wed, Aug 30, 2017 at 10:38 PM Feng Lu <fe...@google.com.invalid>
>> > > wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > *TL;DR*
>> > > > Airflow doesn't have adequate built-in support for managing per-task
>> > > > credentials, the concept of connection helps to certain extent but
>> is
>> > not
>> > > > very satisfactory. The current Airflow KubernetesExecutor work
>> opens up
>> > > the
>> > > > possibility to handle task credentials at the framework level and
>> > > separate
>> > > > workflow business logic from credential/account management by
>> > leveraging
>> > > > the Kubernetes initializer mechanism. At the end of the day, a
>> task/dag
>> > > > only needs to specify an account name and everything else is taken
>> care
>> > > by
>> > > > the Airflow framework in a secure fashion.
>> > > >
>> > > > Detailed design:
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/
>> > > Managing+Per-task+Credentials+in+KubernetesExecutor
>> > > >
>> > > > Critics and comments are welcome :-)
>> > > > Thank you.
>> > > >
>> > > > Feng
>> > > >
>> > >
>> >
>>
>
>