You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Greg Neiheisel <gr...@astronomer.io.INVALID> on 2020/03/24 13:07:23 UTC
Helm Chart for Airflow
Hey everyone,
Over the past few years at Astronomer, we’ve created, managed, and hardened
a production-ready Helm Chart for Airflow (
https://github.com/astronomer/airflow-chart) that is being used by both our
SaaS and Enterprise customers. This chart is battle-tested and running
hundreds of Airflow deployments of varying sizes and runtime environments.
It’s been built up to encapsulate the issues that Airflow users run into in
the real world.
While this chart was originally developed internally for our Astronomer
Platform, we’ve recently decoupled the chart from the rest of our platform
to make it usable by the greater Airflow community. With these changes in
mind, we want to start a conversation about donating this chart to the
Airflow community.
Some of the main features of the chart are:
- It works out of the box. With zero configuration, a user will get a
postgres database, a default user and the KubernetesExecutor ready to run
DAGs.
- Support for Local, Celery (w/ optional KEDA autoscaling) and
Kubernetes executors.
Support for optional pgbouncer. We use this to share a configurable
connection pool size per deployment. Useful for limiting connections to the
metadata database.
- Airflow migration support. A user can push a newer version of Airflow
into an existing release and migrations will automatically run cleanly.
- Prometheus support. Optionally install and configure a statsd-exporter
to ingest Airflow metrics and expose them to Prometheus automatically.
- Resource control. Optionally control the ResourceQuotas and
LimitRanges for each deployment so that no deployment can overload a
cluster.
- Simple optional Elasticsearch support.
- Optional namespace cleanup. Sometimes KubernetesExecutor and
KubernetesPodOperator pods fail for reasons other than the actual task.
This feature helps keep things clean in Kubernetes.
- Support for running locally in KIND (Kubernetes in Docker).
- Automatically tested across many Kubernetes versions with Helm 2 and 3
support.
We’ve found that the cleanest and most reliable way to deploy DAGs to
Kubernetes and manage them at scale is to package them into the actual
docker image, so we have geared this chart towards that method of
operation, though adding other methods should be straightforward.
We would love thoughts from the community and would love to see this chart
help others to get up and running on Kubernetes!
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Cool! Everything is here:
https://github.com/apache/airflow/tree/master/scripts/ci/in_container/kubernetes
In short:
In "kubernetes" dir:
setup_kind_cluster.sh + kind-cluster-conf.yaml -> they are
use to create a kind Kubernetes cluster.
In "docker" subdir:
rebuild_airflow_image.sh -> rebuilds the CI image that we use for
testing and adds bootstrap.sh and airflow-test-env-init.sh so that the CI
image can be run as pod.
In "app" sub-folder we have - potgres.yaml, secrets.yaml and volumes.yaml -
with resource definitions.
We also have there deploy_app.sh that processes the templates
(below) and deploys all the resources to the cluster. It has a bit of logic
because we have to make sure that we are pulling DAGs from the right branch
(the branch we are currently running tests on) and there is a logic to wait
until everything is deployed and responding.
In "app/templates" we have airflow.template, and configmaps.template that
are used to generate the right airlfow.yaml and configmaps.yaml that should
be deployed to Kubernetest by the deploy_app.sh
We run the tests in two different modes "git_sync" - where the DAGs are
downloaded from the branch being tested (hence the logic) and "persistent"
where the DAGs are pre-baked in the image we prepare during
"rebuild_airflow_image.sh"
My plans are to build and use the production image instead of CI one for
those tests. That would make it much faster (CI image is huge). I also plan
to move kind cluster creation outside of the container. Right now we have
docker socket mapped inside the container and we run all the "kind create"
etc. from within container - but it has some problems as we tried to test
it in Github Action (namely DNS was not properly configured in Github
Actions if we created
Kind in this way). So my plan was to allow to do both - create kind
cluster in container (useful for Breeze and local testing) or to create
Kind cluster (or simply use from outside if available).
I think as the next step it should be that for both types of tests
(persistent and git_sync) we simply use the helm chart to install it rather
than the complex template + deploy_app.sh + custom resources - they are
pretty much doing what helm chart should give us out-of-the-box.
J.
On Thu, Mar 26, 2020 at 3:20 PM Greg Neiheisel <gr...@astronomer.io> wrote:
> Yep, we can absolutely pull it into the airflow repo. We've also been
> building up a test suite that currently runs on CircleCI and uses kind
> (Kubernetes in Docker) to test several kubernetes versions with some
> different settings. Right now we're mostly testing the different executors
> since that has the biggest impact on what gets deployed, but that can be
> expanded.
>
> What CRDs are currently being used to run Airflow for the tests?
>
> -
>>
> Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
<3
On Thu, Mar 26, 2020 at 3:59 PM Daniel Imberman <da...@gmail.com>
wrote:
> @Jarek I think with the helm chart + prod image we can go even further
> than that :). We can test CeleryExecutor, with KEDA autoscaling, and a
> bunch of other configurations.
> On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> wrote:
>
> Yeah. I meant Custom Resources not CRDs in my original email :)
>
> On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> We’re not using CRDs for the tests at the moment. We just have deployment
>> files. If anything having the helm chart as a part of the airflow repo
>> could mean that the helm chart becomes the defacto system for testing
>> airflow on kubernetes (we can get rid of all the yams files and run
>> multiple k8s tests with different settings).
>> On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel <gr...@astronomer.io.invalid>,
>> wrote:
>>
>> Yep, we can absolutely pull it into the airflow repo. We've also been
>> building up a test suite that currently runs on CircleCI and uses kind
>> (Kubernetes in Docker) to test several kubernetes versions with some
>> different settings. Right now we're mostly testing the different executors
>> since that has the biggest impact on what gets deployed, but that can be
>> expanded.
>>
>> What CRDs are currently being used to run Airflow for the tests?
>>
>> On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> One thing for the donation.
>>
>> Did you you want to have separate repository Greg ?
>>
>> I think we should simply create a folder in Airflow repo and keep it
>> there (similarly as we keep Dockerfile). I am going to switch our
>> Kubernetes Tests to the production image (will make the tests much
>> faster) and I am going to test the Dockerfile automatically in CI -
>> for now we are using some custom Resource definitions to start Airflow
>> on Kubernetes Cluster for the tests, but we could switch to using the
>> helm chart - this way we can test all three things at once:
>> - Kubernetes Executor
>> - Dockerfile
>> - Helm Chart
>> and we could also add more tests - for example testing different
>> deployment options for the helm chart.
>>
>> Having the Helm chart in Airflow repo would help with that -
>> especially in terms of future changes and testing them automatically.
>>
>> J.
>>
>> On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
>> <ai...@apache.org> wrote:
>>
>>
>> +1 on the donation. Always happy to see more useful stuff for the
>> community :)
>>
>> On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io>
>>
>> wrote:
>>
>>
>> Yep, the cleanup_pods script is set up now as an optional Kubernetes
>> CronJob (
>>
>>
>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>> )
>>
>> that we have run periodically to clean failed pods up and could stay
>> separate.
>>
>> The wait_for_migrations script could definitely be pulled into Airflow.
>> For context, we deploy an initContainer on the scheduler (
>>
>>
>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>> )
>>
>> that runs the upgradedb command before booting the scheduler. This new
>> wait_for_migration script runs in an initContainer on the webserver and
>> workers (
>>
>>
>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>> )
>>
>> so that they don't boot up ahead of a potentially long-running
>>
>> migration
>>
>> and attempt to operate on new or missing columns/tables before the
>> migrations run. This prevents these pods from entering a CrashLoop.
>>
>> On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>>
>> Jarek.Potiuk@polidea.com>
>>
>> wrote:
>>
>>
>> @Tomasz great question. Our images are currently generated from
>>
>> Dockerfiles
>>
>> in this repo https://github.com/astronomer/ap-airflow and get
>>
>> published to
>>
>> DockerHub
>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>
>> For the most part those are typical Airflow images. There's an
>>
>> entrypoint
>>
>> script that we include in the image that handles waiting for the
>>
>> database
>>
>> and redis (if used) to come up, which is pretty generic.
>>
>>
>>
>> I already added waiting for the database (both metadata and celery
>>
>> URL) in
>>
>> the PR:
>>
>>
>>
>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>
>> .
>> It's functionally the same but more generic.
>>
>> The only other
>>
>> thing that I think the Helm Chart uses would be the scripts in this
>>
>> repo
>>
>> https://github.com/astronomer/astronomer-airflow-scripts. Our
>>
>> Dockerfiles
>>
>> pull this package in. These scripts are used to coordinate running
>> migrations and cleaning up failed pods.
>>
>>
>> I see two scripts:
>>
>> * cleanup_pods -> this is (I believe) not needed to run in airflow -
>>
>> this
>>
>> could be run as a separate pod/container?
>> * waiting for migrations -> I think this is a good candidate to add
>> *airflow
>> db wait_for_migration* command and make it part of airflow itself.
>>
>> I think we also have to agree on the Airflow version supported by the
>> official helm chart. I'd suggest we support 1.10.10+ and we
>>
>> incorporate
>>
>> all
>> the changes needed to airflow (like the "db wait_for_migration")
>>
>> into 2.0
>>
>> and 1.10 and we support both - image and helm chart for those versions
>> only. That would help with people migrating to the latest version.
>>
>> WDYT?
>>
>>
>> On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
>> daniel.imberman@gmail.com>
>> wrote:
>>
>> @jarek I agree completely. I think that pairing an official helm
>>
>> chart
>>
>> with the official image would make for a REALLY powerful “up and
>>
>> running
>>
>> with airflow” story :). Tomek and I have also been looking into
>> operator-sdk which has the ability to create custom controllers
>>
>> from
>>
>> helm
>>
>> charts. We might even able to get a 1-2 punch from the same code
>>
>> base
>>
>> :).
>>
>>
>> @kaxil @jarek @aizhamal @ash if there’s no issues, can we please
>>
>> start
>>
>> the
>>
>> process of donation?
>>
>> +1 on my part, of course :)
>>
>>
>>
>> Daniel
>> On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
>>
>> Jarek.Potiuk@polidea.com>,
>>
>> wrote:
>>
>> +1. And it should be paired with the official image we have
>>
>> work in
>>
>> progress on. I looked a lot at the Astronomer's image while
>>
>> preparing
>>
>> my
>>
>> draft and we can make any adjustments needed to make it works
>>
>> with
>>
>> the
>>
>> helm
>>
>> chart - and I am super happy to collaborate on that.
>>
>> PR here: https://github.com/apache/airflow/pull/7832
>>
>> J.
>>
>>
>> On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <kaxilnaik@gmail.com
>>
>>
>> wrote:
>>
>>
>> @Tomasz Urbaszek <to...@polidea.com> :
>> Helm Chart Link: https://github.com/astronomer/airflow-chart
>>
>> On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
>>
>> turbaszek@apache.org>
>>
>> wrote:
>>
>> An official helm chart is something our community needs!
>>
>> Using
>>
>> your
>>
>> chart as the official makes a lot of sens to me because as
>>
>> you
>>
>> mentioned - it's battle tested.
>>
>> One question: what Airflow image do you use? Also, would you
>>
>> mind
>>
>> sharing a link to the chart?
>>
>> Tomek
>>
>>
>> On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
>> <gr...@astronomer.io.invalid> wrote:
>>
>>
>> Hey everyone,
>>
>> Over the past few years at Astronomer, we’ve created,
>>
>> managed,
>>
>> and
>>
>> hardened
>>
>> a production-ready Helm Chart for Airflow (
>> https://github.com/astronomer/airflow-chart) that is
>>
>> being
>>
>> used
>>
>> by
>>
>> both
>>
>> our
>>
>> SaaS and Enterprise customers. This chart is
>>
>> battle-tested and
>>
>> running
>>
>> hundreds of Airflow deployments of varying sizes and
>>
>> runtime
>>
>> environments.
>>
>> It’s been built up to encapsulate the issues that Airflow
>>
>> users
>>
>> run
>>
>> into
>>
>> in
>>
>> the real world.
>>
>> While this chart was originally developed internally for
>>
>> our
>>
>> Astronomer
>>
>> Platform, we’ve recently decoupled the chart from the
>>
>> rest of
>>
>> our
>>
>> platform
>>
>> to make it usable by the greater Airflow community. With
>>
>> these
>>
>> changes
>>
>> in
>>
>> mind, we want to start a conversation about donating this
>>
>> chart
>>
>> to
>>
>> the
>>
>> Airflow community.
>>
>> Some of the main features of the chart are:
>>
>> - It works out of the box. With zero configuration, a user
>>
>> will
>>
>> get
>>
>> a
>>
>> postgres database, a default user and the
>>
>> KubernetesExecutor
>>
>> ready
>>
>> to
>>
>> run
>>
>> DAGs.
>> - Support for Local, Celery (w/ optional KEDA
>>
>> autoscaling) and
>>
>> Kubernetes executors.
>>
>> Support for optional pgbouncer. We use this to share a
>>
>> configurable
>>
>> connection pool size per deployment. Useful for limiting
>>
>> connections to
>>
>> the
>>
>> metadata database.
>>
>> - Airflow migration support. A user can push a newer
>>
>> version
>>
>> of
>>
>> Airflow
>>
>> into an existing release and migrations will
>>
>> automatically run
>>
>> cleanly.
>>
>> - Prometheus support. Optionally install and configure a
>>
>> statsd-exporter
>>
>> to ingest Airflow metrics and expose them to Prometheus
>>
>> automatically.
>>
>> - Resource control. Optionally control the ResourceQuotas
>>
>> and
>>
>> LimitRanges for each deployment so that no deployment can
>>
>> overload
>>
>> a
>>
>> cluster.
>> - Simple optional Elasticsearch support.
>> - Optional namespace cleanup. Sometimes
>>
>> KubernetesExecutor and
>>
>> KubernetesPodOperator pods fail for reasons other than the
>>
>> actual
>>
>> task.
>>
>> This feature helps keep things clean in Kubernetes.
>> - Support for running locally in KIND (Kubernetes in
>>
>> Docker).
>>
>> - Automatically tested across many Kubernetes versions
>>
>> with
>>
>> Helm
>>
>> 2
>>
>> and 3
>>
>> support.
>>
>> We’ve found that the cleanest and most reliable way to
>>
>> deploy
>>
>> DAGs
>>
>> to
>>
>> Kubernetes and manage them at scale is to package them
>>
>> into
>>
>> the
>>
>> actual
>>
>> docker image, so we have geared this chart towards that
>>
>> method of
>>
>> operation, though adding other methods should be
>>
>> straightforward.
>>
>>
>> We would love thoughts from the community and would love
>>
>> to
>>
>> see
>>
>> this
>>
>> chart
>>
>> help others to get up and running on Kubernetes!
>>
>> --
>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>
>>
>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software
>>
>> Engineer
>>
>>
>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
>>
>> <+48%20660%20796%20129>>
>>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>>
>> --
>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
>> <+48%20660%20796%20129>>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>>
>> --
>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>
>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea | Principal Software Engineer
>>
>> M: +48 660 796 129
>>
>>
>>
>> --
>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Congrats! Looking forward to start using it for tests (I am going to use it
also for some performance testing soon).
J.
On Fri, Jun 19, 2020 at 4:09 PM Greg Neiheisel <gr...@astronomer.io> wrote:
> Hey everyone,
>
> If you haven't been following along on the PR
> <https://github.com/apache/airflow/pull/8777>, it was recently merged! The
> helm chart now lives under the chart directory inside the Airflow repo.
>
> Thanks to everyone who's contributed over the years to this chart or any of
> the other open source charts. I think they all ended up pulling parts from
> each other. Happy to have a place where we can all work together on this
> together now.
>
> There's plenty of work to do still, so if you have any improvements, please
> send a PR!
>
>
>
> On Fri, May 8, 2020 at 8:10 AM Kaxil Naik <ka...@gmail.com> wrote:
>
> > Thanks, Greg 🚀
> >
> > On Fri, May 8, 2020 at 4:19 AM Greg Neiheisel <greg@astronomer.io.invalid
> >
> > wrote:
> >
> > > Awesome Jarek. I just opened up a draft PR that adds the chart into the
> > > Airflow repo. https://github.com/apache/airflow/pull/8777
> > >
> > > It still contains some of our packaging, testing and releasing
> scripts. I
> > > figure they can serve as a starting point to discuss how we'd want to
> > > replicate into Airflow infrastructure.
> > >
> > > Looking forward to getting some feedback and moving this along!
> > >
> > > On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > > > Hey Greg,
> > > >
> > > > I just wanted to update you with some recent changes I've done to
> > > automate
> > > > most of the image builds stuff we have in Airflow. That might be
> useful
> > > for
> > > > your efforts.
> > > >
> > > > I am adding (and I will fix some of them very soon) issues based on
> > > > feedback from people on the #prod-docker-image channel:
> > > >
> > >
> >
> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
> > > > But in the meantime all the build/CI infrastructure has nicely fallen
> > > into
> > > > place and It's almost complete (one thing missing but very simple is
> > > > automated builds of release images - for now this is manual but it is
> > > super
> > > > easy change and I will add it before 1.10.11 for sure).
> > > >
> > > > I described the whole build infrastructure here:
> > > > https://github.com/apache/airflow/blob/master/CI.rst including how
> > image
> > > > caching works, how nightly builds and images are triggered. I hope it
> > > will
> > > > be helpful in the helm-chart work. Any questions or asks for the
> image
> > -
> > > > please let me know, it is super easy to update the image now when we
> > have
> > > > infrastructure to build it automatically integrated well with Github
> > > > Actions and DockerHub.
> > > >
> > > > J.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com
> > >
> > > > wrote:
> > > >
> > > >> Awesome, thanks for the update!
> > > >>
> > > >> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io>
> > > >> wrote:
> > > >>
> > > >>> Hey everyone, just wanted to post a quick update on the helm chart.
> > > >>> We're in the process of removing any references to Astronomer
> > specific
> > > >>> things and making them more general features. That work is
> happening
> > on
> > > >>> this PR here https://github.com/astronomer/airflow-chart/pull/66.
> > > >>> Hoping to wrap up the last few outstanding items and open up a PR
> to
> > > >>> Airflow next week sometime!
> > > >>>
> > > >>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
> > > >>> daniel.imberman@gmail.com> wrote:
> > > >>>
> > > >>>> Hi Jarek,
> > > >>>>
> > > >>>> Congrats on getting the image out! This is going to make a lot of
> > > >>>> things easier :). We’ve actually been working on separating out
> KinD
> > > as
> > > >>>> well (and in fact moving it out of travis and into GitHub issues).
> > > Let me
> > > >>>> circle back with Steven (the engineer who’s been working on that)
> > and
> > > get a
> > > >>>> status report from him.
> > > >>>>
> > > >>>> I will also take some time today to get our helm chart to work
> with
> > > >>>> the airflow master image. It’s been a bit crazy around here in the
> > > last few
> > > >>>> weeks but I am consistently pushing to make sure we make the final
> > > changes
> > > >>>> of ripping out any remaining platform components before we donate
> > :).
> > > >>>>
> > > >>>> I agree that docker hub automation will be great. This will also
> > allow
> > > >>>> us to have a nightly build for any integration/UI tests we build.
> > > >>>>
> > > >>>> Thanks again Jarek!
> > > >>>>
> > > >>>> Daniel
> > > >>>>
> > > >>>> via Newton Mail
> > > >>>> <
> > >
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > > >
> > > >>>>
> > > >>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>> Hello Greg and Daniel,
> > > >>>>
> > > >>>> Production image is merged and it's also going to be released in
> > > >>>> 1.10.10. For now, I will manually tag the release in DockerHub
> when
> > we
> > > >>>> release but we will automate that in the future. Currently, we
> have
> > > >>>> *airflow:master* and *airflow:v1-10-test* images (for all
> supported
> > > >>>> python versions) that you can use for testing with the Helm chart!
> > > >>>>
> > > >>>> The next things from my side:
> > > >>>>
> > > >>>> * I will automate DockerHub building for the prod image from
> master
> > -
> > > >>>> for now only CI is automated but we need to add prod image as well
> > > >>>> * I will move kind-cluster creation outside of the container and
> use
> > > >>>> production image for our CI Kubernetes tests. I have a WIP branch
> on
> > > it and
> > > >>>> I think I should have it early next week
> > > >>>> * The next step will be to switch our CI to Github Actions
> finally.
> > I
> > > >>>> think we will run both Travis and GA for a short time to make sure
> > it
> > > works
> > > >>>> * I will ask people for options they would like to have in the
> > image -
> > > >>>> additional packages etc. so that I can work on the next iteration
> of
> > > the
> > > >>>> image. Also if you have any needs to see for the Helm
> integration, I
> > > am
> > > >>>> super happy to incorporate them in the next version. I plan to
> keep
> > > the
> > > >>>> "master" and "v1-10-test" images updated with any new changes and
> > push
> > > >>>> images regularly
> > > >>>> * I am happy to help with the Helm Chart, so let me know if you
> need
> > > >>>> anything!
> > > >>>>
> > > >>>> J.
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hello Daniel and Greg.
> > > >>>>>
> > > >>>>> I think the production image for Airflow is quite ready for the
> > final
> > > >>>>> review and merge. I did some tests and I think it's quite
> complete
> > > now.
> > > >>>>>
> > > >>>>> The PR is running it's Travis build:
> > > >>>>> https://github.com/apache/airflow/pull/7832
> > > >>>>>
> > > >>>>> I already pushed the image to DockerHub - so that the next time
> it
> > is
> > > >>>>> merged It will use it as a base and will be rather quickly built
> > > using
> > > >>>>> those images as a cache. Once it is merged, I will add the
> > automated
> > > build
> > > >>>>> in DockerHub and merge it to 1.10. As the next step I also plan
> to
> > > use the
> > > >>>>> prod image for our Kubernetes tests (that will help with moving
> to
> > > the
> > > >>>>> GitHub Actions from Travis).
> > > >>>>>
> > > >>>>> I added quite a comprehensive documentation - explaining all ins-
> > and
> > > >>>>> outs- of both CI and PROD images
> > > >>>>>
> > >
> >
> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
> > > >>>>> .
> > > >>>>>
> > > >>>>> You can build it yourself using manual build process described in
> > the
> > > >>>>> doc or (easier) using Breeze. That includes using the same
> > > Dockerfile to
> > > >>>>> build older Airflow version from the 1.10.* line.
> > > >>>>>
> > > >>>>> Feel free to use the new image in the Helm Chart you have - happy
> > to
> > > >>>>> review the PRs. The image is available at
> > > 'apache/airflow:master-python3.6'
> > > >>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10
> > and
> > > >>>>> release 1.10.10 we will also have it automatically published for
> > > 1.10.10
> > > >>>>> and beyond.
> > > >>>>>
> > > >>>>> Happy to hear your review comments and hope to merge it very
> soon.
> > > >>>>> J.
> > > >>>>>
> > > >>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Absolutely! Please do :). The more eyes on this the better!
> > > >>>>>>
> > > >>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
> > > >>>>>> daniel.imberman@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> @Jarek Steven and I will look into getting KinD to work on
> GitHub
> > > >>>>>>> actions today. If we can get that working, we can donate the
> helm
> > > chart and
> > > >>>>>>> possibly even deprecate the old Kubernetes tests in the
> process.
> > > >>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
> > > >>>>>>> Jarek.Potiuk@polidea.com>, wrote:
> > > >>>>>>> > We have A POC for Github Actions done by Tomek, and the
> blocker
> > > >>>>>>> there
> > > >>>>>>> > was precisely Kubernetes tests and specifically running kind
> > > >>>>>>> cluster
> > > >>>>>>> > in the Github Actions
> > > >>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
> > > >>>>>>> meantime
> > > >>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed
> this
> > > very
> > > >>>>>>> > thing today at slack:
> > > >>>>>>> >
> > > >>>>>>>
> > > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
> > > >>>>>>> >
> > > >>>>>>> > For me this is the next thing I will look at after merging
> > > >>>>>>> > requirements PR and Prod image PR (and those two are a bit
> > > >>>>>>> > pre-requisites to make it stable and fast). I want to switch
> to
> > > the
> > > >>>>>>> > Production Image to run Kubernetes tests (they will be much
> > > faster)
> > > >>>>>>> > and then we can work out much faster how to run them on
> Github
> > > >>>>>>> > Actions. Then we can add Helm charts to run all those
> different
> > > >>>>>>> > deployments. I am really looking forward to it!
> > > >>>>>>> >
> > > >>>>>>> > I hope - this weekend I move forward all of that and have a
> PR
> > > >>>>>>> that we
> > > >>>>>>> > can start running tests on and replacing the resources with
> > helm
> > > >>>>>>> > chart.
> > > >>>>>>> >
> > > >>>>>>> > J.
> > > >>>>>>> >
> > > >>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> > > >>>>>>> > <da...@gmail.com> wrote:
> > > >>>>>>> > >
> > > >>>>>>> > > @Steven glad to help you out on that.
> > > >>>>>>> > >
> > > >>>>>>> > > For the moment we only use travis, but I believe @jarek has
> > > been
> > > >>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our
> > > first
> > > >>>>>>> GitHub tests?
> > > >>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
> > > >>>>>>> <st...@astronomer.io.invalid>, wrote:
> > > >>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working
> to
> > > >>>>>>> test it. Where
> > > >>>>>>> > > > do we want to run it? Do you all just use Travis only?
> > > >>>>>>> > > >
> > > >>>>>>> > > > Steven
> > > >>>>>>> > > >
> > > >>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
> > > >>>>>>> daniel.imberman@gmail.com>
> > > >>>>>>> > > > wrote:
> > > >>>>>>> > > >
> > > >>>>>>> > > > > @Jarek I think with the helm chart + prod image we can
> go
> > > >>>>>>> even further
> > > >>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
> > > >>>>>>> autoscaling, and a
> > > >>>>>>> > > > > bunch of other configurations.
> > > >>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
> > > >>>>>>> Jarek.Potiuk@polidea.com>,
> > > >>>>>>> > > > > wrote:
> > > >>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my
> original
> > > >>>>>>> email :)
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > > >>>>>>> > > > > daniel.imberman@gmail.com> wrote:
> > > >>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment.
> > We
> > > >>>>>>> just have
> > > >>>>>>> > > > > deployment files. If anything having the helm chart as
> a
> > > >>>>>>> part of the
> > > >>>>>>> > > > > airflow repo could mean that the helm chart becomes the
> > > >>>>>>> defacto system for
> > > >>>>>>> > > > > testing airflow on kubernetes (we can get rid of all
> the
> > > >>>>>>> yams files and run
> > > >>>>>>> > > > > multiple k8s tests with different settings).
> > > >>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > > >>>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
> > > >>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow
> > > >>>>>>> repo. We've also
> > > >>>>>>> > > > > been
> > > >>>>>>> > > > > > > > > building up a test suite that currently runs on
> > > >>>>>>> CircleCI and uses
> > > >>>>>>> > > > > kind
> > > >>>>>>> > > > > > > > > (Kubernetes in Docker) to test several
> kubernetes
> > > >>>>>>> versions with
> > > >>>>>>> > > > > some
> > > >>>>>>> > > > > > > > > different settings. Right now we're mostly
> > testing
> > > >>>>>>> the different
> > > >>>>>>> > > > > executors
> > > >>>>>>> > > > > > > > > since that has the biggest impact on what gets
> > > >>>>>>> deployed, but that
> > > >>>>>>> > > > > can be
> > > >>>>>>> > > > > > > > > expanded.
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > What CRDs are currently being used to run
> Airflow
> > > >>>>>>> for the tests?
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > > >>>>>>> > > > > Jarek.Potiuk@polidea.com>
> > > >>>>>>> > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > > One thing for the donation.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > Did you you want to have separate repository
> > > Greg ?
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > I think we should simply create a folder in
> > > >>>>>>> Airflow repo and
> > > >>>>>>> > > > > keep it
> > > >>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am
> > > >>>>>>> going to switch our
> > > >>>>>>> > > > > > > > > > Kubernetes Tests to the production image
> (will
> > > >>>>>>> make the tests
> > > >>>>>>> > > > > much
> > > >>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
> > > >>>>>>> automatically in
> > > >>>>>>> > > > > CI -
> > > >>>>>>> > > > > > > > > > for now we are using some custom Resource
> > > >>>>>>> definitions to start
> > > >>>>>>> > > > > Airflow
> > > >>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we
> > could
> > > >>>>>>> switch to
> > > >>>>>>> > > > > using the
> > > >>>>>>> > > > > > > > > > helm chart - this way we can test all three
> > > things
> > > >>>>>>> at once:
> > > >>>>>>> > > > > > > > > > - Kubernetes Executor
> > > >>>>>>> > > > > > > > > > - Dockerfile
> > > >>>>>>> > > > > > > > > > - Helm Chart
> > > >>>>>>> > > > > > > > > > and we could also add more tests - for
> example
> > > >>>>>>> testing different
> > > >>>>>>> > > > > > > > > > deployment options for the helm chart.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would
> > help
> > > >>>>>>> with that -
> > > >>>>>>> > > > > > > > > > especially in terms of future changes and
> > testing
> > > >>>>>>> them
> > > >>>>>>> > > > > automatically.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > J.
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal
> > Nurmamat
> > > >>>>>>> kyzy
> > > >>>>>>> > > > > > > > > > <ai...@apache.org> wrote:
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see
> more
> > > >>>>>>> useful stuff for
> > > >>>>>>> > > > > the
> > > >>>>>>> > > > > > > > > > > community :)
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg
> > Neiheisel
> > > <
> > > >>>>>>> > > > > greg@astronomer.io>
> > > >>>>>>> > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up
> now
> > as
> > > >>>>>>> an optional
> > > >>>>>>> > > > > Kubernetes
> > > >>>>>>> > > > > > > > > > > > CronJob (
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > >>>>>>> > > > > > > > > > )
> > > >>>>>>> > > > > > > > > > > > that we have run periodically to clean
> > failed
> > > >>>>>>> pods up and
> > > >>>>>>> > > > > could stay
> > > >>>>>>> > > > > > > > > > > > separate.
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > The wait_for_migrations script could
> > > >>>>>>> definitely be pulled
> > > >>>>>>> > > > > into Airflow.
> > > >>>>>>> > > > > > > > > > > > For context, we deploy an initContainer
> on
> > > the
> > > >>>>>>> scheduler (
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > >>>>>>> > > > > > > > > > )
> > > >>>>>>> > > > > > > > > > > > that runs the upgradedb command before
> > > booting
> > > >>>>>>> the
> > > >>>>>>> > > > > scheduler. This new
> > > >>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
> > > >>>>>>> initContainer on the
> > > >>>>>>> > > > > webserver and
> > > >>>>>>> > > > > > > > > > > > workers (
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > >>>>>>> > > > > > > > > > )
> > > >>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
> > > >>>>>>> potentially
> > > >>>>>>> > > > > long-running
> > > >>>>>>> > > > > > > > > > migration
> > > >>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
> > > >>>>>>> columns/tables
> > > >>>>>>> > > > > before the
> > > >>>>>>> > > > > > > > > > > > migrations run. This prevents these pods
> > from
> > > >>>>>>> entering a
> > > >>>>>>> > > > > CrashLoop.
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek
> > > Potiuk <
> > > >>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
> > > >>>>>>> > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images
> are
> > > >>>>>>> currently
> > > >>>>>>> > > > > generated from
> > > >>>>>>> > > > > > > > > > > > > Dockerfiles
> > > >>>>>>> > > > > > > > > > > > > > in this repo
> > > >>>>>>> https://github.com/astronomer/ap-airflow
> > > >>>>>>> > > > > and get
> > > >>>>>>> > > > > > > > > > > > > published to
> > > >>>>>>> > > > > > > > > > > > > > DockerHub
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > For the most part those are typical
> > > >>>>>>> Airflow images.
> > > >>>>>>> > > > > There's an
> > > >>>>>>> > > > > > > > > > > > > entrypoint
> > > >>>>>>> > > > > > > > > > > > > > script that we include in the image
> > that
> > > >>>>>>> handles waiting
> > > >>>>>>> > > > > for the
> > > >>>>>>> > > > > > > > > > > > > database
> > > >>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which
> > is
> > > >>>>>>> pretty generic.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > I already added waiting for the
> database
> > > >>>>>>> (both metadata
> > > >>>>>>> > > > > and celery
> > > >>>>>>> > > > > > > > > > URL) in
> > > >>>>>>> > > > > > > > > > > > > the PR:
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>>
> > >
> >
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > >>>>>>> > > > > > > > > > > > > .
> > > >>>>>>> > > > > > > > > > > > > It's functionally the same but more
> > > generic.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > The only other
> > > >>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart
> uses
> > > >>>>>>> would be the
> > > >>>>>>> > > > > scripts in this
> > > >>>>>>> > > > > > > > > > repo
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
> > > >>>>>>> > > > > Our
> > > >>>>>>> > > > > > > > > > > > > Dockerfiles
> > > >>>>>>> > > > > > > > > > > > > > pull this package in. These scripts
> are
> > > >>>>>>> used to
> > > >>>>>>> > > > > coordinate running
> > > >>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed
> pods.
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > I see two scripts:
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe)
> not
> > > >>>>>>> needed to run in
> > > >>>>>>> > > > > airflow -
> > > >>>>>>> > > > > > > > > > this
> > > >>>>>>> > > > > > > > > > > > > could be run as a separate
> pod/container?
> > > >>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think
> this
> > is
> > > >>>>>>> a good
> > > >>>>>>> > > > > candidate to add
> > > >>>>>>> > > > > > > > > > > > > *airflow
> > > >>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make
> > it
> > > >>>>>>> part of airflow
> > > >>>>>>> > > > > itself.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > I think we also have to agree on the
> > > Airflow
> > > >>>>>>> version
> > > >>>>>>> > > > > supported by the
> > > >>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we
> > support
> > > >>>>>>> 1.10.10+ and we
> > > >>>>>>> > > > > > > > > > incorporate
> > > >>>>>>> > > > > > > > > > > > > all
> > > >>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the
> > "db
> > > >>>>>>> > > > > wait_for_migration")
> > > >>>>>>> > > > > > > > > > into 2.0
> > > >>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image
> and
> > > >>>>>>> helm chart for
> > > >>>>>>> > > > > those versions
> > > >>>>>>> > > > > > > > > > > > > only. That would help with people
> > migrating
> > > >>>>>>> to the latest
> > > >>>>>>> > > > > version.
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > WDYT?
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM
> Daniel
> > > >>>>>>> Imberman <
> > > >>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
> > > >>>>>>> > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think
> > that
> > > >>>>>>> pairing an
> > > >>>>>>> > > > > official helm
> > > >>>>>>> > > > > > > > > > chart
> > > >>>>>>> > > > > > > > > > > > > > > with the official image would make
> > for
> > > a
> > > >>>>>>> REALLY
> > > >>>>>>> > > > > powerful “up and
> > > >>>>>>> > > > > > > > > > > > > running
> > > >>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I
> > > have
> > > >>>>>>> also been
> > > >>>>>>> > > > > looking into
> > > >>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability
> to
> > > >>>>>>> create custom
> > > >>>>>>> > > > > controllers
> > > >>>>>>> > > > > > > > > > from
> > > >>>>>>> > > > > > > > > > > > > helm
> > > >>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a
> > 1-2
> > > >>>>>>> punch from the
> > > >>>>>>> > > > > same code
> > > >>>>>>> > > > > > > > > > base
> > > >>>>>>> > > > > > > > > > > > > :).
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if
> > there’s
> > > >>>>>>> no issues, can
> > > >>>>>>> > > > > we please
> > > >>>>>>> > > > > > > > > > start
> > > >>>>>>> > > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > process of donation?
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > Daniel
> > > >>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700,
> Jarek
> > > >>>>>>> Potiuk <
> > > >>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > >>>>>>> > > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with
> > the
> > > >>>>>>> official image
> > > >>>>>>> > > > > we have
> > > >>>>>>> > > > > > > > > > work in
> > > >>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at
> the
> > > >>>>>>> Astronomer's
> > > >>>>>>> > > > > image while
> > > >>>>>>> > > > > > > > > > > > > preparing
> > > >>>>>>> > > > > > > > > > > > > > my
> > > >>>>>>> > > > > > > > > > > > > > > > draft and we can make any
> > adjustments
> > > >>>>>>> needed to make
> > > >>>>>>> > > > > it works
> > > >>>>>>> > > > > > > > > > with
> > > >>>>>>> > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > helm
> > > >>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
> > > >>>>>>> collaborate on that.
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > PR here:
> > > >>>>>>> https://github.com/apache/airflow/pull/7832
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > J.
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM
> > Kaxil
> > > >>>>>>> Naik <
> > > >>>>>>> > > > > kaxilnaik@gmail.com
> > > >>>>>>> > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
> > > >>>>>>> tomasz.urbaszek@polidea.com> :
> > > >>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
> > > >>>>>>> > > > > https://github.com/astronomer/airflow-chart
> > > >>>>>>> > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
> > > >>>>>>> Tomasz Urbaszek <
> > > >>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
> > > >>>>>>> > > > > > > > > > > > > > > > > wrote:
> > > >>>>>>> > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is
> > > >>>>>>> something our
> > > >>>>>>> > > > > community needs!
> > > >>>>>>> > > > > > > > > > Using
> > > >>>>>>> > > > > > > > > > > > > your
> > > >>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a
> > lot
> > > >>>>>>> of sens to me
> > > >>>>>>> > > > > because as
> > > >>>>>>> > > > > > > > > > you
> > > >>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle
> tested.
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow
> > image
> > > >>>>>>> do you use?
> > > >>>>>>> > > > > Also, would you
> > > >>>>>>> > > > > > > > > > > > > mind
> > > >>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > Tomek
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07
> PM
> > > >>>>>>> Greg Neiheisel
> > > >>>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid>
> > > >>>>>>> wrote:
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
> > > >>>>>>> Astronomer, we’ve
> > > >>>>>>> > > > > created,
> > > >>>>>>> > > > > > > > > > managed,
> > > >>>>>>> > > > > > > > > > > > > > and
> > > >>>>>>> > > > > > > > > > > > > > > > > > hardened
> > > >>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm
> Chart
> > > >>>>>>> for Airflow (
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> https://github.com/astronomer/airflow-chart)
> > > >>>>>>> > > > > that is
> > > >>>>>>> > > > > > > > > > being
> > > >>>>>>> > > > > > > > > > > > > used
> > > >>>>>>> > > > > > > > > > > > > > by
> > > >>>>>>> > > > > > > > > > > > > > > > > both
> > > >>>>>>> > > > > > > > > > > > > > > > > > our
> > > >>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise
> > customers.
> > > >>>>>>> This chart is
> > > >>>>>>> > > > > > > > > > battle-tested and
> > > >>>>>>> > > > > > > > > > > > > > > running
> > > >>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow
> > deployments
> > > >>>>>>> of varying
> > > >>>>>>> > > > > sizes and
> > > >>>>>>> > > > > > > > > > runtime
> > > >>>>>>> > > > > > > > > > > > > > > > > > environments.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to
> > > >>>>>>> encapsulate the issues
> > > >>>>>>> > > > > that Airflow
> > > >>>>>>> > > > > > > > > > > > > users
> > > >>>>>>> > > > > > > > > > > > > > run
> > > >>>>>>> > > > > > > > > > > > > > > > > into
> > > >>>>>>> > > > > > > > > > > > > > > > > > in
> > > >>>>>>> > > > > > > > > > > > > > > > > > > the real world.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > While this chart was
> > originally
> > > >>>>>>> developed
> > > >>>>>>> > > > > internally for
> > > >>>>>>> > > > > > > > > > our
> > > >>>>>>> > > > > > > > > > > > > > > Astronomer
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
> > > >>>>>>> decoupled the chart
> > > >>>>>>> > > > > from the
> > > >>>>>>> > > > > > > > > > rest of
> > > >>>>>>> > > > > > > > > > > > > our
> > > >>>>>>> > > > > > > > > > > > > > > > > > platform
> > > >>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the
> > > greater
> > > >>>>>>> Airflow
> > > >>>>>>> > > > > community. With
> > > >>>>>>> > > > > > > > > > these
> > > >>>>>>> > > > > > > > > > > > > > > changes
> > > >>>>>>> > > > > > > > > > > > > > > > > in
> > > >>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
> > > >>>>>>> conversation about
> > > >>>>>>> > > > > donating this
> > > >>>>>>> > > > > > > > > > > > > chart
> > > >>>>>>> > > > > > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features
> of
> > > the
> > > >>>>>>> chart are:
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box.
> > With
> > > >>>>>>> zero
> > > >>>>>>> > > > > configuration, a user
> > > >>>>>>> > > > > > > > > > > > > will
> > > >>>>>>> > > > > > > > > > > > > > get
> > > >>>>>>> > > > > > > > > > > > > > > > > a
> > > >>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a
> default
> > > >>>>>>> user and the
> > > >>>>>>> > > > > > > > > > KubernetesExecutor
> > > >>>>>>> > > > > > > > > > > > > > ready
> > > >>>>>>> > > > > > > > > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > > > > > > run
> > > >>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery
> > (w/
> > > >>>>>>> optional KEDA
> > > >>>>>>> > > > > > > > > > autoscaling) and
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Support for optional
> > pgbouncer.
> > > >>>>>>> We use this to
> > > >>>>>>> > > > > share a
> > > >>>>>>> > > > > > > > > > > > > > configurable
> > > >>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
> > > >>>>>>> deployment. Useful
> > > >>>>>>> > > > > for limiting
> > > >>>>>>> > > > > > > > > > > > > > > connections to
> > > >>>>>>> > > > > > > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration
> support.
> > A
> > > >>>>>>> user can push a
> > > >>>>>>> > > > > newer
> > > >>>>>>> > > > > > > > > > version
> > > >>>>>>> > > > > > > > > > > > > of
> > > >>>>>>> > > > > > > > > > > > > > > > > > Airflow
> > > >>>>>>> > > > > > > > > > > > > > > > > > > into an existing release
> and
> > > >>>>>>> migrations will
> > > >>>>>>> > > > > > > > > > automatically run
> > > >>>>>>> > > > > > > > > > > > > > > > > > cleanly.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support.
> > > Optionally
> > > >>>>>>> install and
> > > >>>>>>> > > > > configure a
> > > >>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
> > > >>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics
> and
> > > >>>>>>> expose them to
> > > >>>>>>> > > > > Prometheus
> > > >>>>>>> > > > > > > > > > > > > > > > > automatically.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Resource control.
> > Optionally
> > > >>>>>>> control the
> > > >>>>>>> > > > > ResourceQuotas
> > > >>>>>>> > > > > > > > > > and
> > > >>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each
> > deployment
> > > >>>>>>> so that no
> > > >>>>>>> > > > > deployment can
> > > >>>>>>> > > > > > > > > > > > > > overload
> > > >>>>>>> > > > > > > > > > > > > > > a
> > > >>>>>>> > > > > > > > > > > > > > > > > > > cluster.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional
> > Elasticsearch
> > > >>>>>>> support.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace
> cleanup.
> > > >>>>>>> Sometimes
> > > >>>>>>> > > > > > > > > > KubernetesExecutor and
> > > >>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods
> > fail
> > > >>>>>>> for reasons
> > > >>>>>>> > > > > other than the
> > > >>>>>>> > > > > > > > > > > > > actual
> > > >>>>>>> > > > > > > > > > > > > > > > > > task.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep
> > things
> > > >>>>>>> clean in
> > > >>>>>>> > > > > Kubernetes.
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for running
> locally
> > > in
> > > >>>>>>> KIND
> > > >>>>>>> > > > > (Kubernetes in
> > > >>>>>>> > > > > > > > > > Docker).
> > > >>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested
> across
> > > >>>>>>> many Kubernetes
> > > >>>>>>> > > > > versions
> > > >>>>>>> > > > > > > > > > with
> > > >>>>>>> > > > > > > > > > > > > Helm
> > > >>>>>>> > > > > > > > > > > > > > 2
> > > >>>>>>> > > > > > > > > > > > > > > > > > and 3
> > > >>>>>>> > > > > > > > > > > > > > > > > > > support.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the
> cleanest
> > > >>>>>>> and most
> > > >>>>>>> > > > > reliable way to
> > > >>>>>>> > > > > > > > > > deploy
> > > >>>>>>> > > > > > > > > > > > > > DAGs
> > > >>>>>>> > > > > > > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them
> at
> > > >>>>>>> scale is to
> > > >>>>>>> > > > > package them
> > > >>>>>>> > > > > > > > > > into
> > > >>>>>>> > > > > > > > > > > > > the
> > > >>>>>>> > > > > > > > > > > > > > > actual
> > > >>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have
> > geared
> > > >>>>>>> this chart
> > > >>>>>>> > > > > towards that
> > > >>>>>>> > > > > > > > > > > > > method of
> > > >>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding
> > other
> > > >>>>>>> methods should
> > > >>>>>>> > > > > be
> > > >>>>>>> > > > > > > > > > > > > straightforward.
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from
> > the
> > > >>>>>>> community and
> > > >>>>>>> > > > > would love
> > > >>>>>>> > > > > > > > > > to
> > > >>>>>>> > > > > > > > > > > > > see
> > > >>>>>>> > > > > > > > > > > > > > > this
> > > >>>>>>> > > > > > > > > > > > > > > > > > chart
> > > >>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and
> > > >>>>>>> running on
> > > >>>>>>> > > > > Kubernetes!
> > > >>>>>>> > > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief
> > > >>>>>>> Architect
> > > >>>>>>> > > > > Astronomer.io
> > > >>>>>>> > > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > > > > > > > > > > > Polidea <
> https://www.polidea.com/>
> > |
> > > >>>>>>> Principal
> > > >>>>>>> > > > > Software
> > > >>>>>>> > > > > > > > > > Engineer
> > > >>>>>>> > > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
> > > >>>>>>> <+48%20660%20796%20129>
> > > >>>>>>> > > > > <+48660796129
> > > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > > >>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
> > > >>>>>>> https://www.polidea.com/>
> > > >>>>>>> > > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > >>>>>>> Astronomer.io
> > > >>>>>>> > > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
> > > >>>>>>> Principal Software
> > > >>>>>>> > > > > Engineer
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > > M: +48 660 796 129
> > <+48%20660%20796%20129>
> > > >>>>>>> <+48660796129
> > > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > > >>>>>>> > > > > > > > > > > > > [image: Polidea] <
> > https://www.polidea.com/
> > > >
> > > >>>>>>> > > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > > > > --
> > > >>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > >>>>>>> Astronomer.io
> > > >>>>>>> > > > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > --
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > > > M: +48 660 796 129
> > > >>>>>>> > > > > > > > > >
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > >
> > > >>>>>>> > > > > > > > > --
> > > >>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect
> Astronomer.io
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > > > --
> > > >>>>>>> > > > > > Jarek Potiuk
> > > >>>>>>> > > > > > Polidea | Principal Software Engineer
> > > >>>>>>> > > > > > M: +48 660 796 129
> > > >>>>>>> > > > > >
> > > >>>>>>> > > > >
> > > >>>>>>> >
> > > >>>>>>> >
> > > >>>>>>> >
> > > >>>>>>> > --
> > > >>>>>>> >
> > > >>>>>>> > Jarek Potiuk
> > > >>>>>>> > Polidea | Principal Software Engineer
> > > >>>>>>> >
> > > >>>>>>> > M: +48 660 796 129
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>>
> > > >>>>>> Jarek Potiuk
> > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > >>>>>>
> > > >>>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>>
> > > >>>>> Jarek Potiuk
> > > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>>>>
> > > >>>>> M: +48 660 796 129 <+48660796129>
> > > >>>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>>
> > > >>>> Jarek Potiuk
> > > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>>>
> > > >>>> M: +48 660 796 129 <+48660796129>
> > > >>>> [image: Polidea] <https://www.polidea.com/>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>> --
> > > >>> *Greg Neiheisel* / Chief Architect Astronomer.io
> > > >>>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Jarek Potiuk
> > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >>
> > > >> M: +48 660 796 129 <+48660796129>
> > > >> [image: Polidea] <https://www.polidea.com/>
> > > >>
> > > >>
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > > >
> > > >
> > >
> > > --
> > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >
> >
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Greg Neiheisel <gr...@astronomer.io>.
Hey everyone,
If you haven't been following along on the PR
<https://github.com/apache/airflow/pull/8777>, it was recently merged! The
helm chart now lives under the chart directory inside the Airflow repo.
Thanks to everyone who's contributed over the years to this chart or any of
the other open source charts. I think they all ended up pulling parts from
each other. Happy to have a place where we can all work together on this
together now.
There's plenty of work to do still, so if you have any improvements, please
send a PR!
On Fri, May 8, 2020 at 8:10 AM Kaxil Naik <ka...@gmail.com> wrote:
> Thanks, Greg 🚀
>
> On Fri, May 8, 2020 at 4:19 AM Greg Neiheisel <gr...@astronomer.io.invalid>
> wrote:
>
> > Awesome Jarek. I just opened up a draft PR that adds the chart into the
> > Airflow repo. https://github.com/apache/airflow/pull/8777
> >
> > It still contains some of our packaging, testing and releasing scripts. I
> > figure they can serve as a starting point to discuss how we'd want to
> > replicate into Airflow infrastructure.
> >
> > Looking forward to getting some feedback and moving this along!
> >
> > On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> > > Hey Greg,
> > >
> > > I just wanted to update you with some recent changes I've done to
> > automate
> > > most of the image builds stuff we have in Airflow. That might be useful
> > for
> > > your efforts.
> > >
> > > I am adding (and I will fix some of them very soon) issues based on
> > > feedback from people on the #prod-docker-image channel:
> > >
> >
> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
> > > But in the meantime all the build/CI infrastructure has nicely fallen
> > into
> > > place and It's almost complete (one thing missing but very simple is
> > > automated builds of release images - for now this is manual but it is
> > super
> > > easy change and I will add it before 1.10.11 for sure).
> > >
> > > I described the whole build infrastructure here:
> > > https://github.com/apache/airflow/blob/master/CI.rst including how
> image
> > > caching works, how nightly builds and images are triggered. I hope it
> > will
> > > be helpful in the helm-chart work. Any questions or asks for the image
> -
> > > please let me know, it is super easy to update the image now when we
> have
> > > infrastructure to build it automatically integrated well with Github
> > > Actions and DockerHub.
> > >
> > > J.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <Jarek.Potiuk@polidea.com
> >
> > > wrote:
> > >
> > >> Awesome, thanks for the update!
> > >>
> > >> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io>
> > >> wrote:
> > >>
> > >>> Hey everyone, just wanted to post a quick update on the helm chart.
> > >>> We're in the process of removing any references to Astronomer
> specific
> > >>> things and making them more general features. That work is happening
> on
> > >>> this PR here https://github.com/astronomer/airflow-chart/pull/66.
> > >>> Hoping to wrap up the last few outstanding items and open up a PR to
> > >>> Airflow next week sometime!
> > >>>
> > >>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
> > >>> daniel.imberman@gmail.com> wrote:
> > >>>
> > >>>> Hi Jarek,
> > >>>>
> > >>>> Congrats on getting the image out! This is going to make a lot of
> > >>>> things easier :). We’ve actually been working on separating out KinD
> > as
> > >>>> well (and in fact moving it out of travis and into GitHub issues).
> > Let me
> > >>>> circle back with Steven (the engineer who’s been working on that)
> and
> > get a
> > >>>> status report from him.
> > >>>>
> > >>>> I will also take some time today to get our helm chart to work with
> > >>>> the airflow master image. It’s been a bit crazy around here in the
> > last few
> > >>>> weeks but I am consistently pushing to make sure we make the final
> > changes
> > >>>> of ripping out any remaining platform components before we donate
> :).
> > >>>>
> > >>>> I agree that docker hub automation will be great. This will also
> allow
> > >>>> us to have a nightly build for any integration/UI tests we build.
> > >>>>
> > >>>> Thanks again Jarek!
> > >>>>
> > >>>> Daniel
> > >>>>
> > >>>> via Newton Mail
> > >>>> <
> >
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> > >
> > >>>>
> > >>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > >>>> wrote:
> > >>>>
> > >>>> Hello Greg and Daniel,
> > >>>>
> > >>>> Production image is merged and it's also going to be released in
> > >>>> 1.10.10. For now, I will manually tag the release in DockerHub when
> we
> > >>>> release but we will automate that in the future. Currently, we have
> > >>>> *airflow:master* and *airflow:v1-10-test* images (for all supported
> > >>>> python versions) that you can use for testing with the Helm chart!
> > >>>>
> > >>>> The next things from my side:
> > >>>>
> > >>>> * I will automate DockerHub building for the prod image from master
> -
> > >>>> for now only CI is automated but we need to add prod image as well
> > >>>> * I will move kind-cluster creation outside of the container and use
> > >>>> production image for our CI Kubernetes tests. I have a WIP branch on
> > it and
> > >>>> I think I should have it early next week
> > >>>> * The next step will be to switch our CI to Github Actions finally.
> I
> > >>>> think we will run both Travis and GA for a short time to make sure
> it
> > works
> > >>>> * I will ask people for options they would like to have in the
> image -
> > >>>> additional packages etc. so that I can work on the next iteration of
> > the
> > >>>> image. Also if you have any needs to see for the Helm integration, I
> > am
> > >>>> super happy to incorporate them in the next version. I plan to keep
> > the
> > >>>> "master" and "v1-10-test" images updated with any new changes and
> push
> > >>>> images regularly
> > >>>> * I am happy to help with the Helm Chart, so let me know if you need
> > >>>> anything!
> > >>>>
> > >>>> J.
> > >>>>
> > >>>>
> > >>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Hello Daniel and Greg.
> > >>>>>
> > >>>>> I think the production image for Airflow is quite ready for the
> final
> > >>>>> review and merge. I did some tests and I think it's quite complete
> > now.
> > >>>>>
> > >>>>> The PR is running it's Travis build:
> > >>>>> https://github.com/apache/airflow/pull/7832
> > >>>>>
> > >>>>> I already pushed the image to DockerHub - so that the next time it
> is
> > >>>>> merged It will use it as a base and will be rather quickly built
> > using
> > >>>>> those images as a cache. Once it is merged, I will add the
> automated
> > build
> > >>>>> in DockerHub and merge it to 1.10. As the next step I also plan to
> > use the
> > >>>>> prod image for our Kubernetes tests (that will help with moving to
> > the
> > >>>>> GitHub Actions from Travis).
> > >>>>>
> > >>>>> I added quite a comprehensive documentation - explaining all ins-
> and
> > >>>>> outs- of both CI and PROD images
> > >>>>>
> >
> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
> > >>>>> .
> > >>>>>
> > >>>>> You can build it yourself using manual build process described in
> the
> > >>>>> doc or (easier) using Breeze. That includes using the same
> > Dockerfile to
> > >>>>> build older Airflow version from the 1.10.* line.
> > >>>>>
> > >>>>> Feel free to use the new image in the Helm Chart you have - happy
> to
> > >>>>> review the PRs. The image is available at
> > 'apache/airflow:master-python3.6'
> > >>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10
> and
> > >>>>> release 1.10.10 we will also have it automatically published for
> > 1.10.10
> > >>>>> and beyond.
> > >>>>>
> > >>>>> Happy to hear your review comments and hope to merge it very soon.
> > >>>>> J.
> > >>>>>
> > >>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Absolutely! Please do :). The more eyes on this the better!
> > >>>>>>
> > >>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
> > >>>>>> daniel.imberman@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
> > >>>>>>> actions today. If we can get that working, we can donate the helm
> > chart and
> > >>>>>>> possibly even deprecate the old Kubernetes tests in the process.
> > >>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
> > >>>>>>> Jarek.Potiuk@polidea.com>, wrote:
> > >>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker
> > >>>>>>> there
> > >>>>>>> > was precisely Kubernetes tests and specifically running kind
> > >>>>>>> cluster
> > >>>>>>> > in the Github Actions
> > >>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
> > >>>>>>> meantime
> > >>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this
> > very
> > >>>>>>> > thing today at slack:
> > >>>>>>> >
> > >>>>>>>
> > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
> > >>>>>>> >
> > >>>>>>> > For me this is the next thing I will look at after merging
> > >>>>>>> > requirements PR and Prod image PR (and those two are a bit
> > >>>>>>> > pre-requisites to make it stable and fast). I want to switch to
> > the
> > >>>>>>> > Production Image to run Kubernetes tests (they will be much
> > faster)
> > >>>>>>> > and then we can work out much faster how to run them on Github
> > >>>>>>> > Actions. Then we can add Helm charts to run all those different
> > >>>>>>> > deployments. I am really looking forward to it!
> > >>>>>>> >
> > >>>>>>> > I hope - this weekend I move forward all of that and have a PR
> > >>>>>>> that we
> > >>>>>>> > can start running tests on and replacing the resources with
> helm
> > >>>>>>> > chart.
> > >>>>>>> >
> > >>>>>>> > J.
> > >>>>>>> >
> > >>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> > >>>>>>> > <da...@gmail.com> wrote:
> > >>>>>>> > >
> > >>>>>>> > > @Steven glad to help you out on that.
> > >>>>>>> > >
> > >>>>>>> > > For the moment we only use travis, but I believe @jarek has
> > been
> > >>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our
> > first
> > >>>>>>> GitHub tests?
> > >>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
> > >>>>>>> <st...@astronomer.io.invalid>, wrote:
> > >>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to
> > >>>>>>> test it. Where
> > >>>>>>> > > > do we want to run it? Do you all just use Travis only?
> > >>>>>>> > > >
> > >>>>>>> > > > Steven
> > >>>>>>> > > >
> > >>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
> > >>>>>>> daniel.imberman@gmail.com>
> > >>>>>>> > > > wrote:
> > >>>>>>> > > >
> > >>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go
> > >>>>>>> even further
> > >>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
> > >>>>>>> autoscaling, and a
> > >>>>>>> > > > > bunch of other configurations.
> > >>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
> > >>>>>>> Jarek.Potiuk@polidea.com>,
> > >>>>>>> > > > > wrote:
> > >>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original
> > >>>>>>> email :)
> > >>>>>>> > > > > >
> > >>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > >>>>>>> > > > > daniel.imberman@gmail.com> wrote:
> > >>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment.
> We
> > >>>>>>> just have
> > >>>>>>> > > > > deployment files. If anything having the helm chart as a
> > >>>>>>> part of the
> > >>>>>>> > > > > airflow repo could mean that the helm chart becomes the
> > >>>>>>> defacto system for
> > >>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the
> > >>>>>>> yams files and run
> > >>>>>>> > > > > multiple k8s tests with different settings).
> > >>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > >>>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
> > >>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow
> > >>>>>>> repo. We've also
> > >>>>>>> > > > > been
> > >>>>>>> > > > > > > > > building up a test suite that currently runs on
> > >>>>>>> CircleCI and uses
> > >>>>>>> > > > > kind
> > >>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
> > >>>>>>> versions with
> > >>>>>>> > > > > some
> > >>>>>>> > > > > > > > > different settings. Right now we're mostly
> testing
> > >>>>>>> the different
> > >>>>>>> > > > > executors
> > >>>>>>> > > > > > > > > since that has the biggest impact on what gets
> > >>>>>>> deployed, but that
> > >>>>>>> > > > > can be
> > >>>>>>> > > > > > > > > expanded.
> > >>>>>>> > > > > > > > >
> > >>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow
> > >>>>>>> for the tests?
> > >>>>>>> > > > > > > > >
> > >>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > >>>>>>> > > > > Jarek.Potiuk@polidea.com>
> > >>>>>>> > > > > > > > > wrote:
> > >>>>>>> > > > > > > > >
> > >>>>>>> > > > > > > > > > One thing for the donation.
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > Did you you want to have separate repository
> > Greg ?
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > I think we should simply create a folder in
> > >>>>>>> Airflow repo and
> > >>>>>>> > > > > keep it
> > >>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am
> > >>>>>>> going to switch our
> > >>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will
> > >>>>>>> make the tests
> > >>>>>>> > > > > much
> > >>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
> > >>>>>>> automatically in
> > >>>>>>> > > > > CI -
> > >>>>>>> > > > > > > > > > for now we are using some custom Resource
> > >>>>>>> definitions to start
> > >>>>>>> > > > > Airflow
> > >>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we
> could
> > >>>>>>> switch to
> > >>>>>>> > > > > using the
> > >>>>>>> > > > > > > > > > helm chart - this way we can test all three
> > things
> > >>>>>>> at once:
> > >>>>>>> > > > > > > > > > - Kubernetes Executor
> > >>>>>>> > > > > > > > > > - Dockerfile
> > >>>>>>> > > > > > > > > > - Helm Chart
> > >>>>>>> > > > > > > > > > and we could also add more tests - for example
> > >>>>>>> testing different
> > >>>>>>> > > > > > > > > > deployment options for the helm chart.
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would
> help
> > >>>>>>> with that -
> > >>>>>>> > > > > > > > > > especially in terms of future changes and
> testing
> > >>>>>>> them
> > >>>>>>> > > > > automatically.
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > J.
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal
> Nurmamat
> > >>>>>>> kyzy
> > >>>>>>> > > > > > > > > > <ai...@apache.org> wrote:
> > >>>>>>> > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more
> > >>>>>>> useful stuff for
> > >>>>>>> > > > > the
> > >>>>>>> > > > > > > > > > > community :)
> > >>>>>>> > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg
> Neiheisel
> > <
> > >>>>>>> > > > > greg@astronomer.io>
> > >>>>>>> > > > > > > > > > wrote:
> > >>>>>>> > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now
> as
> > >>>>>>> an optional
> > >>>>>>> > > > > Kubernetes
> > >>>>>>> > > > > > > > > > > > CronJob (
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > >
> > >>>>>>>
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > >>>>>>> > > > > > > > > > )
> > >>>>>>> > > > > > > > > > > > that we have run periodically to clean
> failed
> > >>>>>>> pods up and
> > >>>>>>> > > > > could stay
> > >>>>>>> > > > > > > > > > > > separate.
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > The wait_for_migrations script could
> > >>>>>>> definitely be pulled
> > >>>>>>> > > > > into Airflow.
> > >>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on
> > the
> > >>>>>>> scheduler (
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > >
> > >>>>>>>
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > >>>>>>> > > > > > > > > > )
> > >>>>>>> > > > > > > > > > > > that runs the upgradedb command before
> > booting
> > >>>>>>> the
> > >>>>>>> > > > > scheduler. This new
> > >>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
> > >>>>>>> initContainer on the
> > >>>>>>> > > > > webserver and
> > >>>>>>> > > > > > > > > > > > workers (
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > >
> > >>>>>>>
> >
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > >>>>>>> > > > > > > > > > )
> > >>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
> > >>>>>>> potentially
> > >>>>>>> > > > > long-running
> > >>>>>>> > > > > > > > > > migration
> > >>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
> > >>>>>>> columns/tables
> > >>>>>>> > > > > before the
> > >>>>>>> > > > > > > > > > > > migrations run. This prevents these pods
> from
> > >>>>>>> entering a
> > >>>>>>> > > > > CrashLoop.
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek
> > Potiuk <
> > >>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
> > >>>>>>> > > > > > > > > > > > wrote:
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
> > >>>>>>> currently
> > >>>>>>> > > > > generated from
> > >>>>>>> > > > > > > > > > > > > Dockerfiles
> > >>>>>>> > > > > > > > > > > > > > in this repo
> > >>>>>>> https://github.com/astronomer/ap-airflow
> > >>>>>>> > > > > and get
> > >>>>>>> > > > > > > > > > > > > published to
> > >>>>>>> > > > > > > > > > > > > > DockerHub
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > >
> > >>>>>>>
> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > For the most part those are typical
> > >>>>>>> Airflow images.
> > >>>>>>> > > > > There's an
> > >>>>>>> > > > > > > > > > > > > entrypoint
> > >>>>>>> > > > > > > > > > > > > > script that we include in the image
> that
> > >>>>>>> handles waiting
> > >>>>>>> > > > > for the
> > >>>>>>> > > > > > > > > > > > > database
> > >>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which
> is
> > >>>>>>> pretty generic.
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > I already added waiting for the database
> > >>>>>>> (both metadata
> > >>>>>>> > > > > and celery
> > >>>>>>> > > > > > > > > > URL) in
> > >>>>>>> > > > > > > > > > > > > the PR:
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > >
> > >>>>>>>
> >
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > >>>>>>> > > > > > > > > > > > > .
> > >>>>>>> > > > > > > > > > > > > It's functionally the same but more
> > generic.
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > The only other
> > >>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses
> > >>>>>>> would be the
> > >>>>>>> > > > > scripts in this
> > >>>>>>> > > > > > > > > > repo
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
> > >>>>>>> > > > > Our
> > >>>>>>> > > > > > > > > > > > > Dockerfiles
> > >>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are
> > >>>>>>> used to
> > >>>>>>> > > > > coordinate running
> > >>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > I see two scripts:
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
> > >>>>>>> needed to run in
> > >>>>>>> > > > > airflow -
> > >>>>>>> > > > > > > > > > this
> > >>>>>>> > > > > > > > > > > > > could be run as a separate pod/container?
> > >>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this
> is
> > >>>>>>> a good
> > >>>>>>> > > > > candidate to add
> > >>>>>>> > > > > > > > > > > > > *airflow
> > >>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make
> it
> > >>>>>>> part of airflow
> > >>>>>>> > > > > itself.
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > I think we also have to agree on the
> > Airflow
> > >>>>>>> version
> > >>>>>>> > > > > supported by the
> > >>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we
> support
> > >>>>>>> 1.10.10+ and we
> > >>>>>>> > > > > > > > > > incorporate
> > >>>>>>> > > > > > > > > > > > > all
> > >>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the
> "db
> > >>>>>>> > > > > wait_for_migration")
> > >>>>>>> > > > > > > > > > into 2.0
> > >>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and
> > >>>>>>> helm chart for
> > >>>>>>> > > > > those versions
> > >>>>>>> > > > > > > > > > > > > only. That would help with people
> migrating
> > >>>>>>> to the latest
> > >>>>>>> > > > > version.
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > WDYT?
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
> > >>>>>>> Imberman <
> > >>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
> > >>>>>>> > > > > > > > > > > > > > wrote:
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think
> that
> > >>>>>>> pairing an
> > >>>>>>> > > > > official helm
> > >>>>>>> > > > > > > > > > chart
> > >>>>>>> > > > > > > > > > > > > > > with the official image would make
> for
> > a
> > >>>>>>> REALLY
> > >>>>>>> > > > > powerful “up and
> > >>>>>>> > > > > > > > > > > > > running
> > >>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I
> > have
> > >>>>>>> also been
> > >>>>>>> > > > > looking into
> > >>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
> > >>>>>>> create custom
> > >>>>>>> > > > > controllers
> > >>>>>>> > > > > > > > > > from
> > >>>>>>> > > > > > > > > > > > > helm
> > >>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a
> 1-2
> > >>>>>>> punch from the
> > >>>>>>> > > > > same code
> > >>>>>>> > > > > > > > > > base
> > >>>>>>> > > > > > > > > > > > > :).
> > >>>>>>> > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if
> there’s
> > >>>>>>> no issues, can
> > >>>>>>> > > > > we please
> > >>>>>>> > > > > > > > > > start
> > >>>>>>> > > > > > > > > > > > > > the
> > >>>>>>> > > > > > > > > > > > > > > process of donation?
> > >>>>>>> > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
> > >>>>>>> > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > Daniel
> > >>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
> > >>>>>>> Potiuk <
> > >>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > >>>>>>> > > > > > > > > > > > > > > wrote:
> > >>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with
> the
> > >>>>>>> official image
> > >>>>>>> > > > > we have
> > >>>>>>> > > > > > > > > > work in
> > >>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
> > >>>>>>> Astronomer's
> > >>>>>>> > > > > image while
> > >>>>>>> > > > > > > > > > > > > preparing
> > >>>>>>> > > > > > > > > > > > > > my
> > >>>>>>> > > > > > > > > > > > > > > > draft and we can make any
> adjustments
> > >>>>>>> needed to make
> > >>>>>>> > > > > it works
> > >>>>>>> > > > > > > > > > with
> > >>>>>>> > > > > > > > > > > > > the
> > >>>>>>> > > > > > > > > > > > > > > helm
> > >>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
> > >>>>>>> collaborate on that.
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > PR here:
> > >>>>>>> https://github.com/apache/airflow/pull/7832
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > J.
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM
> Kaxil
> > >>>>>>> Naik <
> > >>>>>>> > > > > kaxilnaik@gmail.com
> > >>>>>>> > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > wrote:
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
> > >>>>>>> tomasz.urbaszek@polidea.com> :
> > >>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
> > >>>>>>> > > > > https://github.com/astronomer/airflow-chart
> > >>>>>>> > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
> > >>>>>>> Tomasz Urbaszek <
> > >>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
> > >>>>>>> > > > > > > > > > > > > > > > > wrote:
> > >>>>>>> > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is
> > >>>>>>> something our
> > >>>>>>> > > > > community needs!
> > >>>>>>> > > > > > > > > > Using
> > >>>>>>> > > > > > > > > > > > > your
> > >>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a
> lot
> > >>>>>>> of sens to me
> > >>>>>>> > > > > because as
> > >>>>>>> > > > > > > > > > you
> > >>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
> > >>>>>>> > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow
> image
> > >>>>>>> do you use?
> > >>>>>>> > > > > Also, would you
> > >>>>>>> > > > > > > > > > > > > mind
> > >>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
> > >>>>>>> > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > Tomek
> > >>>>>>> > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM
> > >>>>>>> Greg Neiheisel
> > >>>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid>
> > >>>>>>> wrote:
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
> > >>>>>>> Astronomer, we’ve
> > >>>>>>> > > > > created,
> > >>>>>>> > > > > > > > > > managed,
> > >>>>>>> > > > > > > > > > > > > > and
> > >>>>>>> > > > > > > > > > > > > > > > > > hardened
> > >>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart
> > >>>>>>> for Airflow (
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> https://github.com/astronomer/airflow-chart)
> > >>>>>>> > > > > that is
> > >>>>>>> > > > > > > > > > being
> > >>>>>>> > > > > > > > > > > > > used
> > >>>>>>> > > > > > > > > > > > > > by
> > >>>>>>> > > > > > > > > > > > > > > > > both
> > >>>>>>> > > > > > > > > > > > > > > > > > our
> > >>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise
> customers.
> > >>>>>>> This chart is
> > >>>>>>> > > > > > > > > > battle-tested and
> > >>>>>>> > > > > > > > > > > > > > > running
> > >>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow
> deployments
> > >>>>>>> of varying
> > >>>>>>> > > > > sizes and
> > >>>>>>> > > > > > > > > > runtime
> > >>>>>>> > > > > > > > > > > > > > > > > > environments.
> > >>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to
> > >>>>>>> encapsulate the issues
> > >>>>>>> > > > > that Airflow
> > >>>>>>> > > > > > > > > > > > > users
> > >>>>>>> > > > > > > > > > > > > > run
> > >>>>>>> > > > > > > > > > > > > > > > > into
> > >>>>>>> > > > > > > > > > > > > > > > > > in
> > >>>>>>> > > > > > > > > > > > > > > > > > > the real world.
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > While this chart was
> originally
> > >>>>>>> developed
> > >>>>>>> > > > > internally for
> > >>>>>>> > > > > > > > > > our
> > >>>>>>> > > > > > > > > > > > > > > Astronomer
> > >>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
> > >>>>>>> decoupled the chart
> > >>>>>>> > > > > from the
> > >>>>>>> > > > > > > > > > rest of
> > >>>>>>> > > > > > > > > > > > > our
> > >>>>>>> > > > > > > > > > > > > > > > > > platform
> > >>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the
> > greater
> > >>>>>>> Airflow
> > >>>>>>> > > > > community. With
> > >>>>>>> > > > > > > > > > these
> > >>>>>>> > > > > > > > > > > > > > > changes
> > >>>>>>> > > > > > > > > > > > > > > > > in
> > >>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
> > >>>>>>> conversation about
> > >>>>>>> > > > > donating this
> > >>>>>>> > > > > > > > > > > > > chart
> > >>>>>>> > > > > > > > > > > > > > to
> > >>>>>>> > > > > > > > > > > > > > > the
> > >>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of
> > the
> > >>>>>>> chart are:
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box.
> With
> > >>>>>>> zero
> > >>>>>>> > > > > configuration, a user
> > >>>>>>> > > > > > > > > > > > > will
> > >>>>>>> > > > > > > > > > > > > > get
> > >>>>>>> > > > > > > > > > > > > > > > > a
> > >>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default
> > >>>>>>> user and the
> > >>>>>>> > > > > > > > > > KubernetesExecutor
> > >>>>>>> > > > > > > > > > > > > > ready
> > >>>>>>> > > > > > > > > > > > > > > > > to
> > >>>>>>> > > > > > > > > > > > > > > > > > run
> > >>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery
> (w/
> > >>>>>>> optional KEDA
> > >>>>>>> > > > > > > > > > autoscaling) and
> > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > Support for optional
> pgbouncer.
> > >>>>>>> We use this to
> > >>>>>>> > > > > share a
> > >>>>>>> > > > > > > > > > > > > > configurable
> > >>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
> > >>>>>>> deployment. Useful
> > >>>>>>> > > > > for limiting
> > >>>>>>> > > > > > > > > > > > > > > connections to
> > >>>>>>> > > > > > > > > > > > > > > > > > the
> > >>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support.
> A
> > >>>>>>> user can push a
> > >>>>>>> > > > > newer
> > >>>>>>> > > > > > > > > > version
> > >>>>>>> > > > > > > > > > > > > of
> > >>>>>>> > > > > > > > > > > > > > > > > > Airflow
> > >>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and
> > >>>>>>> migrations will
> > >>>>>>> > > > > > > > > > automatically run
> > >>>>>>> > > > > > > > > > > > > > > > > > cleanly.
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support.
> > Optionally
> > >>>>>>> install and
> > >>>>>>> > > > > configure a
> > >>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
> > >>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
> > >>>>>>> expose them to
> > >>>>>>> > > > > Prometheus
> > >>>>>>> > > > > > > > > > > > > > > > > automatically.
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Resource control.
> Optionally
> > >>>>>>> control the
> > >>>>>>> > > > > ResourceQuotas
> > >>>>>>> > > > > > > > > > and
> > >>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each
> deployment
> > >>>>>>> so that no
> > >>>>>>> > > > > deployment can
> > >>>>>>> > > > > > > > > > > > > > overload
> > >>>>>>> > > > > > > > > > > > > > > a
> > >>>>>>> > > > > > > > > > > > > > > > > > > cluster.
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional
> Elasticsearch
> > >>>>>>> support.
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
> > >>>>>>> Sometimes
> > >>>>>>> > > > > > > > > > KubernetesExecutor and
> > >>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods
> fail
> > >>>>>>> for reasons
> > >>>>>>> > > > > other than the
> > >>>>>>> > > > > > > > > > > > > actual
> > >>>>>>> > > > > > > > > > > > > > > > > > task.
> > >>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep
> things
> > >>>>>>> clean in
> > >>>>>>> > > > > Kubernetes.
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally
> > in
> > >>>>>>> KIND
> > >>>>>>> > > > > (Kubernetes in
> > >>>>>>> > > > > > > > > > Docker).
> > >>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across
> > >>>>>>> many Kubernetes
> > >>>>>>> > > > > versions
> > >>>>>>> > > > > > > > > > with
> > >>>>>>> > > > > > > > > > > > > Helm
> > >>>>>>> > > > > > > > > > > > > > 2
> > >>>>>>> > > > > > > > > > > > > > > > > > and 3
> > >>>>>>> > > > > > > > > > > > > > > > > > > support.
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest
> > >>>>>>> and most
> > >>>>>>> > > > > reliable way to
> > >>>>>>> > > > > > > > > > deploy
> > >>>>>>> > > > > > > > > > > > > > DAGs
> > >>>>>>> > > > > > > > > > > > > > > to
> > >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at
> > >>>>>>> scale is to
> > >>>>>>> > > > > package them
> > >>>>>>> > > > > > > > > > into
> > >>>>>>> > > > > > > > > > > > > the
> > >>>>>>> > > > > > > > > > > > > > > actual
> > >>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have
> geared
> > >>>>>>> this chart
> > >>>>>>> > > > > towards that
> > >>>>>>> > > > > > > > > > > > > method of
> > >>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding
> other
> > >>>>>>> methods should
> > >>>>>>> > > > > be
> > >>>>>>> > > > > > > > > > > > > straightforward.
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from
> the
> > >>>>>>> community and
> > >>>>>>> > > > > would love
> > >>>>>>> > > > > > > > > > to
> > >>>>>>> > > > > > > > > > > > > see
> > >>>>>>> > > > > > > > > > > > > > > this
> > >>>>>>> > > > > > > > > > > > > > > > > > chart
> > >>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and
> > >>>>>>> running on
> > >>>>>>> > > > > Kubernetes!
> > >>>>>>> > > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > > > > --
> > >>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief
> > >>>>>>> Architect
> > >>>>>>> > > > > Astronomer.io
> > >>>>>>> > > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > --
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
> > >>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/>
> |
> > >>>>>>> Principal
> > >>>>>>> > > > > Software
> > >>>>>>> > > > > > > > > > Engineer
> > >>>>>>> > > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
> > >>>>>>> <+48%20660%20796%20129>
> > >>>>>>> > > > > <+48660796129
> > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > >>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
> > >>>>>>> https://www.polidea.com/>
> > >>>>>>> > > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > > --
> > >>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > >>>>>>> Astronomer.io
> > >>>>>>> > > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > --
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > Jarek Potiuk
> > >>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
> > >>>>>>> Principal Software
> > >>>>>>> > > > > Engineer
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > > M: +48 660 796 129
> <+48%20660%20796%20129>
> > >>>>>>> <+48660796129
> > >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > >>>>>>> > > > > > > > > > > > > [image: Polidea] <
> https://www.polidea.com/
> > >
> > >>>>>>> > > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > > > > --
> > >>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > >>>>>>> Astronomer.io
> > >>>>>>> > > > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > --
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > Jarek Potiuk
> > >>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > > > M: +48 660 796 129
> > >>>>>>> > > > > > > > > >
> > >>>>>>> > > > > > > > >
> > >>>>>>> > > > > > > > >
> > >>>>>>> > > > > > > > > --
> > >>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >>>>>>> > > > > >
> > >>>>>>> > > > > >
> > >>>>>>> > > > > > --
> > >>>>>>> > > > > > Jarek Potiuk
> > >>>>>>> > > > > > Polidea | Principal Software Engineer
> > >>>>>>> > > > > > M: +48 660 796 129
> > >>>>>>> > > > > >
> > >>>>>>> > > > >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> >
> > >>>>>>> > --
> > >>>>>>> >
> > >>>>>>> > Jarek Potiuk
> > >>>>>>> > Polidea | Principal Software Engineer
> > >>>>>>> >
> > >>>>>>> > M: +48 660 796 129
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>>
> > >>>>>> Jarek Potiuk
> > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>>>>>
> > >>>>>> M: +48 660 796 129 <+48660796129>
> > >>>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> Jarek Potiuk
> > >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>>>>
> > >>>>> M: +48 660 796 129 <+48660796129>
> > >>>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>> --
> > >>>>
> > >>>> Jarek Potiuk
> > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>>>
> > >>>> M: +48 660 796 129 <+48660796129>
> > >>>> [image: Polidea] <https://www.polidea.com/>
> > >>>>
> > >>>>
> > >>>
> > >>> --
> > >>> *Greg Neiheisel* / Chief Architect Astronomer.io
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48660796129>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >>
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> > >
> > >
> >
> > --
> > *Greg Neiheisel* / Chief Architect Astronomer.io
> >
>
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
Thanks, Greg 🚀
On Fri, May 8, 2020 at 4:19 AM Greg Neiheisel <gr...@astronomer.io.invalid>
wrote:
> Awesome Jarek. I just opened up a draft PR that adds the chart into the
> Airflow repo. https://github.com/apache/airflow/pull/8777
>
> It still contains some of our packaging, testing and releasing scripts. I
> figure they can serve as a starting point to discuss how we'd want to
> replicate into Airflow infrastructure.
>
> Looking forward to getting some feedback and moving this along!
>
> On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > Hey Greg,
> >
> > I just wanted to update you with some recent changes I've done to
> automate
> > most of the image builds stuff we have in Airflow. That might be useful
> for
> > your efforts.
> >
> > I am adding (and I will fix some of them very soon) issues based on
> > feedback from people on the #prod-docker-image channel:
> >
> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
> > But in the meantime all the build/CI infrastructure has nicely fallen
> into
> > place and It's almost complete (one thing missing but very simple is
> > automated builds of release images - for now this is manual but it is
> super
> > easy change and I will add it before 1.10.11 for sure).
> >
> > I described the whole build infrastructure here:
> > https://github.com/apache/airflow/blob/master/CI.rst including how image
> > caching works, how nightly builds and images are triggered. I hope it
> will
> > be helpful in the helm-chart work. Any questions or asks for the image -
> > please let me know, it is super easy to update the image now when we have
> > infrastructure to build it automatically integrated well with Github
> > Actions and DockerHub.
> >
> > J.
> >
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> >> Awesome, thanks for the update!
> >>
> >> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io>
> >> wrote:
> >>
> >>> Hey everyone, just wanted to post a quick update on the helm chart.
> >>> We're in the process of removing any references to Astronomer specific
> >>> things and making them more general features. That work is happening on
> >>> this PR here https://github.com/astronomer/airflow-chart/pull/66.
> >>> Hoping to wrap up the last few outstanding items and open up a PR to
> >>> Airflow next week sometime!
> >>>
> >>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
> >>> daniel.imberman@gmail.com> wrote:
> >>>
> >>>> Hi Jarek,
> >>>>
> >>>> Congrats on getting the image out! This is going to make a lot of
> >>>> things easier :). We’ve actually been working on separating out KinD
> as
> >>>> well (and in fact moving it out of travis and into GitHub issues).
> Let me
> >>>> circle back with Steven (the engineer who’s been working on that) and
> get a
> >>>> status report from him.
> >>>>
> >>>> I will also take some time today to get our helm chart to work with
> >>>> the airflow master image. It’s been a bit crazy around here in the
> last few
> >>>> weeks but I am consistently pushing to make sure we make the final
> changes
> >>>> of ripping out any remaining platform components before we donate :).
> >>>>
> >>>> I agree that docker hub automation will be great. This will also allow
> >>>> us to have a nightly build for any integration/UI tests we build.
> >>>>
> >>>> Thanks again Jarek!
> >>>>
> >>>> Daniel
> >>>>
> >>>> via Newton Mail
> >>>> <
> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2
> >
> >>>>
> >>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> >>>> wrote:
> >>>>
> >>>> Hello Greg and Daniel,
> >>>>
> >>>> Production image is merged and it's also going to be released in
> >>>> 1.10.10. For now, I will manually tag the release in DockerHub when we
> >>>> release but we will automate that in the future. Currently, we have
> >>>> *airflow:master* and *airflow:v1-10-test* images (for all supported
> >>>> python versions) that you can use for testing with the Helm chart!
> >>>>
> >>>> The next things from my side:
> >>>>
> >>>> * I will automate DockerHub building for the prod image from master -
> >>>> for now only CI is automated but we need to add prod image as well
> >>>> * I will move kind-cluster creation outside of the container and use
> >>>> production image for our CI Kubernetes tests. I have a WIP branch on
> it and
> >>>> I think I should have it early next week
> >>>> * The next step will be to switch our CI to Github Actions finally. I
> >>>> think we will run both Travis and GA for a short time to make sure it
> works
> >>>> * I will ask people for options they would like to have in the image -
> >>>> additional packages etc. so that I can work on the next iteration of
> the
> >>>> image. Also if you have any needs to see for the Helm integration, I
> am
> >>>> super happy to incorporate them in the next version. I plan to keep
> the
> >>>> "master" and "v1-10-test" images updated with any new changes and push
> >>>> images regularly
> >>>> * I am happy to help with the Helm Chart, so let me know if you need
> >>>> anything!
> >>>>
> >>>> J.
> >>>>
> >>>>
> >>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> >>>> wrote:
> >>>>
> >>>>> Hello Daniel and Greg.
> >>>>>
> >>>>> I think the production image for Airflow is quite ready for the final
> >>>>> review and merge. I did some tests and I think it's quite complete
> now.
> >>>>>
> >>>>> The PR is running it's Travis build:
> >>>>> https://github.com/apache/airflow/pull/7832
> >>>>>
> >>>>> I already pushed the image to DockerHub - so that the next time it is
> >>>>> merged It will use it as a base and will be rather quickly built
> using
> >>>>> those images as a cache. Once it is merged, I will add the automated
> build
> >>>>> in DockerHub and merge it to 1.10. As the next step I also plan to
> use the
> >>>>> prod image for our Kubernetes tests (that will help with moving to
> the
> >>>>> GitHub Actions from Travis).
> >>>>>
> >>>>> I added quite a comprehensive documentation - explaining all ins- and
> >>>>> outs- of both CI and PROD images
> >>>>>
> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
> >>>>> .
> >>>>>
> >>>>> You can build it yourself using manual build process described in the
> >>>>> doc or (easier) using Breeze. That includes using the same
> Dockerfile to
> >>>>> build older Airflow version from the 1.10.* line.
> >>>>>
> >>>>> Feel free to use the new image in the Helm Chart you have - happy to
> >>>>> review the PRs. The image is available at
> 'apache/airflow:master-python3.6'
> >>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
> >>>>> release 1.10.10 we will also have it automatically published for
> 1.10.10
> >>>>> and beyond.
> >>>>>
> >>>>> Happy to hear your review comments and hope to merge it very soon.
> >>>>> J.
> >>>>>
> >>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Absolutely! Please do :). The more eyes on this the better!
> >>>>>>
> >>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
> >>>>>> daniel.imberman@gmail.com> wrote:
> >>>>>>
> >>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
> >>>>>>> actions today. If we can get that working, we can donate the helm
> chart and
> >>>>>>> possibly even deprecate the old Kubernetes tests in the process.
> >>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
> >>>>>>> Jarek.Potiuk@polidea.com>, wrote:
> >>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker
> >>>>>>> there
> >>>>>>> > was precisely Kubernetes tests and specifically running kind
> >>>>>>> cluster
> >>>>>>> > in the Github Actions
> >>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
> >>>>>>> meantime
> >>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this
> very
> >>>>>>> > thing today at slack:
> >>>>>>> >
> >>>>>>>
> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
> >>>>>>> >
> >>>>>>> > For me this is the next thing I will look at after merging
> >>>>>>> > requirements PR and Prod image PR (and those two are a bit
> >>>>>>> > pre-requisites to make it stable and fast). I want to switch to
> the
> >>>>>>> > Production Image to run Kubernetes tests (they will be much
> faster)
> >>>>>>> > and then we can work out much faster how to run them on Github
> >>>>>>> > Actions. Then we can add Helm charts to run all those different
> >>>>>>> > deployments. I am really looking forward to it!
> >>>>>>> >
> >>>>>>> > I hope - this weekend I move forward all of that and have a PR
> >>>>>>> that we
> >>>>>>> > can start running tests on and replacing the resources with helm
> >>>>>>> > chart.
> >>>>>>> >
> >>>>>>> > J.
> >>>>>>> >
> >>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> >>>>>>> > <da...@gmail.com> wrote:
> >>>>>>> > >
> >>>>>>> > > @Steven glad to help you out on that.
> >>>>>>> > >
> >>>>>>> > > For the moment we only use travis, but I believe @jarek has
> been
> >>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our
> first
> >>>>>>> GitHub tests?
> >>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
> >>>>>>> <st...@astronomer.io.invalid>, wrote:
> >>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to
> >>>>>>> test it. Where
> >>>>>>> > > > do we want to run it? Do you all just use Travis only?
> >>>>>>> > > >
> >>>>>>> > > > Steven
> >>>>>>> > > >
> >>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
> >>>>>>> daniel.imberman@gmail.com>
> >>>>>>> > > > wrote:
> >>>>>>> > > >
> >>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go
> >>>>>>> even further
> >>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
> >>>>>>> autoscaling, and a
> >>>>>>> > > > > bunch of other configurations.
> >>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
> >>>>>>> Jarek.Potiuk@polidea.com>,
> >>>>>>> > > > > wrote:
> >>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original
> >>>>>>> email :)
> >>>>>>> > > > > >
> >>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> >>>>>>> > > > > daniel.imberman@gmail.com> wrote:
> >>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We
> >>>>>>> just have
> >>>>>>> > > > > deployment files. If anything having the helm chart as a
> >>>>>>> part of the
> >>>>>>> > > > > airflow repo could mean that the helm chart becomes the
> >>>>>>> defacto system for
> >>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the
> >>>>>>> yams files and run
> >>>>>>> > > > > multiple k8s tests with different settings).
> >>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> >>>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
> >>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow
> >>>>>>> repo. We've also
> >>>>>>> > > > > been
> >>>>>>> > > > > > > > > building up a test suite that currently runs on
> >>>>>>> CircleCI and uses
> >>>>>>> > > > > kind
> >>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
> >>>>>>> versions with
> >>>>>>> > > > > some
> >>>>>>> > > > > > > > > different settings. Right now we're mostly testing
> >>>>>>> the different
> >>>>>>> > > > > executors
> >>>>>>> > > > > > > > > since that has the biggest impact on what gets
> >>>>>>> deployed, but that
> >>>>>>> > > > > can be
> >>>>>>> > > > > > > > > expanded.
> >>>>>>> > > > > > > > >
> >>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow
> >>>>>>> for the tests?
> >>>>>>> > > > > > > > >
> >>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> >>>>>>> > > > > Jarek.Potiuk@polidea.com>
> >>>>>>> > > > > > > > > wrote:
> >>>>>>> > > > > > > > >
> >>>>>>> > > > > > > > > > One thing for the donation.
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > Did you you want to have separate repository
> Greg ?
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > I think we should simply create a folder in
> >>>>>>> Airflow repo and
> >>>>>>> > > > > keep it
> >>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am
> >>>>>>> going to switch our
> >>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will
> >>>>>>> make the tests
> >>>>>>> > > > > much
> >>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
> >>>>>>> automatically in
> >>>>>>> > > > > CI -
> >>>>>>> > > > > > > > > > for now we are using some custom Resource
> >>>>>>> definitions to start
> >>>>>>> > > > > Airflow
> >>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
> >>>>>>> switch to
> >>>>>>> > > > > using the
> >>>>>>> > > > > > > > > > helm chart - this way we can test all three
> things
> >>>>>>> at once:
> >>>>>>> > > > > > > > > > - Kubernetes Executor
> >>>>>>> > > > > > > > > > - Dockerfile
> >>>>>>> > > > > > > > > > - Helm Chart
> >>>>>>> > > > > > > > > > and we could also add more tests - for example
> >>>>>>> testing different
> >>>>>>> > > > > > > > > > deployment options for the helm chart.
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help
> >>>>>>> with that -
> >>>>>>> > > > > > > > > > especially in terms of future changes and testing
> >>>>>>> them
> >>>>>>> > > > > automatically.
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > J.
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat
> >>>>>>> kyzy
> >>>>>>> > > > > > > > > > <ai...@apache.org> wrote:
> >>>>>>> > > > > > > > > > >
> >>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more
> >>>>>>> useful stuff for
> >>>>>>> > > > > the
> >>>>>>> > > > > > > > > > > community :)
> >>>>>>> > > > > > > > > > >
> >>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel
> <
> >>>>>>> > > > > greg@astronomer.io>
> >>>>>>> > > > > > > > > > wrote:
> >>>>>>> > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as
> >>>>>>> an optional
> >>>>>>> > > > > Kubernetes
> >>>>>>> > > > > > > > > > > > CronJob (
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > >
> >>>>>>>
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> >>>>>>> > > > > > > > > > )
> >>>>>>> > > > > > > > > > > > that we have run periodically to clean failed
> >>>>>>> pods up and
> >>>>>>> > > > > could stay
> >>>>>>> > > > > > > > > > > > separate.
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > The wait_for_migrations script could
> >>>>>>> definitely be pulled
> >>>>>>> > > > > into Airflow.
> >>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on
> the
> >>>>>>> scheduler (
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > >
> >>>>>>>
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> >>>>>>> > > > > > > > > > )
> >>>>>>> > > > > > > > > > > > that runs the upgradedb command before
> booting
> >>>>>>> the
> >>>>>>> > > > > scheduler. This new
> >>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
> >>>>>>> initContainer on the
> >>>>>>> > > > > webserver and
> >>>>>>> > > > > > > > > > > > workers (
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > >
> >>>>>>>
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> >>>>>>> > > > > > > > > > )
> >>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
> >>>>>>> potentially
> >>>>>>> > > > > long-running
> >>>>>>> > > > > > > > > > migration
> >>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
> >>>>>>> columns/tables
> >>>>>>> > > > > before the
> >>>>>>> > > > > > > > > > > > migrations run. This prevents these pods from
> >>>>>>> entering a
> >>>>>>> > > > > CrashLoop.
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek
> Potiuk <
> >>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
> >>>>>>> > > > > > > > > > > > wrote:
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
> >>>>>>> currently
> >>>>>>> > > > > generated from
> >>>>>>> > > > > > > > > > > > > Dockerfiles
> >>>>>>> > > > > > > > > > > > > > in this repo
> >>>>>>> https://github.com/astronomer/ap-airflow
> >>>>>>> > > > > and get
> >>>>>>> > > > > > > > > > > > > published to
> >>>>>>> > > > > > > > > > > > > > DockerHub
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > >
> >>>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > For the most part those are typical
> >>>>>>> Airflow images.
> >>>>>>> > > > > There's an
> >>>>>>> > > > > > > > > > > > > entrypoint
> >>>>>>> > > > > > > > > > > > > > script that we include in the image that
> >>>>>>> handles waiting
> >>>>>>> > > > > for the
> >>>>>>> > > > > > > > > > > > > database
> >>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
> >>>>>>> pretty generic.
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > I already added waiting for the database
> >>>>>>> (both metadata
> >>>>>>> > > > > and celery
> >>>>>>> > > > > > > > > > URL) in
> >>>>>>> > > > > > > > > > > > > the PR:
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > >
> >>>>>>>
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> >>>>>>> > > > > > > > > > > > > .
> >>>>>>> > > > > > > > > > > > > It's functionally the same but more
> generic.
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > The only other
> >>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses
> >>>>>>> would be the
> >>>>>>> > > > > scripts in this
> >>>>>>> > > > > > > > > > repo
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
> >>>>>>> > > > > Our
> >>>>>>> > > > > > > > > > > > > Dockerfiles
> >>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are
> >>>>>>> used to
> >>>>>>> > > > > coordinate running
> >>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > I see two scripts:
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
> >>>>>>> needed to run in
> >>>>>>> > > > > airflow -
> >>>>>>> > > > > > > > > > this
> >>>>>>> > > > > > > > > > > > > could be run as a separate pod/container?
> >>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is
> >>>>>>> a good
> >>>>>>> > > > > candidate to add
> >>>>>>> > > > > > > > > > > > > *airflow
> >>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it
> >>>>>>> part of airflow
> >>>>>>> > > > > itself.
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > I think we also have to agree on the
> Airflow
> >>>>>>> version
> >>>>>>> > > > > supported by the
> >>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
> >>>>>>> 1.10.10+ and we
> >>>>>>> > > > > > > > > > incorporate
> >>>>>>> > > > > > > > > > > > > all
> >>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
> >>>>>>> > > > > wait_for_migration")
> >>>>>>> > > > > > > > > > into 2.0
> >>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and
> >>>>>>> helm chart for
> >>>>>>> > > > > those versions
> >>>>>>> > > > > > > > > > > > > only. That would help with people migrating
> >>>>>>> to the latest
> >>>>>>> > > > > version.
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > WDYT?
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
> >>>>>>> Imberman <
> >>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
> >>>>>>> > > > > > > > > > > > > > wrote:
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
> >>>>>>> pairing an
> >>>>>>> > > > > official helm
> >>>>>>> > > > > > > > > > chart
> >>>>>>> > > > > > > > > > > > > > > with the official image would make for
> a
> >>>>>>> REALLY
> >>>>>>> > > > > powerful “up and
> >>>>>>> > > > > > > > > > > > > running
> >>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I
> have
> >>>>>>> also been
> >>>>>>> > > > > looking into
> >>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
> >>>>>>> create custom
> >>>>>>> > > > > controllers
> >>>>>>> > > > > > > > > > from
> >>>>>>> > > > > > > > > > > > > helm
> >>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
> >>>>>>> punch from the
> >>>>>>> > > > > same code
> >>>>>>> > > > > > > > > > base
> >>>>>>> > > > > > > > > > > > > :).
> >>>>>>> > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s
> >>>>>>> no issues, can
> >>>>>>> > > > > we please
> >>>>>>> > > > > > > > > > start
> >>>>>>> > > > > > > > > > > > > > the
> >>>>>>> > > > > > > > > > > > > > > process of donation?
> >>>>>>> > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
> >>>>>>> > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > Daniel
> >>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
> >>>>>>> Potiuk <
> >>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> >>>>>>> > > > > > > > > > > > > > > wrote:
> >>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
> >>>>>>> official image
> >>>>>>> > > > > we have
> >>>>>>> > > > > > > > > > work in
> >>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
> >>>>>>> Astronomer's
> >>>>>>> > > > > image while
> >>>>>>> > > > > > > > > > > > > preparing
> >>>>>>> > > > > > > > > > > > > > my
> >>>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
> >>>>>>> needed to make
> >>>>>>> > > > > it works
> >>>>>>> > > > > > > > > > with
> >>>>>>> > > > > > > > > > > > > the
> >>>>>>> > > > > > > > > > > > > > > helm
> >>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
> >>>>>>> collaborate on that.
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > PR here:
> >>>>>>> https://github.com/apache/airflow/pull/7832
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > J.
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil
> >>>>>>> Naik <
> >>>>>>> > > > > kaxilnaik@gmail.com
> >>>>>>> > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > wrote:
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
> >>>>>>> tomasz.urbaszek@polidea.com> :
> >>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
> >>>>>>> > > > > https://github.com/astronomer/airflow-chart
> >>>>>>> > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
> >>>>>>> Tomasz Urbaszek <
> >>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
> >>>>>>> > > > > > > > > > > > > > > > > wrote:
> >>>>>>> > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is
> >>>>>>> something our
> >>>>>>> > > > > community needs!
> >>>>>>> > > > > > > > > > Using
> >>>>>>> > > > > > > > > > > > > your
> >>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot
> >>>>>>> of sens to me
> >>>>>>> > > > > because as
> >>>>>>> > > > > > > > > > you
> >>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
> >>>>>>> > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image
> >>>>>>> do you use?
> >>>>>>> > > > > Also, would you
> >>>>>>> > > > > > > > > > > > > mind
> >>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
> >>>>>>> > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > Tomek
> >>>>>>> > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM
> >>>>>>> Greg Neiheisel
> >>>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid>
> >>>>>>> wrote:
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
> >>>>>>> Astronomer, we’ve
> >>>>>>> > > > > created,
> >>>>>>> > > > > > > > > > managed,
> >>>>>>> > > > > > > > > > > > > > and
> >>>>>>> > > > > > > > > > > > > > > > > > hardened
> >>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart
> >>>>>>> for Airflow (
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> https://github.com/astronomer/airflow-chart)
> >>>>>>> > > > > that is
> >>>>>>> > > > > > > > > > being
> >>>>>>> > > > > > > > > > > > > used
> >>>>>>> > > > > > > > > > > > > > by
> >>>>>>> > > > > > > > > > > > > > > > > both
> >>>>>>> > > > > > > > > > > > > > > > > > our
> >>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers.
> >>>>>>> This chart is
> >>>>>>> > > > > > > > > > battle-tested and
> >>>>>>> > > > > > > > > > > > > > > running
> >>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments
> >>>>>>> of varying
> >>>>>>> > > > > sizes and
> >>>>>>> > > > > > > > > > runtime
> >>>>>>> > > > > > > > > > > > > > > > > > environments.
> >>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to
> >>>>>>> encapsulate the issues
> >>>>>>> > > > > that Airflow
> >>>>>>> > > > > > > > > > > > > users
> >>>>>>> > > > > > > > > > > > > > run
> >>>>>>> > > > > > > > > > > > > > > > > into
> >>>>>>> > > > > > > > > > > > > > > > > > in
> >>>>>>> > > > > > > > > > > > > > > > > > > the real world.
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally
> >>>>>>> developed
> >>>>>>> > > > > internally for
> >>>>>>> > > > > > > > > > our
> >>>>>>> > > > > > > > > > > > > > > Astronomer
> >>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
> >>>>>>> decoupled the chart
> >>>>>>> > > > > from the
> >>>>>>> > > > > > > > > > rest of
> >>>>>>> > > > > > > > > > > > > our
> >>>>>>> > > > > > > > > > > > > > > > > > platform
> >>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the
> greater
> >>>>>>> Airflow
> >>>>>>> > > > > community. With
> >>>>>>> > > > > > > > > > these
> >>>>>>> > > > > > > > > > > > > > > changes
> >>>>>>> > > > > > > > > > > > > > > > > in
> >>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
> >>>>>>> conversation about
> >>>>>>> > > > > donating this
> >>>>>>> > > > > > > > > > > > > chart
> >>>>>>> > > > > > > > > > > > > > to
> >>>>>>> > > > > > > > > > > > > > > the
> >>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of
> the
> >>>>>>> chart are:
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With
> >>>>>>> zero
> >>>>>>> > > > > configuration, a user
> >>>>>>> > > > > > > > > > > > > will
> >>>>>>> > > > > > > > > > > > > > get
> >>>>>>> > > > > > > > > > > > > > > > > a
> >>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default
> >>>>>>> user and the
> >>>>>>> > > > > > > > > > KubernetesExecutor
> >>>>>>> > > > > > > > > > > > > > ready
> >>>>>>> > > > > > > > > > > > > > > > > to
> >>>>>>> > > > > > > > > > > > > > > > > > run
> >>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
> >>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
> >>>>>>> optional KEDA
> >>>>>>> > > > > > > > > > autoscaling) and
> >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer.
> >>>>>>> We use this to
> >>>>>>> > > > > share a
> >>>>>>> > > > > > > > > > > > > > configurable
> >>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
> >>>>>>> deployment. Useful
> >>>>>>> > > > > for limiting
> >>>>>>> > > > > > > > > > > > > > > connections to
> >>>>>>> > > > > > > > > > > > > > > > > > the
> >>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A
> >>>>>>> user can push a
> >>>>>>> > > > > newer
> >>>>>>> > > > > > > > > > version
> >>>>>>> > > > > > > > > > > > > of
> >>>>>>> > > > > > > > > > > > > > > > > > Airflow
> >>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and
> >>>>>>> migrations will
> >>>>>>> > > > > > > > > > automatically run
> >>>>>>> > > > > > > > > > > > > > > > > > cleanly.
> >>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support.
> Optionally
> >>>>>>> install and
> >>>>>>> > > > > configure a
> >>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
> >>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
> >>>>>>> expose them to
> >>>>>>> > > > > Prometheus
> >>>>>>> > > > > > > > > > > > > > > > > automatically.
> >>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
> >>>>>>> control the
> >>>>>>> > > > > ResourceQuotas
> >>>>>>> > > > > > > > > > and
> >>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment
> >>>>>>> so that no
> >>>>>>> > > > > deployment can
> >>>>>>> > > > > > > > > > > > > > overload
> >>>>>>> > > > > > > > > > > > > > > a
> >>>>>>> > > > > > > > > > > > > > > > > > > cluster.
> >>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
> >>>>>>> support.
> >>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
> >>>>>>> Sometimes
> >>>>>>> > > > > > > > > > KubernetesExecutor and
> >>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail
> >>>>>>> for reasons
> >>>>>>> > > > > other than the
> >>>>>>> > > > > > > > > > > > > actual
> >>>>>>> > > > > > > > > > > > > > > > > > task.
> >>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things
> >>>>>>> clean in
> >>>>>>> > > > > Kubernetes.
> >>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally
> in
> >>>>>>> KIND
> >>>>>>> > > > > (Kubernetes in
> >>>>>>> > > > > > > > > > Docker).
> >>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across
> >>>>>>> many Kubernetes
> >>>>>>> > > > > versions
> >>>>>>> > > > > > > > > > with
> >>>>>>> > > > > > > > > > > > > Helm
> >>>>>>> > > > > > > > > > > > > > 2
> >>>>>>> > > > > > > > > > > > > > > > > > and 3
> >>>>>>> > > > > > > > > > > > > > > > > > > support.
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest
> >>>>>>> and most
> >>>>>>> > > > > reliable way to
> >>>>>>> > > > > > > > > > deploy
> >>>>>>> > > > > > > > > > > > > > DAGs
> >>>>>>> > > > > > > > > > > > > > > to
> >>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at
> >>>>>>> scale is to
> >>>>>>> > > > > package them
> >>>>>>> > > > > > > > > > into
> >>>>>>> > > > > > > > > > > > > the
> >>>>>>> > > > > > > > > > > > > > > actual
> >>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared
> >>>>>>> this chart
> >>>>>>> > > > > towards that
> >>>>>>> > > > > > > > > > > > > method of
> >>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other
> >>>>>>> methods should
> >>>>>>> > > > > be
> >>>>>>> > > > > > > > > > > > > straightforward.
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
> >>>>>>> community and
> >>>>>>> > > > > would love
> >>>>>>> > > > > > > > > > to
> >>>>>>> > > > > > > > > > > > > see
> >>>>>>> > > > > > > > > > > > > > > this
> >>>>>>> > > > > > > > > > > > > > > > > > chart
> >>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and
> >>>>>>> running on
> >>>>>>> > > > > Kubernetes!
> >>>>>>> > > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > > > > --
> >>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief
> >>>>>>> Architect
> >>>>>>> > > > > Astronomer.io
> >>>>>>> > > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > --
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
> >>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
> >>>>>>> Principal
> >>>>>>> > > > > Software
> >>>>>>> > > > > > > > > > Engineer
> >>>>>>> > > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
> >>>>>>> <+48%20660%20796%20129>
> >>>>>>> > > > > <+48660796129
> >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> >>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
> >>>>>>> https://www.polidea.com/>
> >>>>>>> > > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > > --
> >>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> >>>>>>> Astronomer.io
> >>>>>>> > > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > --
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > Jarek Potiuk
> >>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
> >>>>>>> Principal Software
> >>>>>>> > > > > Engineer
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> >>>>>>> <+48660796129
> >>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
> >>>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/
> >
> >>>>>>> > > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > > > > --
> >>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> >>>>>>> Astronomer.io
> >>>>>>> > > > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > --
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > Jarek Potiuk
> >>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > > > M: +48 660 796 129
> >>>>>>> > > > > > > > > >
> >>>>>>> > > > > > > > >
> >>>>>>> > > > > > > > >
> >>>>>>> > > > > > > > > --
> >>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> >>>>>>> > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > > --
> >>>>>>> > > > > > Jarek Potiuk
> >>>>>>> > > > > > Polidea | Principal Software Engineer
> >>>>>>> > > > > > M: +48 660 796 129
> >>>>>>> > > > > >
> >>>>>>> > > > >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > --
> >>>>>>> >
> >>>>>>> > Jarek Potiuk
> >>>>>>> > Polidea | Principal Software Engineer
> >>>>>>> >
> >>>>>>> > M: +48 660 796 129
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Jarek Potiuk
> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>>
> >>>>>> M: +48 660 796 129 <+48660796129>
> >>>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Jarek Potiuk
> >>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>>
> >>>>> M: +48 660 796 129 <+48660796129>
> >>>>> [image: Polidea] <https://www.polidea.com/>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>>
> >>>> Jarek Potiuk
> >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>>>
> >>>> M: +48 660 796 129 <+48660796129>
> >>>> [image: Polidea] <https://www.polidea.com/>
> >>>>
> >>>>
> >>>
> >>> --
> >>> *Greg Neiheisel* / Chief Architect Astronomer.io
> >>>
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48660796129>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >>
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
> >
> >
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Cool. Wil look into it over weekend. If you think anything can be improved
in the image to make it a better fit - fill free to create new issue or
comment the existing issues
https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
On Fri, May 8, 2020 at 5:19 AM Greg Neiheisel <gr...@astronomer.io> wrote:
> Awesome Jarek. I just opened up a draft PR that adds the chart into the
> Airflow repo. https://github.com/apache/airflow/pull/8777
>
> It still contains some of our packaging, testing and releasing scripts. I
> figure they can serve as a starting point to discuss how we'd want to
> replicate into Airflow infrastructure.
>
> Looking forward to getting some feedback and moving this along!
>
> On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Hey Greg,
>>
>> I just wanted to update you with some recent changes I've done to
>> automate most of the image builds stuff we have in Airflow. That might be
>> useful for your efforts.
>>
>> I am adding (and I will fix some of them very soon) issues based on
>> feedback from people on the #prod-docker-image channel:
>> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
>> But in the meantime all the build/CI infrastructure has nicely fallen into
>> place and It's almost complete (one thing missing but very simple is
>> automated builds of release images - for now this is manual but it is super
>> easy change and I will add it before 1.10.11 for sure).
>>
>> I described the whole build infrastructure here:
>> https://github.com/apache/airflow/blob/master/CI.rst including how image
>> caching works, how nightly builds and images are triggered. I hope it will
>> be helpful in the helm-chart work. Any questions or asks for the image -
>> please let me know, it is super easy to update the image now when we have
>> infrastructure to build it automatically integrated well with Github
>> Actions and DockerHub.
>>
>> J.
>>
>>
>>
>>
>>
>>
>>
>>
>> On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Awesome, thanks for the update!
>>>
>>> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io>
>>> wrote:
>>>
>>>> Hey everyone, just wanted to post a quick update on the helm chart.
>>>> We're in the process of removing any references to Astronomer specific
>>>> things and making them more general features. That work is happening on
>>>> this PR here https://github.com/astronomer/airflow-chart/pull/66.
>>>> Hoping to wrap up the last few outstanding items and open up a PR to
>>>> Airflow next week sometime!
>>>>
>>>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
>>>> daniel.imberman@gmail.com> wrote:
>>>>
>>>>> Hi Jarek,
>>>>>
>>>>> Congrats on getting the image out! This is going to make a lot of
>>>>> things easier :). We’ve actually been working on separating out KinD as
>>>>> well (and in fact moving it out of travis and into GitHub issues). Let me
>>>>> circle back with Steven (the engineer who’s been working on that) and get a
>>>>> status report from him.
>>>>>
>>>>> I will also take some time today to get our helm chart to work with
>>>>> the airflow master image. It’s been a bit crazy around here in the last few
>>>>> weeks but I am consistently pushing to make sure we make the final changes
>>>>> of ripping out any remaining platform components before we donate :).
>>>>>
>>>>> I agree that docker hub automation will be great. This will also allow
>>>>> us to have a nightly build for any integration/UI tests we build.
>>>>>
>>>>> Thanks again Jarek!
>>>>>
>>>>> Daniel
>>>>>
>>>>> via Newton Mail
>>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>>>>>
>>>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <Ja...@polidea.com>
>>>>> wrote:
>>>>>
>>>>> Hello Greg and Daniel,
>>>>>
>>>>> Production image is merged and it's also going to be released in
>>>>> 1.10.10. For now, I will manually tag the release in DockerHub when we
>>>>> release but we will automate that in the future. Currently, we have
>>>>> *airflow:master* and *airflow:v1-10-test* images (for all supported
>>>>> python versions) that you can use for testing with the Helm chart!
>>>>>
>>>>> The next things from my side:
>>>>>
>>>>> * I will automate DockerHub building for the prod image from master -
>>>>> for now only CI is automated but we need to add prod image as well
>>>>> * I will move kind-cluster creation outside of the container and use
>>>>> production image for our CI Kubernetes tests. I have a WIP branch on it and
>>>>> I think I should have it early next week
>>>>> * The next step will be to switch our CI to Github Actions finally. I
>>>>> think we will run both Travis and GA for a short time to make sure it works
>>>>> * I will ask people for options they would like to have in the image -
>>>>> additional packages etc. so that I can work on the next iteration of the
>>>>> image. Also if you have any needs to see for the Helm integration, I am
>>>>> super happy to incorporate them in the next version. I plan to keep the
>>>>> "master" and "v1-10-test" images updated with any new changes and push
>>>>> images regularly
>>>>> * I am happy to help with the Helm Chart, so let me know if you need
>>>>> anything!
>>>>>
>>>>> J.
>>>>>
>>>>>
>>>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>
>>>>>> Hello Daniel and Greg.
>>>>>>
>>>>>> I think the production image for Airflow is quite ready for the final
>>>>>> review and merge. I did some tests and I think it's quite complete now.
>>>>>>
>>>>>> The PR is running it's Travis build:
>>>>>> https://github.com/apache/airflow/pull/7832
>>>>>>
>>>>>> I already pushed the image to DockerHub - so that the next time it is
>>>>>> merged It will use it as a base and will be rather quickly built using
>>>>>> those images as a cache. Once it is merged, I will add the automated build
>>>>>> in DockerHub and merge it to 1.10. As the next step I also plan to use the
>>>>>> prod image for our Kubernetes tests (that will help with moving to the
>>>>>> GitHub Actions from Travis).
>>>>>>
>>>>>> I added quite a comprehensive documentation - explaining all ins- and
>>>>>> outs- of both CI and PROD images
>>>>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
>>>>>> .
>>>>>>
>>>>>> You can build it yourself using manual build process described in the
>>>>>> doc or (easier) using Breeze. That includes using the same Dockerfile to
>>>>>> build older Airflow version from the 1.10.* line.
>>>>>>
>>>>>> Feel free to use the new image in the Helm Chart you have - happy to
>>>>>> review the PRs. The image is available at 'apache/airflow:master-python3.6'
>>>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
>>>>>> release 1.10.10 we will also have it automatically published for 1.10.10
>>>>>> and beyond.
>>>>>>
>>>>>> Happy to hear your review comments and hope to merge it very soon.
>>>>>> J.
>>>>>>
>>>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <
>>>>>> Jarek.Potiuk@polidea.com> wrote:
>>>>>>
>>>>>>> Absolutely! Please do :). The more eyes on this the better!
>>>>>>>
>>>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
>>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>>
>>>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
>>>>>>>> actions today. If we can get that working, we can donate the helm chart and
>>>>>>>> possibly even deprecate the old Kubernetes tests in the process.
>>>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
>>>>>>>> Jarek.Potiuk@polidea.com>, wrote:
>>>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker
>>>>>>>> there
>>>>>>>> > was precisely Kubernetes tests and specifically running kind
>>>>>>>> cluster
>>>>>>>> > in the Github Actions
>>>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
>>>>>>>> meantime
>>>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this
>>>>>>>> very
>>>>>>>> > thing today at slack:
>>>>>>>> >
>>>>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>>>>>>>> >
>>>>>>>> > For me this is the next thing I will look at after merging
>>>>>>>> > requirements PR and Prod image PR (and those two are a bit
>>>>>>>> > pre-requisites to make it stable and fast). I want to switch to
>>>>>>>> the
>>>>>>>> > Production Image to run Kubernetes tests (they will be much
>>>>>>>> faster)
>>>>>>>> > and then we can work out much faster how to run them on Github
>>>>>>>> > Actions. Then we can add Helm charts to run all those different
>>>>>>>> > deployments. I am really looking forward to it!
>>>>>>>> >
>>>>>>>> > I hope - this weekend I move forward all of that and have a PR
>>>>>>>> that we
>>>>>>>> > can start running tests on and replacing the resources with helm
>>>>>>>> > chart.
>>>>>>>> >
>>>>>>>> > J.
>>>>>>>> >
>>>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>>>>>>>> > <da...@gmail.com> wrote:
>>>>>>>> > >
>>>>>>>> > > @Steven glad to help you out on that.
>>>>>>>> > >
>>>>>>>> > > For the moment we only use travis, but I believe @jarek has
>>>>>>>> been looking to move to GitHub CI/CD. Perhaps we can use this as our first
>>>>>>>> GitHub tests?
>>>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>>>>>>>> <st...@astronomer.io.invalid>, wrote:
>>>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to
>>>>>>>> test it. Where
>>>>>>>> > > > do we want to run it? Do you all just use Travis only?
>>>>>>>> > > >
>>>>>>>> > > > Steven
>>>>>>>> > > >
>>>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>>>>>>>> daniel.imberman@gmail.com>
>>>>>>>> > > > wrote:
>>>>>>>> > > >
>>>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go
>>>>>>>> even further
>>>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
>>>>>>>> autoscaling, and a
>>>>>>>> > > > > bunch of other configurations.
>>>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>>>>>>>> Jarek.Potiuk@polidea.com>,
>>>>>>>> > > > > wrote:
>>>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original
>>>>>>>> email :)
>>>>>>>> > > > > >
>>>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>>>>>>>> > > > > daniel.imberman@gmail.com> wrote:
>>>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We
>>>>>>>> just have
>>>>>>>> > > > > deployment files. If anything having the helm chart as a
>>>>>>>> part of the
>>>>>>>> > > > > airflow repo could mean that the helm chart becomes the
>>>>>>>> defacto system for
>>>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the
>>>>>>>> yams files and run
>>>>>>>> > > > > multiple k8s tests with different settings).
>>>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>>>>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
>>>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow
>>>>>>>> repo. We've also
>>>>>>>> > > > > been
>>>>>>>> > > > > > > > > building up a test suite that currently runs on
>>>>>>>> CircleCI and uses
>>>>>>>> > > > > kind
>>>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>>>>>>>> versions with
>>>>>>>> > > > > some
>>>>>>>> > > > > > > > > different settings. Right now we're mostly testing
>>>>>>>> the different
>>>>>>>> > > > > executors
>>>>>>>> > > > > > > > > since that has the biggest impact on what gets
>>>>>>>> deployed, but that
>>>>>>>> > > > > can be
>>>>>>>> > > > > > > > > expanded.
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow
>>>>>>>> for the tests?
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>>>>>>>> > > > > Jarek.Potiuk@polidea.com>
>>>>>>>> > > > > > > > > wrote:
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > > One thing for the donation.
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > Did you you want to have separate repository Greg
>>>>>>>> ?
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > I think we should simply create a folder in
>>>>>>>> Airflow repo and
>>>>>>>> > > > > keep it
>>>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am
>>>>>>>> going to switch our
>>>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will
>>>>>>>> make the tests
>>>>>>>> > > > > much
>>>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
>>>>>>>> automatically in
>>>>>>>> > > > > CI -
>>>>>>>> > > > > > > > > > for now we are using some custom Resource
>>>>>>>> definitions to start
>>>>>>>> > > > > Airflow
>>>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>>>>>>>> switch to
>>>>>>>> > > > > using the
>>>>>>>> > > > > > > > > > helm chart - this way we can test all three
>>>>>>>> things at once:
>>>>>>>> > > > > > > > > > - Kubernetes Executor
>>>>>>>> > > > > > > > > > - Dockerfile
>>>>>>>> > > > > > > > > > - Helm Chart
>>>>>>>> > > > > > > > > > and we could also add more tests - for example
>>>>>>>> testing different
>>>>>>>> > > > > > > > > > deployment options for the helm chart.
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help
>>>>>>>> with that -
>>>>>>>> > > > > > > > > > especially in terms of future changes and testing
>>>>>>>> them
>>>>>>>> > > > > automatically.
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > J.
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat
>>>>>>>> kyzy
>>>>>>>> > > > > > > > > > <ai...@apache.org> wrote:
>>>>>>>> > > > > > > > > > >
>>>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more
>>>>>>>> useful stuff for
>>>>>>>> > > > > the
>>>>>>>> > > > > > > > > > > community :)
>>>>>>>> > > > > > > > > > >
>>>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>>>>>>>> > > > > greg@astronomer.io>
>>>>>>>> > > > > > > > > > wrote:
>>>>>>>> > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as
>>>>>>>> an optional
>>>>>>>> > > > > Kubernetes
>>>>>>>> > > > > > > > > > > > CronJob (
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > >
>>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>>>>>>>> > > > > > > > > > )
>>>>>>>> > > > > > > > > > > > that we have run periodically to clean failed
>>>>>>>> pods up and
>>>>>>>> > > > > could stay
>>>>>>>> > > > > > > > > > > > separate.
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > The wait_for_migrations script could
>>>>>>>> definitely be pulled
>>>>>>>> > > > > into Airflow.
>>>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on
>>>>>>>> the scheduler (
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > >
>>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>>>>>>>> > > > > > > > > > )
>>>>>>>> > > > > > > > > > > > that runs the upgradedb command before
>>>>>>>> booting the
>>>>>>>> > > > > scheduler. This new
>>>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
>>>>>>>> initContainer on the
>>>>>>>> > > > > webserver and
>>>>>>>> > > > > > > > > > > > workers (
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > >
>>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>>>>>>>> > > > > > > > > > )
>>>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
>>>>>>>> potentially
>>>>>>>> > > > > long-running
>>>>>>>> > > > > > > > > > migration
>>>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
>>>>>>>> columns/tables
>>>>>>>> > > > > before the
>>>>>>>> > > > > > > > > > > > migrations run. This prevents these pods from
>>>>>>>> entering a
>>>>>>>> > > > > CrashLoop.
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk
>>>>>>>> <
>>>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>>>>>>>> > > > > > > > > > > > wrote:
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
>>>>>>>> currently
>>>>>>>> > > > > generated from
>>>>>>>> > > > > > > > > > > > > Dockerfiles
>>>>>>>> > > > > > > > > > > > > > in this repo
>>>>>>>> https://github.com/astronomer/ap-airflow
>>>>>>>> > > > > and get
>>>>>>>> > > > > > > > > > > > > published to
>>>>>>>> > > > > > > > > > > > > > DockerHub
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > >
>>>>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > For the most part those are typical
>>>>>>>> Airflow images.
>>>>>>>> > > > > There's an
>>>>>>>> > > > > > > > > > > > > entrypoint
>>>>>>>> > > > > > > > > > > > > > script that we include in the image that
>>>>>>>> handles waiting
>>>>>>>> > > > > for the
>>>>>>>> > > > > > > > > > > > > database
>>>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
>>>>>>>> pretty generic.
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > I already added waiting for the database
>>>>>>>> (both metadata
>>>>>>>> > > > > and celery
>>>>>>>> > > > > > > > > > URL) in
>>>>>>>> > > > > > > > > > > > > the PR:
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > >
>>>>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>>>>>>> > > > > > > > > > > > > .
>>>>>>>> > > > > > > > > > > > > It's functionally the same but more generic.
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > The only other
>>>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses
>>>>>>>> would be the
>>>>>>>> > > > > scripts in this
>>>>>>>> > > > > > > > > > repo
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
>>>>>>>> > > > > Our
>>>>>>>> > > > > > > > > > > > > Dockerfiles
>>>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are
>>>>>>>> used to
>>>>>>>> > > > > coordinate running
>>>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > I see two scripts:
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
>>>>>>>> needed to run in
>>>>>>>> > > > > airflow -
>>>>>>>> > > > > > > > > > this
>>>>>>>> > > > > > > > > > > > > could be run as a separate pod/container?
>>>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is
>>>>>>>> a good
>>>>>>>> > > > > candidate to add
>>>>>>>> > > > > > > > > > > > > *airflow
>>>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it
>>>>>>>> part of airflow
>>>>>>>> > > > > itself.
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > I think we also have to agree on the
>>>>>>>> Airflow version
>>>>>>>> > > > > supported by the
>>>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>>>>>>>> 1.10.10+ and we
>>>>>>>> > > > > > > > > > incorporate
>>>>>>>> > > > > > > > > > > > > all
>>>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>>>>>>>> > > > > wait_for_migration")
>>>>>>>> > > > > > > > > > into 2.0
>>>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and
>>>>>>>> helm chart for
>>>>>>>> > > > > those versions
>>>>>>>> > > > > > > > > > > > > only. That would help with people migrating
>>>>>>>> to the latest
>>>>>>>> > > > > version.
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > WDYT?
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>>>>>>>> Imberman <
>>>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>>>>>>>> > > > > > > > > > > > > > wrote:
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>>>>>>>> pairing an
>>>>>>>> > > > > official helm
>>>>>>>> > > > > > > > > > chart
>>>>>>>> > > > > > > > > > > > > > > with the official image would make for
>>>>>>>> a REALLY
>>>>>>>> > > > > powerful “up and
>>>>>>>> > > > > > > > > > > > > running
>>>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I
>>>>>>>> have also been
>>>>>>>> > > > > looking into
>>>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
>>>>>>>> create custom
>>>>>>>> > > > > controllers
>>>>>>>> > > > > > > > > > from
>>>>>>>> > > > > > > > > > > > > helm
>>>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
>>>>>>>> punch from the
>>>>>>>> > > > > same code
>>>>>>>> > > > > > > > > > base
>>>>>>>> > > > > > > > > > > > > :).
>>>>>>>> > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s
>>>>>>>> no issues, can
>>>>>>>> > > > > we please
>>>>>>>> > > > > > > > > > start
>>>>>>>> > > > > > > > > > > > > > the
>>>>>>>> > > > > > > > > > > > > > > process of donation?
>>>>>>>> > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
>>>>>>>> > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > Daniel
>>>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
>>>>>>>> Potiuk <
>>>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>>>>>>>> > > > > > > > > > > > > > > wrote:
>>>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>>>>>>>> official image
>>>>>>>> > > > > we have
>>>>>>>> > > > > > > > > > work in
>>>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>>>>>>>> Astronomer's
>>>>>>>> > > > > image while
>>>>>>>> > > > > > > > > > > > > preparing
>>>>>>>> > > > > > > > > > > > > > my
>>>>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>>>>>>>> needed to make
>>>>>>>> > > > > it works
>>>>>>>> > > > > > > > > > with
>>>>>>>> > > > > > > > > > > > > the
>>>>>>>> > > > > > > > > > > > > > > helm
>>>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
>>>>>>>> collaborate on that.
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > PR here:
>>>>>>>> https://github.com/apache/airflow/pull/7832
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > J.
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil
>>>>>>>> Naik <
>>>>>>>> > > > > kaxilnaik@gmail.com
>>>>>>>> > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > wrote:
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>>>>>>>> tomasz.urbaszek@polidea.com> :
>>>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
>>>>>>>> > > > > https://github.com/astronomer/airflow-chart
>>>>>>>> > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
>>>>>>>> Tomasz Urbaszek <
>>>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
>>>>>>>> > > > > > > > > > > > > > > > > wrote:
>>>>>>>> > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is
>>>>>>>> something our
>>>>>>>> > > > > community needs!
>>>>>>>> > > > > > > > > > Using
>>>>>>>> > > > > > > > > > > > > your
>>>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot
>>>>>>>> of sens to me
>>>>>>>> > > > > because as
>>>>>>>> > > > > > > > > > you
>>>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image
>>>>>>>> do you use?
>>>>>>>> > > > > Also, would you
>>>>>>>> > > > > > > > > > > > > mind
>>>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > Tomek
>>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM
>>>>>>>> Greg Neiheisel
>>>>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid>
>>>>>>>> wrote:
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
>>>>>>>> Astronomer, we’ve
>>>>>>>> > > > > created,
>>>>>>>> > > > > > > > > > managed,
>>>>>>>> > > > > > > > > > > > > > and
>>>>>>>> > > > > > > > > > > > > > > > > > hardened
>>>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart
>>>>>>>> for Airflow (
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> https://github.com/astronomer/airflow-chart)
>>>>>>>> > > > > that is
>>>>>>>> > > > > > > > > > being
>>>>>>>> > > > > > > > > > > > > used
>>>>>>>> > > > > > > > > > > > > > by
>>>>>>>> > > > > > > > > > > > > > > > > both
>>>>>>>> > > > > > > > > > > > > > > > > > our
>>>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers.
>>>>>>>> This chart is
>>>>>>>> > > > > > > > > > battle-tested and
>>>>>>>> > > > > > > > > > > > > > > running
>>>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments
>>>>>>>> of varying
>>>>>>>> > > > > sizes and
>>>>>>>> > > > > > > > > > runtime
>>>>>>>> > > > > > > > > > > > > > > > > > environments.
>>>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to
>>>>>>>> encapsulate the issues
>>>>>>>> > > > > that Airflow
>>>>>>>> > > > > > > > > > > > > users
>>>>>>>> > > > > > > > > > > > > > run
>>>>>>>> > > > > > > > > > > > > > > > > into
>>>>>>>> > > > > > > > > > > > > > > > > > in
>>>>>>>> > > > > > > > > > > > > > > > > > > the real world.
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally
>>>>>>>> developed
>>>>>>>> > > > > internally for
>>>>>>>> > > > > > > > > > our
>>>>>>>> > > > > > > > > > > > > > > Astronomer
>>>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
>>>>>>>> decoupled the chart
>>>>>>>> > > > > from the
>>>>>>>> > > > > > > > > > rest of
>>>>>>>> > > > > > > > > > > > > our
>>>>>>>> > > > > > > > > > > > > > > > > > platform
>>>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the
>>>>>>>> greater Airflow
>>>>>>>> > > > > community. With
>>>>>>>> > > > > > > > > > these
>>>>>>>> > > > > > > > > > > > > > > changes
>>>>>>>> > > > > > > > > > > > > > > > > in
>>>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
>>>>>>>> conversation about
>>>>>>>> > > > > donating this
>>>>>>>> > > > > > > > > > > > > chart
>>>>>>>> > > > > > > > > > > > > > to
>>>>>>>> > > > > > > > > > > > > > > the
>>>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of
>>>>>>>> the chart are:
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With
>>>>>>>> zero
>>>>>>>> > > > > configuration, a user
>>>>>>>> > > > > > > > > > > > > will
>>>>>>>> > > > > > > > > > > > > > get
>>>>>>>> > > > > > > > > > > > > > > > > a
>>>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default
>>>>>>>> user and the
>>>>>>>> > > > > > > > > > KubernetesExecutor
>>>>>>>> > > > > > > > > > > > > > ready
>>>>>>>> > > > > > > > > > > > > > > > > to
>>>>>>>> > > > > > > > > > > > > > > > > > run
>>>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
>>>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>>>>>>>> optional KEDA
>>>>>>>> > > > > > > > > > autoscaling) and
>>>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer.
>>>>>>>> We use this to
>>>>>>>> > > > > share a
>>>>>>>> > > > > > > > > > > > > > configurable
>>>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
>>>>>>>> deployment. Useful
>>>>>>>> > > > > for limiting
>>>>>>>> > > > > > > > > > > > > > > connections to
>>>>>>>> > > > > > > > > > > > > > > > > > the
>>>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A
>>>>>>>> user can push a
>>>>>>>> > > > > newer
>>>>>>>> > > > > > > > > > version
>>>>>>>> > > > > > > > > > > > > of
>>>>>>>> > > > > > > > > > > > > > > > > > Airflow
>>>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and
>>>>>>>> migrations will
>>>>>>>> > > > > > > > > > automatically run
>>>>>>>> > > > > > > > > > > > > > > > > > cleanly.
>>>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support.
>>>>>>>> Optionally install and
>>>>>>>> > > > > configure a
>>>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
>>>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
>>>>>>>> expose them to
>>>>>>>> > > > > Prometheus
>>>>>>>> > > > > > > > > > > > > > > > > automatically.
>>>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>>>>>>>> control the
>>>>>>>> > > > > ResourceQuotas
>>>>>>>> > > > > > > > > > and
>>>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment
>>>>>>>> so that no
>>>>>>>> > > > > deployment can
>>>>>>>> > > > > > > > > > > > > > overload
>>>>>>>> > > > > > > > > > > > > > > a
>>>>>>>> > > > > > > > > > > > > > > > > > > cluster.
>>>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>>>>>>>> support.
>>>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>>>>>>>> Sometimes
>>>>>>>> > > > > > > > > > KubernetesExecutor and
>>>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail
>>>>>>>> for reasons
>>>>>>>> > > > > other than the
>>>>>>>> > > > > > > > > > > > > actual
>>>>>>>> > > > > > > > > > > > > > > > > > task.
>>>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things
>>>>>>>> clean in
>>>>>>>> > > > > Kubernetes.
>>>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally
>>>>>>>> in KIND
>>>>>>>> > > > > (Kubernetes in
>>>>>>>> > > > > > > > > > Docker).
>>>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across
>>>>>>>> many Kubernetes
>>>>>>>> > > > > versions
>>>>>>>> > > > > > > > > > with
>>>>>>>> > > > > > > > > > > > > Helm
>>>>>>>> > > > > > > > > > > > > > 2
>>>>>>>> > > > > > > > > > > > > > > > > > and 3
>>>>>>>> > > > > > > > > > > > > > > > > > > support.
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest
>>>>>>>> and most
>>>>>>>> > > > > reliable way to
>>>>>>>> > > > > > > > > > deploy
>>>>>>>> > > > > > > > > > > > > > DAGs
>>>>>>>> > > > > > > > > > > > > > > to
>>>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at
>>>>>>>> scale is to
>>>>>>>> > > > > package them
>>>>>>>> > > > > > > > > > into
>>>>>>>> > > > > > > > > > > > > the
>>>>>>>> > > > > > > > > > > > > > > actual
>>>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared
>>>>>>>> this chart
>>>>>>>> > > > > towards that
>>>>>>>> > > > > > > > > > > > > method of
>>>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other
>>>>>>>> methods should
>>>>>>>> > > > > be
>>>>>>>> > > > > > > > > > > > > straightforward.
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>>>>>>>> community and
>>>>>>>> > > > > would love
>>>>>>>> > > > > > > > > > to
>>>>>>>> > > > > > > > > > > > > see
>>>>>>>> > > > > > > > > > > > > > > this
>>>>>>>> > > > > > > > > > > > > > > > > > chart
>>>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and
>>>>>>>> running on
>>>>>>>> > > > > Kubernetes!
>>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > > > > --
>>>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief
>>>>>>>> Architect
>>>>>>>> > > > > Astronomer.io
>>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > --
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
>>>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>>>>> Principal
>>>>>>>> > > > > Software
>>>>>>>> > > > > > > > > > Engineer
>>>>>>>> > > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
>>>>>>>> <+48%20660%20796%20129>
>>>>>>>> > > > > <+48660796129
>>>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
>>>>>>>> https://www.polidea.com/>
>>>>>>>> > > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > > --
>>>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>>>>> Astronomer.io
>>>>>>>> > > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > --
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > Jarek Potiuk
>>>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>>>>> Principal Software
>>>>>>>> > > > > Engineer
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>>>>>>> <+48660796129
>>>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>>>>> > > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > > > > --
>>>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>>>>> Astronomer.io
>>>>>>>> > > > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > --
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > Jarek Potiuk
>>>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > > > M: +48 660 796 129
>>>>>>>> > > > > > > > > >
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > >
>>>>>>>> > > > > > > > > --
>>>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>>>>> > > > > >
>>>>>>>> > > > > >
>>>>>>>> > > > > > --
>>>>>>>> > > > > > Jarek Potiuk
>>>>>>>> > > > > > Polidea | Principal Software Engineer
>>>>>>>> > > > > > M: +48 660 796 129
>>>>>>>> > > > > >
>>>>>>>> > > > >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> >
>>>>>>>> > Jarek Potiuk
>>>>>>>> > Polidea | Principal Software Engineer
>>>>>>>> >
>>>>>>>> > M: +48 660 796 129
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Jarek Potiuk
>>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>>
>>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>
>>>>
>>>> --
>>>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Greg Neiheisel <gr...@astronomer.io.INVALID>.
Awesome Jarek. I just opened up a draft PR that adds the chart into the
Airflow repo. https://github.com/apache/airflow/pull/8777
It still contains some of our packaging, testing and releasing scripts. I
figure they can serve as a starting point to discuss how we'd want to
replicate into Airflow infrastructure.
Looking forward to getting some feedback and moving this along!
On Tue, Apr 28, 2020 at 8:00 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hey Greg,
>
> I just wanted to update you with some recent changes I've done to automate
> most of the image builds stuff we have in Airflow. That might be useful for
> your efforts.
>
> I am adding (and I will fix some of them very soon) issues based on
> feedback from people on the #prod-docker-image channel:
> https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
> But in the meantime all the build/CI infrastructure has nicely fallen into
> place and It's almost complete (one thing missing but very simple is
> automated builds of release images - for now this is manual but it is super
> easy change and I will add it before 1.10.11 for sure).
>
> I described the whole build infrastructure here:
> https://github.com/apache/airflow/blob/master/CI.rst including how image
> caching works, how nightly builds and images are triggered. I hope it will
> be helpful in the helm-chart work. Any questions or asks for the image -
> please let me know, it is super easy to update the image now when we have
> infrastructure to build it automatically integrated well with Github
> Actions and DockerHub.
>
> J.
>
>
>
>
>
>
>
>
> On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Awesome, thanks for the update!
>>
>> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io>
>> wrote:
>>
>>> Hey everyone, just wanted to post a quick update on the helm chart.
>>> We're in the process of removing any references to Astronomer specific
>>> things and making them more general features. That work is happening on
>>> this PR here https://github.com/astronomer/airflow-chart/pull/66.
>>> Hoping to wrap up the last few outstanding items and open up a PR to
>>> Airflow next week sometime!
>>>
>>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
>>> daniel.imberman@gmail.com> wrote:
>>>
>>>> Hi Jarek,
>>>>
>>>> Congrats on getting the image out! This is going to make a lot of
>>>> things easier :). We’ve actually been working on separating out KinD as
>>>> well (and in fact moving it out of travis and into GitHub issues). Let me
>>>> circle back with Steven (the engineer who’s been working on that) and get a
>>>> status report from him.
>>>>
>>>> I will also take some time today to get our helm chart to work with
>>>> the airflow master image. It’s been a bit crazy around here in the last few
>>>> weeks but I am consistently pushing to make sure we make the final changes
>>>> of ripping out any remaining platform components before we donate :).
>>>>
>>>> I agree that docker hub automation will be great. This will also allow
>>>> us to have a nightly build for any integration/UI tests we build.
>>>>
>>>> Thanks again Jarek!
>>>>
>>>> Daniel
>>>>
>>>> via Newton Mail
>>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>>>>
>>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>> Hello Greg and Daniel,
>>>>
>>>> Production image is merged and it's also going to be released in
>>>> 1.10.10. For now, I will manually tag the release in DockerHub when we
>>>> release but we will automate that in the future. Currently, we have
>>>> *airflow:master* and *airflow:v1-10-test* images (for all supported
>>>> python versions) that you can use for testing with the Helm chart!
>>>>
>>>> The next things from my side:
>>>>
>>>> * I will automate DockerHub building for the prod image from master -
>>>> for now only CI is automated but we need to add prod image as well
>>>> * I will move kind-cluster creation outside of the container and use
>>>> production image for our CI Kubernetes tests. I have a WIP branch on it and
>>>> I think I should have it early next week
>>>> * The next step will be to switch our CI to Github Actions finally. I
>>>> think we will run both Travis and GA for a short time to make sure it works
>>>> * I will ask people for options they would like to have in the image -
>>>> additional packages etc. so that I can work on the next iteration of the
>>>> image. Also if you have any needs to see for the Helm integration, I am
>>>> super happy to incorporate them in the next version. I plan to keep the
>>>> "master" and "v1-10-test" images updated with any new changes and push
>>>> images regularly
>>>> * I am happy to help with the Helm Chart, so let me know if you need
>>>> anything!
>>>>
>>>> J.
>>>>
>>>>
>>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>>> Hello Daniel and Greg.
>>>>>
>>>>> I think the production image for Airflow is quite ready for the final
>>>>> review and merge. I did some tests and I think it's quite complete now.
>>>>>
>>>>> The PR is running it's Travis build:
>>>>> https://github.com/apache/airflow/pull/7832
>>>>>
>>>>> I already pushed the image to DockerHub - so that the next time it is
>>>>> merged It will use it as a base and will be rather quickly built using
>>>>> those images as a cache. Once it is merged, I will add the automated build
>>>>> in DockerHub and merge it to 1.10. As the next step I also plan to use the
>>>>> prod image for our Kubernetes tests (that will help with moving to the
>>>>> GitHub Actions from Travis).
>>>>>
>>>>> I added quite a comprehensive documentation - explaining all ins- and
>>>>> outs- of both CI and PROD images
>>>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
>>>>> .
>>>>>
>>>>> You can build it yourself using manual build process described in the
>>>>> doc or (easier) using Breeze. That includes using the same Dockerfile to
>>>>> build older Airflow version from the 1.10.* line.
>>>>>
>>>>> Feel free to use the new image in the Helm Chart you have - happy to
>>>>> review the PRs. The image is available at 'apache/airflow:master-python3.6'
>>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
>>>>> release 1.10.10 we will also have it automatically published for 1.10.10
>>>>> and beyond.
>>>>>
>>>>> Happy to hear your review comments and hope to merge it very soon.
>>>>> J.
>>>>>
>>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <Ja...@polidea.com>
>>>>> wrote:
>>>>>
>>>>>> Absolutely! Please do :). The more eyes on this the better!
>>>>>>
>>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
>>>>>> daniel.imberman@gmail.com> wrote:
>>>>>>
>>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
>>>>>>> actions today. If we can get that working, we can donate the helm chart and
>>>>>>> possibly even deprecate the old Kubernetes tests in the process.
>>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
>>>>>>> Jarek.Potiuk@polidea.com>, wrote:
>>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker
>>>>>>> there
>>>>>>> > was precisely Kubernetes tests and specifically running kind
>>>>>>> cluster
>>>>>>> > in the Github Actions
>>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
>>>>>>> meantime
>>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this very
>>>>>>> > thing today at slack:
>>>>>>> >
>>>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>>>>>>> >
>>>>>>> > For me this is the next thing I will look at after merging
>>>>>>> > requirements PR and Prod image PR (and those two are a bit
>>>>>>> > pre-requisites to make it stable and fast). I want to switch to the
>>>>>>> > Production Image to run Kubernetes tests (they will be much faster)
>>>>>>> > and then we can work out much faster how to run them on Github
>>>>>>> > Actions. Then we can add Helm charts to run all those different
>>>>>>> > deployments. I am really looking forward to it!
>>>>>>> >
>>>>>>> > I hope - this weekend I move forward all of that and have a PR
>>>>>>> that we
>>>>>>> > can start running tests on and replacing the resources with helm
>>>>>>> > chart.
>>>>>>> >
>>>>>>> > J.
>>>>>>> >
>>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>>>>>>> > <da...@gmail.com> wrote:
>>>>>>> > >
>>>>>>> > > @Steven glad to help you out on that.
>>>>>>> > >
>>>>>>> > > For the moment we only use travis, but I believe @jarek has been
>>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first
>>>>>>> GitHub tests?
>>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>>>>>>> <st...@astronomer.io.invalid>, wrote:
>>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to
>>>>>>> test it. Where
>>>>>>> > > > do we want to run it? Do you all just use Travis only?
>>>>>>> > > >
>>>>>>> > > > Steven
>>>>>>> > > >
>>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>>>>>>> daniel.imberman@gmail.com>
>>>>>>> > > > wrote:
>>>>>>> > > >
>>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go
>>>>>>> even further
>>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
>>>>>>> autoscaling, and a
>>>>>>> > > > > bunch of other configurations.
>>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>>>>>>> Jarek.Potiuk@polidea.com>,
>>>>>>> > > > > wrote:
>>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original
>>>>>>> email :)
>>>>>>> > > > > >
>>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>>>>>>> > > > > daniel.imberman@gmail.com> wrote:
>>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We
>>>>>>> just have
>>>>>>> > > > > deployment files. If anything having the helm chart as a
>>>>>>> part of the
>>>>>>> > > > > airflow repo could mean that the helm chart becomes the
>>>>>>> defacto system for
>>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the
>>>>>>> yams files and run
>>>>>>> > > > > multiple k8s tests with different settings).
>>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>>>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
>>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow
>>>>>>> repo. We've also
>>>>>>> > > > > been
>>>>>>> > > > > > > > > building up a test suite that currently runs on
>>>>>>> CircleCI and uses
>>>>>>> > > > > kind
>>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>>>>>>> versions with
>>>>>>> > > > > some
>>>>>>> > > > > > > > > different settings. Right now we're mostly testing
>>>>>>> the different
>>>>>>> > > > > executors
>>>>>>> > > > > > > > > since that has the biggest impact on what gets
>>>>>>> deployed, but that
>>>>>>> > > > > can be
>>>>>>> > > > > > > > > expanded.
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow
>>>>>>> for the tests?
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>>>>>>> > > > > Jarek.Potiuk@polidea.com>
>>>>>>> > > > > > > > > wrote:
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > > One thing for the donation.
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > Did you you want to have separate repository Greg ?
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > I think we should simply create a folder in
>>>>>>> Airflow repo and
>>>>>>> > > > > keep it
>>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am
>>>>>>> going to switch our
>>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will
>>>>>>> make the tests
>>>>>>> > > > > much
>>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
>>>>>>> automatically in
>>>>>>> > > > > CI -
>>>>>>> > > > > > > > > > for now we are using some custom Resource
>>>>>>> definitions to start
>>>>>>> > > > > Airflow
>>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>>>>>>> switch to
>>>>>>> > > > > using the
>>>>>>> > > > > > > > > > helm chart - this way we can test all three things
>>>>>>> at once:
>>>>>>> > > > > > > > > > - Kubernetes Executor
>>>>>>> > > > > > > > > > - Dockerfile
>>>>>>> > > > > > > > > > - Helm Chart
>>>>>>> > > > > > > > > > and we could also add more tests - for example
>>>>>>> testing different
>>>>>>> > > > > > > > > > deployment options for the helm chart.
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help
>>>>>>> with that -
>>>>>>> > > > > > > > > > especially in terms of future changes and testing
>>>>>>> them
>>>>>>> > > > > automatically.
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > J.
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat
>>>>>>> kyzy
>>>>>>> > > > > > > > > > <ai...@apache.org> wrote:
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more
>>>>>>> useful stuff for
>>>>>>> > > > > the
>>>>>>> > > > > > > > > > > community :)
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>>>>>>> > > > > greg@astronomer.io>
>>>>>>> > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as
>>>>>>> an optional
>>>>>>> > > > > Kubernetes
>>>>>>> > > > > > > > > > > > CronJob (
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > >
>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>>>>>>> > > > > > > > > > )
>>>>>>> > > > > > > > > > > > that we have run periodically to clean failed
>>>>>>> pods up and
>>>>>>> > > > > could stay
>>>>>>> > > > > > > > > > > > separate.
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > The wait_for_migrations script could
>>>>>>> definitely be pulled
>>>>>>> > > > > into Airflow.
>>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on the
>>>>>>> scheduler (
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > >
>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>>>>>>> > > > > > > > > > )
>>>>>>> > > > > > > > > > > > that runs the upgradedb command before booting
>>>>>>> the
>>>>>>> > > > > scheduler. This new
>>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
>>>>>>> initContainer on the
>>>>>>> > > > > webserver and
>>>>>>> > > > > > > > > > > > workers (
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > >
>>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>>>>>>> > > > > > > > > > )
>>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
>>>>>>> potentially
>>>>>>> > > > > long-running
>>>>>>> > > > > > > > > > migration
>>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
>>>>>>> columns/tables
>>>>>>> > > > > before the
>>>>>>> > > > > > > > > > > > migrations run. This prevents these pods from
>>>>>>> entering a
>>>>>>> > > > > CrashLoop.
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>>>>>>> > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
>>>>>>> currently
>>>>>>> > > > > generated from
>>>>>>> > > > > > > > > > > > > Dockerfiles
>>>>>>> > > > > > > > > > > > > > in this repo
>>>>>>> https://github.com/astronomer/ap-airflow
>>>>>>> > > > > and get
>>>>>>> > > > > > > > > > > > > published to
>>>>>>> > > > > > > > > > > > > > DockerHub
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > >
>>>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > For the most part those are typical
>>>>>>> Airflow images.
>>>>>>> > > > > There's an
>>>>>>> > > > > > > > > > > > > entrypoint
>>>>>>> > > > > > > > > > > > > > script that we include in the image that
>>>>>>> handles waiting
>>>>>>> > > > > for the
>>>>>>> > > > > > > > > > > > > database
>>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
>>>>>>> pretty generic.
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > I already added waiting for the database
>>>>>>> (both metadata
>>>>>>> > > > > and celery
>>>>>>> > > > > > > > > > URL) in
>>>>>>> > > > > > > > > > > > > the PR:
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > >
>>>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>>>>>> > > > > > > > > > > > > .
>>>>>>> > > > > > > > > > > > > It's functionally the same but more generic.
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > The only other
>>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses
>>>>>>> would be the
>>>>>>> > > > > scripts in this
>>>>>>> > > > > > > > > > repo
>>>>>>> > > > > > > > > > > > > >
>>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
>>>>>>> > > > > Our
>>>>>>> > > > > > > > > > > > > Dockerfiles
>>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are
>>>>>>> used to
>>>>>>> > > > > coordinate running
>>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > I see two scripts:
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
>>>>>>> needed to run in
>>>>>>> > > > > airflow -
>>>>>>> > > > > > > > > > this
>>>>>>> > > > > > > > > > > > > could be run as a separate pod/container?
>>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is
>>>>>>> a good
>>>>>>> > > > > candidate to add
>>>>>>> > > > > > > > > > > > > *airflow
>>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it
>>>>>>> part of airflow
>>>>>>> > > > > itself.
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > I think we also have to agree on the Airflow
>>>>>>> version
>>>>>>> > > > > supported by the
>>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>>>>>>> 1.10.10+ and we
>>>>>>> > > > > > > > > > incorporate
>>>>>>> > > > > > > > > > > > > all
>>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>>>>>>> > > > > wait_for_migration")
>>>>>>> > > > > > > > > > into 2.0
>>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and
>>>>>>> helm chart for
>>>>>>> > > > > those versions
>>>>>>> > > > > > > > > > > > > only. That would help with people migrating
>>>>>>> to the latest
>>>>>>> > > > > version.
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > WDYT?
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>>>>>>> Imberman <
>>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>>>>>>> > > > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>>>>>>> pairing an
>>>>>>> > > > > official helm
>>>>>>> > > > > > > > > > chart
>>>>>>> > > > > > > > > > > > > > > with the official image would make for a
>>>>>>> REALLY
>>>>>>> > > > > powerful “up and
>>>>>>> > > > > > > > > > > > > running
>>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have
>>>>>>> also been
>>>>>>> > > > > looking into
>>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
>>>>>>> create custom
>>>>>>> > > > > controllers
>>>>>>> > > > > > > > > > from
>>>>>>> > > > > > > > > > > > > helm
>>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
>>>>>>> punch from the
>>>>>>> > > > > same code
>>>>>>> > > > > > > > > > base
>>>>>>> > > > > > > > > > > > > :).
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s
>>>>>>> no issues, can
>>>>>>> > > > > we please
>>>>>>> > > > > > > > > > start
>>>>>>> > > > > > > > > > > > > > the
>>>>>>> > > > > > > > > > > > > > > process of donation?
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > Daniel
>>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
>>>>>>> Potiuk <
>>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>>>>>>> > > > > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>>>>>>> official image
>>>>>>> > > > > we have
>>>>>>> > > > > > > > > > work in
>>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>>>>>>> Astronomer's
>>>>>>> > > > > image while
>>>>>>> > > > > > > > > > > > > preparing
>>>>>>> > > > > > > > > > > > > > my
>>>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>>>>>>> needed to make
>>>>>>> > > > > it works
>>>>>>> > > > > > > > > > with
>>>>>>> > > > > > > > > > > > > the
>>>>>>> > > > > > > > > > > > > > > helm
>>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
>>>>>>> collaborate on that.
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > PR here:
>>>>>>> https://github.com/apache/airflow/pull/7832
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > J.
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil
>>>>>>> Naik <
>>>>>>> > > > > kaxilnaik@gmail.com
>>>>>>> > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>>>>>>> tomasz.urbaszek@polidea.com> :
>>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
>>>>>>> > > > > https://github.com/astronomer/airflow-chart
>>>>>>> > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
>>>>>>> Tomasz Urbaszek <
>>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
>>>>>>> > > > > > > > > > > > > > > > > wrote:
>>>>>>> > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is
>>>>>>> something our
>>>>>>> > > > > community needs!
>>>>>>> > > > > > > > > > Using
>>>>>>> > > > > > > > > > > > > your
>>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot
>>>>>>> of sens to me
>>>>>>> > > > > because as
>>>>>>> > > > > > > > > > you
>>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image
>>>>>>> do you use?
>>>>>>> > > > > Also, would you
>>>>>>> > > > > > > > > > > > > mind
>>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > Tomek
>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM
>>>>>>> Greg Neiheisel
>>>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid>
>>>>>>> wrote:
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
>>>>>>> Astronomer, we’ve
>>>>>>> > > > > created,
>>>>>>> > > > > > > > > > managed,
>>>>>>> > > > > > > > > > > > > > and
>>>>>>> > > > > > > > > > > > > > > > > > hardened
>>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart
>>>>>>> for Airflow (
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> https://github.com/astronomer/airflow-chart)
>>>>>>> > > > > that is
>>>>>>> > > > > > > > > > being
>>>>>>> > > > > > > > > > > > > used
>>>>>>> > > > > > > > > > > > > > by
>>>>>>> > > > > > > > > > > > > > > > > both
>>>>>>> > > > > > > > > > > > > > > > > > our
>>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers.
>>>>>>> This chart is
>>>>>>> > > > > > > > > > battle-tested and
>>>>>>> > > > > > > > > > > > > > > running
>>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments
>>>>>>> of varying
>>>>>>> > > > > sizes and
>>>>>>> > > > > > > > > > runtime
>>>>>>> > > > > > > > > > > > > > > > > > environments.
>>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to
>>>>>>> encapsulate the issues
>>>>>>> > > > > that Airflow
>>>>>>> > > > > > > > > > > > > users
>>>>>>> > > > > > > > > > > > > > run
>>>>>>> > > > > > > > > > > > > > > > > into
>>>>>>> > > > > > > > > > > > > > > > > > in
>>>>>>> > > > > > > > > > > > > > > > > > > the real world.
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally
>>>>>>> developed
>>>>>>> > > > > internally for
>>>>>>> > > > > > > > > > our
>>>>>>> > > > > > > > > > > > > > > Astronomer
>>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
>>>>>>> decoupled the chart
>>>>>>> > > > > from the
>>>>>>> > > > > > > > > > rest of
>>>>>>> > > > > > > > > > > > > our
>>>>>>> > > > > > > > > > > > > > > > > > platform
>>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater
>>>>>>> Airflow
>>>>>>> > > > > community. With
>>>>>>> > > > > > > > > > these
>>>>>>> > > > > > > > > > > > > > > changes
>>>>>>> > > > > > > > > > > > > > > > > in
>>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
>>>>>>> conversation about
>>>>>>> > > > > donating this
>>>>>>> > > > > > > > > > > > > chart
>>>>>>> > > > > > > > > > > > > > to
>>>>>>> > > > > > > > > > > > > > > the
>>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of the
>>>>>>> chart are:
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With
>>>>>>> zero
>>>>>>> > > > > configuration, a user
>>>>>>> > > > > > > > > > > > > will
>>>>>>> > > > > > > > > > > > > > get
>>>>>>> > > > > > > > > > > > > > > > > a
>>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default
>>>>>>> user and the
>>>>>>> > > > > > > > > > KubernetesExecutor
>>>>>>> > > > > > > > > > > > > > ready
>>>>>>> > > > > > > > > > > > > > > > > to
>>>>>>> > > > > > > > > > > > > > > > > > run
>>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
>>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>>>>>>> optional KEDA
>>>>>>> > > > > > > > > > autoscaling) and
>>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer.
>>>>>>> We use this to
>>>>>>> > > > > share a
>>>>>>> > > > > > > > > > > > > > configurable
>>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
>>>>>>> deployment. Useful
>>>>>>> > > > > for limiting
>>>>>>> > > > > > > > > > > > > > > connections to
>>>>>>> > > > > > > > > > > > > > > > > > the
>>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A
>>>>>>> user can push a
>>>>>>> > > > > newer
>>>>>>> > > > > > > > > > version
>>>>>>> > > > > > > > > > > > > of
>>>>>>> > > > > > > > > > > > > > > > > > Airflow
>>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and
>>>>>>> migrations will
>>>>>>> > > > > > > > > > automatically run
>>>>>>> > > > > > > > > > > > > > > > > > cleanly.
>>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
>>>>>>> install and
>>>>>>> > > > > configure a
>>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
>>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
>>>>>>> expose them to
>>>>>>> > > > > Prometheus
>>>>>>> > > > > > > > > > > > > > > > > automatically.
>>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>>>>>>> control the
>>>>>>> > > > > ResourceQuotas
>>>>>>> > > > > > > > > > and
>>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment
>>>>>>> so that no
>>>>>>> > > > > deployment can
>>>>>>> > > > > > > > > > > > > > overload
>>>>>>> > > > > > > > > > > > > > > a
>>>>>>> > > > > > > > > > > > > > > > > > > cluster.
>>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>>>>>>> support.
>>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>>>>>>> Sometimes
>>>>>>> > > > > > > > > > KubernetesExecutor and
>>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail
>>>>>>> for reasons
>>>>>>> > > > > other than the
>>>>>>> > > > > > > > > > > > > actual
>>>>>>> > > > > > > > > > > > > > > > > > task.
>>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things
>>>>>>> clean in
>>>>>>> > > > > Kubernetes.
>>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally in
>>>>>>> KIND
>>>>>>> > > > > (Kubernetes in
>>>>>>> > > > > > > > > > Docker).
>>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across
>>>>>>> many Kubernetes
>>>>>>> > > > > versions
>>>>>>> > > > > > > > > > with
>>>>>>> > > > > > > > > > > > > Helm
>>>>>>> > > > > > > > > > > > > > 2
>>>>>>> > > > > > > > > > > > > > > > > > and 3
>>>>>>> > > > > > > > > > > > > > > > > > > support.
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest
>>>>>>> and most
>>>>>>> > > > > reliable way to
>>>>>>> > > > > > > > > > deploy
>>>>>>> > > > > > > > > > > > > > DAGs
>>>>>>> > > > > > > > > > > > > > > to
>>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at
>>>>>>> scale is to
>>>>>>> > > > > package them
>>>>>>> > > > > > > > > > into
>>>>>>> > > > > > > > > > > > > the
>>>>>>> > > > > > > > > > > > > > > actual
>>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared
>>>>>>> this chart
>>>>>>> > > > > towards that
>>>>>>> > > > > > > > > > > > > method of
>>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other
>>>>>>> methods should
>>>>>>> > > > > be
>>>>>>> > > > > > > > > > > > > straightforward.
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>>>>>>> community and
>>>>>>> > > > > would love
>>>>>>> > > > > > > > > > to
>>>>>>> > > > > > > > > > > > > see
>>>>>>> > > > > > > > > > > > > > > this
>>>>>>> > > > > > > > > > > > > > > > > > chart
>>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and
>>>>>>> running on
>>>>>>> > > > > Kubernetes!
>>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > > > > --
>>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief
>>>>>>> Architect
>>>>>>> > > > > Astronomer.io
>>>>>>> > > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > --
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
>>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>>>> Principal
>>>>>>> > > > > Software
>>>>>>> > > > > > > > > > Engineer
>>>>>>> > > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
>>>>>>> <+48%20660%20796%20129>
>>>>>>> > > > > <+48660796129
>>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
>>>>>>> https://www.polidea.com/>
>>>>>>> > > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > > --
>>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>>>> Astronomer.io
>>>>>>> > > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > --
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > Jarek Potiuk
>>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>>>> Principal Software
>>>>>>> > > > > Engineer
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>>>>>> <+48660796129
>>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>>>> > > > > > > > > > > > >
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > > > > --
>>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>>>> Astronomer.io
>>>>>>> > > > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > --
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > Jarek Potiuk
>>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > > > M: +48 660 796 129
>>>>>>> > > > > > > > > >
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > >
>>>>>>> > > > > > > > > --
>>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>>>> > > > > >
>>>>>>> > > > > >
>>>>>>> > > > > > --
>>>>>>> > > > > > Jarek Potiuk
>>>>>>> > > > > > Polidea | Principal Software Engineer
>>>>>>> > > > > > M: +48 660 796 129
>>>>>>> > > > > >
>>>>>>> > > > >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> >
>>>>>>> > Jarek Potiuk
>>>>>>> > Polidea | Principal Software Engineer
>>>>>>> >
>>>>>>> > M: +48 660 796 129
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Jarek Potiuk
>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>>
>>>>>> M: +48 660 796 129 <+48660796129>
>>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>>
>>> --
>>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hey Greg,
I just wanted to update you with some recent changes I've done to automate
most of the image builds stuff we have in Airflow. That might be useful for
your efforts.
I am adding (and I will fix some of them very soon) issues based on
feedback from people on the #prod-docker-image channel:
https://github.com/apache/airflow/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%3Aproduction-image
But in the meantime all the build/CI infrastructure has nicely fallen into
place and It's almost complete (one thing missing but very simple is
automated builds of release images - for now this is manual but it is super
easy change and I will add it before 1.10.11 for sure).
I described the whole build infrastructure here:
https://github.com/apache/airflow/blob/master/CI.rst including how image
caching works, how nightly builds and images are triggered. I hope it will
be helpful in the helm-chart work. Any questions or asks for the image -
please let me know, it is super easy to update the image now when we have
infrastructure to build it automatically integrated well with Github
Actions and DockerHub.
J.
On Fri, Apr 10, 2020 at 7:23 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Awesome, thanks for the update!
>
> On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io> wrote:
>
>> Hey everyone, just wanted to post a quick update on the helm chart. We're
>> in the process of removing any references to Astronomer specific things and
>> making them more general features. That work is happening on this PR here
>> https://github.com/astronomer/airflow-chart/pull/66. Hoping to wrap up
>> the last few outstanding items and open up a PR to Airflow next week
>> sometime!
>>
>> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <
>> daniel.imberman@gmail.com> wrote:
>>
>>> Hi Jarek,
>>>
>>> Congrats on getting the image out! This is going to make a lot of things
>>> easier :). We’ve actually been working on separating out KinD as well (and
>>> in fact moving it out of travis and into GitHub issues). Let me circle back
>>> with Steven (the engineer who’s been working on that) and get a status
>>> report from him.
>>>
>>> I will also take some time today to get our helm chart to work with
>>> the airflow master image. It’s been a bit crazy around here in the last few
>>> weeks but I am consistently pushing to make sure we make the final changes
>>> of ripping out any remaining platform components before we donate :).
>>>
>>> I agree that docker hub automation will be great. This will also allow
>>> us to have a nightly build for any integration/UI tests we build.
>>>
>>> Thanks again Jarek!
>>>
>>> Daniel
>>>
>>> via Newton Mail
>>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>>>
>>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>> Hello Greg and Daniel,
>>>
>>> Production image is merged and it's also going to be released in
>>> 1.10.10. For now, I will manually tag the release in DockerHub when we
>>> release but we will automate that in the future. Currently, we have
>>> *airflow:master* and *airflow:v1-10-test* images (for all supported
>>> python versions) that you can use for testing with the Helm chart!
>>>
>>> The next things from my side:
>>>
>>> * I will automate DockerHub building for the prod image from master -
>>> for now only CI is automated but we need to add prod image as well
>>> * I will move kind-cluster creation outside of the container and use
>>> production image for our CI Kubernetes tests. I have a WIP branch on it and
>>> I think I should have it early next week
>>> * The next step will be to switch our CI to Github Actions finally. I
>>> think we will run both Travis and GA for a short time to make sure it works
>>> * I will ask people for options they would like to have in the image -
>>> additional packages etc. so that I can work on the next iteration of the
>>> image. Also if you have any needs to see for the Helm integration, I am
>>> super happy to incorporate them in the next version. I plan to keep the
>>> "master" and "v1-10-test" images updated with any new changes and push
>>> images regularly
>>> * I am happy to help with the Helm Chart, so let me know if you need
>>> anything!
>>>
>>> J.
>>>
>>>
>>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> Hello Daniel and Greg.
>>>>
>>>> I think the production image for Airflow is quite ready for the final
>>>> review and merge. I did some tests and I think it's quite complete now.
>>>>
>>>> The PR is running it's Travis build:
>>>> https://github.com/apache/airflow/pull/7832
>>>>
>>>> I already pushed the image to DockerHub - so that the next time it is
>>>> merged It will use it as a base and will be rather quickly built using
>>>> those images as a cache. Once it is merged, I will add the automated build
>>>> in DockerHub and merge it to 1.10. As the next step I also plan to use the
>>>> prod image for our Kubernetes tests (that will help with moving to the
>>>> GitHub Actions from Travis).
>>>>
>>>> I added quite a comprehensive documentation - explaining all ins- and
>>>> outs- of both CI and PROD images
>>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
>>>> .
>>>>
>>>> You can build it yourself using manual build process described in the
>>>> doc or (easier) using Breeze. That includes using the same Dockerfile to
>>>> build older Airflow version from the 1.10.* line.
>>>>
>>>> Feel free to use the new image in the Helm Chart you have - happy to
>>>> review the PRs. The image is available at 'apache/airflow:master-python3.6'
>>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
>>>> release 1.10.10 we will also have it automatically published for 1.10.10
>>>> and beyond.
>>>>
>>>> Happy to hear your review comments and hope to merge it very soon.
>>>> J.
>>>>
>>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <Ja...@polidea.com>
>>>> wrote:
>>>>
>>>>> Absolutely! Please do :). The more eyes on this the better!
>>>>>
>>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
>>>>> daniel.imberman@gmail.com> wrote:
>>>>>
>>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
>>>>>> actions today. If we can get that working, we can donate the helm chart and
>>>>>> possibly even deprecate the old Kubernetes tests in the process.
>>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <
>>>>>> Jarek.Potiuk@polidea.com>, wrote:
>>>>>> > We have A POC for Github Actions done by Tomek, and the blocker
>>>>>> there
>>>>>> > was precisely Kubernetes tests and specifically running kind cluster
>>>>>> > in the Github Actions
>>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
>>>>>> meantime
>>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this very
>>>>>> > thing today at slack:
>>>>>> >
>>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>>>>>> >
>>>>>> > For me this is the next thing I will look at after merging
>>>>>> > requirements PR and Prod image PR (and those two are a bit
>>>>>> > pre-requisites to make it stable and fast). I want to switch to the
>>>>>> > Production Image to run Kubernetes tests (they will be much faster)
>>>>>> > and then we can work out much faster how to run them on Github
>>>>>> > Actions. Then we can add Helm charts to run all those different
>>>>>> > deployments. I am really looking forward to it!
>>>>>> >
>>>>>> > I hope - this weekend I move forward all of that and have a PR that
>>>>>> we
>>>>>> > can start running tests on and replacing the resources with helm
>>>>>> > chart.
>>>>>> >
>>>>>> > J.
>>>>>> >
>>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>>>>>> > <da...@gmail.com> wrote:
>>>>>> > >
>>>>>> > > @Steven glad to help you out on that.
>>>>>> > >
>>>>>> > > For the moment we only use travis, but I believe @jarek has been
>>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first
>>>>>> GitHub tests?
>>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>>>>>> <st...@astronomer.io.invalid>, wrote:
>>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to test
>>>>>> it. Where
>>>>>> > > > do we want to run it? Do you all just use Travis only?
>>>>>> > > >
>>>>>> > > > Steven
>>>>>> > > >
>>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>>>>>> daniel.imberman@gmail.com>
>>>>>> > > > wrote:
>>>>>> > > >
>>>>>> > > > > @Jarek I think with the helm chart + prod image we can go
>>>>>> even further
>>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
>>>>>> autoscaling, and a
>>>>>> > > > > bunch of other configurations.
>>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>>>>>> Jarek.Potiuk@polidea.com>,
>>>>>> > > > > wrote:
>>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original
>>>>>> email :)
>>>>>> > > > > >
>>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>>>>>> > > > > daniel.imberman@gmail.com> wrote:
>>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We
>>>>>> just have
>>>>>> > > > > deployment files. If anything having the helm chart as a part
>>>>>> of the
>>>>>> > > > > airflow repo could mean that the helm chart becomes the
>>>>>> defacto system for
>>>>>> > > > > testing airflow on kubernetes (we can get rid of all the yams
>>>>>> files and run
>>>>>> > > > > multiple k8s tests with different settings).
>>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
>>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo.
>>>>>> We've also
>>>>>> > > > > been
>>>>>> > > > > > > > > building up a test suite that currently runs on
>>>>>> CircleCI and uses
>>>>>> > > > > kind
>>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>>>>>> versions with
>>>>>> > > > > some
>>>>>> > > > > > > > > different settings. Right now we're mostly testing
>>>>>> the different
>>>>>> > > > > executors
>>>>>> > > > > > > > > since that has the biggest impact on what gets
>>>>>> deployed, but that
>>>>>> > > > > can be
>>>>>> > > > > > > > > expanded.
>>>>>> > > > > > > > >
>>>>>> > > > > > > > > What CRDs are currently being used to run Airflow for
>>>>>> the tests?
>>>>>> > > > > > > > >
>>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>>>>>> > > > > Jarek.Potiuk@polidea.com>
>>>>>> > > > > > > > > wrote:
>>>>>> > > > > > > > >
>>>>>> > > > > > > > > > One thing for the donation.
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > Did you you want to have separate repository Greg ?
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > I think we should simply create a folder in Airflow
>>>>>> repo and
>>>>>> > > > > keep it
>>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going
>>>>>> to switch our
>>>>>> > > > > > > > > > Kubernetes Tests to the production image (will make
>>>>>> the tests
>>>>>> > > > > much
>>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
>>>>>> automatically in
>>>>>> > > > > CI -
>>>>>> > > > > > > > > > for now we are using some custom Resource
>>>>>> definitions to start
>>>>>> > > > > Airflow
>>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>>>>>> switch to
>>>>>> > > > > using the
>>>>>> > > > > > > > > > helm chart - this way we can test all three things
>>>>>> at once:
>>>>>> > > > > > > > > > - Kubernetes Executor
>>>>>> > > > > > > > > > - Dockerfile
>>>>>> > > > > > > > > > - Helm Chart
>>>>>> > > > > > > > > > and we could also add more tests - for example
>>>>>> testing different
>>>>>> > > > > > > > > > deployment options for the helm chart.
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help
>>>>>> with that -
>>>>>> > > > > > > > > > especially in terms of future changes and testing
>>>>>> them
>>>>>> > > > > automatically.
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > J.
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat
>>>>>> kyzy
>>>>>> > > > > > > > > > <ai...@apache.org> wrote:
>>>>>> > > > > > > > > > >
>>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more
>>>>>> useful stuff for
>>>>>> > > > > the
>>>>>> > > > > > > > > > > community :)
>>>>>> > > > > > > > > > >
>>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>>>>>> > > > > greg@astronomer.io>
>>>>>> > > > > > > > > > wrote:
>>>>>> > > > > > > > > > >
>>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as
>>>>>> an optional
>>>>>> > > > > Kubernetes
>>>>>> > > > > > > > > > > > CronJob (
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > >
>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>>>>>> > > > > > > > > > )
>>>>>> > > > > > > > > > > > that we have run periodically to clean failed
>>>>>> pods up and
>>>>>> > > > > could stay
>>>>>> > > > > > > > > > > > separate.
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > > > > The wait_for_migrations script could definitely
>>>>>> be pulled
>>>>>> > > > > into Airflow.
>>>>>> > > > > > > > > > > > For context, we deploy an initContainer on the
>>>>>> scheduler (
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > >
>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>>>>>> > > > > > > > > > )
>>>>>> > > > > > > > > > > > that runs the upgradedb command before booting
>>>>>> the
>>>>>> > > > > scheduler. This new
>>>>>> > > > > > > > > > > > wait_for_migration script runs in an
>>>>>> initContainer on the
>>>>>> > > > > webserver and
>>>>>> > > > > > > > > > > > workers (
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > >
>>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>>>>>> > > > > > > > > > )
>>>>>> > > > > > > > > > > > so that they don't boot up ahead of a
>>>>>> potentially
>>>>>> > > > > long-running
>>>>>> > > > > > > > > > migration
>>>>>> > > > > > > > > > > > and attempt to operate on new or missing
>>>>>> columns/tables
>>>>>> > > > > before the
>>>>>> > > > > > > > > > > > migrations run. This prevents these pods from
>>>>>> entering a
>>>>>> > > > > CrashLoop.
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>>>>>> > > > > > > > > > > > wrote:
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
>>>>>> currently
>>>>>> > > > > generated from
>>>>>> > > > > > > > > > > > > Dockerfiles
>>>>>> > > > > > > > > > > > > > in this repo
>>>>>> https://github.com/astronomer/ap-airflow
>>>>>> > > > > and get
>>>>>> > > > > > > > > > > > > published to
>>>>>> > > > > > > > > > > > > > DockerHub
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > >
>>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > For the most part those are typical Airflow
>>>>>> images.
>>>>>> > > > > There's an
>>>>>> > > > > > > > > > > > > entrypoint
>>>>>> > > > > > > > > > > > > > script that we include in the image that
>>>>>> handles waiting
>>>>>> > > > > for the
>>>>>> > > > > > > > > > > > > database
>>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
>>>>>> pretty generic.
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > I already added waiting for the database
>>>>>> (both metadata
>>>>>> > > > > and celery
>>>>>> > > > > > > > > > URL) in
>>>>>> > > > > > > > > > > > > the PR:
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > >
>>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>>>>> > > > > > > > > > > > > .
>>>>>> > > > > > > > > > > > > It's functionally the same but more generic.
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > The only other
>>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses
>>>>>> would be the
>>>>>> > > > > scripts in this
>>>>>> > > > > > > > > > repo
>>>>>> > > > > > > > > > > > > >
>>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
>>>>>> > > > > Our
>>>>>> > > > > > > > > > > > > Dockerfiles
>>>>>> > > > > > > > > > > > > > pull this package in. These scripts are
>>>>>> used to
>>>>>> > > > > coordinate running
>>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > I see two scripts:
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
>>>>>> needed to run in
>>>>>> > > > > airflow -
>>>>>> > > > > > > > > > this
>>>>>> > > > > > > > > > > > > could be run as a separate pod/container?
>>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is a
>>>>>> good
>>>>>> > > > > candidate to add
>>>>>> > > > > > > > > > > > > *airflow
>>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it
>>>>>> part of airflow
>>>>>> > > > > itself.
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > I think we also have to agree on the Airflow
>>>>>> version
>>>>>> > > > > supported by the
>>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>>>>>> 1.10.10+ and we
>>>>>> > > > > > > > > > incorporate
>>>>>> > > > > > > > > > > > > all
>>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>>>>>> > > > > wait_for_migration")
>>>>>> > > > > > > > > > into 2.0
>>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and helm
>>>>>> chart for
>>>>>> > > > > those versions
>>>>>> > > > > > > > > > > > > only. That would help with people migrating
>>>>>> to the latest
>>>>>> > > > > version.
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > WDYT?
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>>>>>> Imberman <
>>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>>>>>> > > > > > > > > > > > > > wrote:
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>>>>>> pairing an
>>>>>> > > > > official helm
>>>>>> > > > > > > > > > chart
>>>>>> > > > > > > > > > > > > > > with the official image would make for a
>>>>>> REALLY
>>>>>> > > > > powerful “up and
>>>>>> > > > > > > > > > > > > running
>>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have
>>>>>> also been
>>>>>> > > > > looking into
>>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
>>>>>> create custom
>>>>>> > > > > controllers
>>>>>> > > > > > > > > > from
>>>>>> > > > > > > > > > > > > helm
>>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
>>>>>> punch from the
>>>>>> > > > > same code
>>>>>> > > > > > > > > > base
>>>>>> > > > > > > > > > > > > :).
>>>>>> > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s
>>>>>> no issues, can
>>>>>> > > > > we please
>>>>>> > > > > > > > > > start
>>>>>> > > > > > > > > > > > > > the
>>>>>> > > > > > > > > > > > > > > process of donation?
>>>>>> > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
>>>>>> > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > Daniel
>>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
>>>>>> Potiuk <
>>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>>>>>> > > > > > > > > > > > > > > wrote:
>>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>>>>>> official image
>>>>>> > > > > we have
>>>>>> > > > > > > > > > work in
>>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>>>>>> Astronomer's
>>>>>> > > > > image while
>>>>>> > > > > > > > > > > > > preparing
>>>>>> > > > > > > > > > > > > > my
>>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>>>>>> needed to make
>>>>>> > > > > it works
>>>>>> > > > > > > > > > with
>>>>>> > > > > > > > > > > > > the
>>>>>> > > > > > > > > > > > > > > helm
>>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
>>>>>> collaborate on that.
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > PR here:
>>>>>> https://github.com/apache/airflow/pull/7832
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > J.
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil
>>>>>> Naik <
>>>>>> > > > > kaxilnaik@gmail.com
>>>>>> > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > wrote:
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>>>>>> tomasz.urbaszek@polidea.com> :
>>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
>>>>>> > > > > https://github.com/astronomer/airflow-chart
>>>>>> > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM
>>>>>> Tomasz Urbaszek <
>>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
>>>>>> > > > > > > > > > > > > > > > > wrote:
>>>>>> > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > An official helm chart is something
>>>>>> our
>>>>>> > > > > community needs!
>>>>>> > > > > > > > > > Using
>>>>>> > > > > > > > > > > > > your
>>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot
>>>>>> of sens to me
>>>>>> > > > > because as
>>>>>> > > > > > > > > > you
>>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>>>>>> > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image do
>>>>>> you use?
>>>>>> > > > > Also, would you
>>>>>> > > > > > > > > > > > > mind
>>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>>>>>> > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > Tomek
>>>>>> > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM
>>>>>> Greg Neiheisel
>>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
>>>>>> Astronomer, we’ve
>>>>>> > > > > created,
>>>>>> > > > > > > > > > managed,
>>>>>> > > > > > > > > > > > > > and
>>>>>> > > > > > > > > > > > > > > > > > hardened
>>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for
>>>>>> Airflow (
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> https://github.com/astronomer/airflow-chart)
>>>>>> > > > > that is
>>>>>> > > > > > > > > > being
>>>>>> > > > > > > > > > > > > used
>>>>>> > > > > > > > > > > > > > by
>>>>>> > > > > > > > > > > > > > > > > both
>>>>>> > > > > > > > > > > > > > > > > > our
>>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers.
>>>>>> This chart is
>>>>>> > > > > > > > > > battle-tested and
>>>>>> > > > > > > > > > > > > > > running
>>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments
>>>>>> of varying
>>>>>> > > > > sizes and
>>>>>> > > > > > > > > > runtime
>>>>>> > > > > > > > > > > > > > > > > > environments.
>>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate
>>>>>> the issues
>>>>>> > > > > that Airflow
>>>>>> > > > > > > > > > > > > users
>>>>>> > > > > > > > > > > > > > run
>>>>>> > > > > > > > > > > > > > > > > into
>>>>>> > > > > > > > > > > > > > > > > > in
>>>>>> > > > > > > > > > > > > > > > > > > the real world.
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally
>>>>>> developed
>>>>>> > > > > internally for
>>>>>> > > > > > > > > > our
>>>>>> > > > > > > > > > > > > > > Astronomer
>>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently
>>>>>> decoupled the chart
>>>>>> > > > > from the
>>>>>> > > > > > > > > > rest of
>>>>>> > > > > > > > > > > > > our
>>>>>> > > > > > > > > > > > > > > > > > platform
>>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater
>>>>>> Airflow
>>>>>> > > > > community. With
>>>>>> > > > > > > > > > these
>>>>>> > > > > > > > > > > > > > > changes
>>>>>> > > > > > > > > > > > > > > > > in
>>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
>>>>>> conversation about
>>>>>> > > > > donating this
>>>>>> > > > > > > > > > > > > chart
>>>>>> > > > > > > > > > > > > > to
>>>>>> > > > > > > > > > > > > > > the
>>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of the
>>>>>> chart are:
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With
>>>>>> zero
>>>>>> > > > > configuration, a user
>>>>>> > > > > > > > > > > > > will
>>>>>> > > > > > > > > > > > > > get
>>>>>> > > > > > > > > > > > > > > > > a
>>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default user
>>>>>> and the
>>>>>> > > > > > > > > > KubernetesExecutor
>>>>>> > > > > > > > > > > > > > ready
>>>>>> > > > > > > > > > > > > > > > > to
>>>>>> > > > > > > > > > > > > > > > > > run
>>>>>> > > > > > > > > > > > > > > > > > > DAGs.
>>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>>>>>> optional KEDA
>>>>>> > > > > > > > > > autoscaling) and
>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer.
>>>>>> We use this to
>>>>>> > > > > share a
>>>>>> > > > > > > > > > > > > > configurable
>>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
>>>>>> deployment. Useful
>>>>>> > > > > for limiting
>>>>>> > > > > > > > > > > > > > > connections to
>>>>>> > > > > > > > > > > > > > > > > > the
>>>>>> > > > > > > > > > > > > > > > > > > metadata database.
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A
>>>>>> user can push a
>>>>>> > > > > newer
>>>>>> > > > > > > > > > version
>>>>>> > > > > > > > > > > > > of
>>>>>> > > > > > > > > > > > > > > > > > Airflow
>>>>>> > > > > > > > > > > > > > > > > > > into an existing release and
>>>>>> migrations will
>>>>>> > > > > > > > > > automatically run
>>>>>> > > > > > > > > > > > > > > > > > cleanly.
>>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
>>>>>> install and
>>>>>> > > > > configure a
>>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
>>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
>>>>>> expose them to
>>>>>> > > > > Prometheus
>>>>>> > > > > > > > > > > > > > > > > automatically.
>>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>>>>>> control the
>>>>>> > > > > ResourceQuotas
>>>>>> > > > > > > > > > and
>>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment
>>>>>> so that no
>>>>>> > > > > deployment can
>>>>>> > > > > > > > > > > > > > overload
>>>>>> > > > > > > > > > > > > > > a
>>>>>> > > > > > > > > > > > > > > > > > > cluster.
>>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>>>>>> support.
>>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>>>>>> Sometimes
>>>>>> > > > > > > > > > KubernetesExecutor and
>>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail
>>>>>> for reasons
>>>>>> > > > > other than the
>>>>>> > > > > > > > > > > > > actual
>>>>>> > > > > > > > > > > > > > > > > > task.
>>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things
>>>>>> clean in
>>>>>> > > > > Kubernetes.
>>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally in
>>>>>> KIND
>>>>>> > > > > (Kubernetes in
>>>>>> > > > > > > > > > Docker).
>>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across
>>>>>> many Kubernetes
>>>>>> > > > > versions
>>>>>> > > > > > > > > > with
>>>>>> > > > > > > > > > > > > Helm
>>>>>> > > > > > > > > > > > > > 2
>>>>>> > > > > > > > > > > > > > > > > > and 3
>>>>>> > > > > > > > > > > > > > > > > > > support.
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and
>>>>>> most
>>>>>> > > > > reliable way to
>>>>>> > > > > > > > > > deploy
>>>>>> > > > > > > > > > > > > > DAGs
>>>>>> > > > > > > > > > > > > > > to
>>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at
>>>>>> scale is to
>>>>>> > > > > package them
>>>>>> > > > > > > > > > into
>>>>>> > > > > > > > > > > > > the
>>>>>> > > > > > > > > > > > > > > actual
>>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared
>>>>>> this chart
>>>>>> > > > > towards that
>>>>>> > > > > > > > > > > > > method of
>>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other
>>>>>> methods should
>>>>>> > > > > be
>>>>>> > > > > > > > > > > > > straightforward.
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>>>>>> community and
>>>>>> > > > > would love
>>>>>> > > > > > > > > > to
>>>>>> > > > > > > > > > > > > see
>>>>>> > > > > > > > > > > > > > > this
>>>>>> > > > > > > > > > > > > > > > > > chart
>>>>>> > > > > > > > > > > > > > > > > > > help others to get up and running
>>>>>> on
>>>>>> > > > > Kubernetes!
>>>>>> > > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > > > > --
>>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>>> > > > > Astronomer.io
>>>>>> > > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > --
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
>>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>>> Principal
>>>>>> > > > > Software
>>>>>> > > > > > > > > > Engineer
>>>>>> > > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
>>>>>> <+48%20660%20796%20129>
>>>>>> > > > > <+48660796129
>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
>>>>>> https://www.polidea.com/>
>>>>>> > > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > > --
>>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>>> Astronomer.io
>>>>>> > > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > --
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > Jarek Potiuk
>>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>>> Principal Software
>>>>>> > > > > Engineer
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>>>>> <+48660796129
>>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>>> > > > > > > > > > > > >
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > > > > --
>>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>>> > > > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > --
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > Jarek Potiuk
>>>>>> > > > > > > > > > Polidea | Principal Software Engineer
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > > > M: +48 660 796 129
>>>>>> > > > > > > > > >
>>>>>> > > > > > > > >
>>>>>> > > > > > > > >
>>>>>> > > > > > > > > --
>>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>>> > > > > >
>>>>>> > > > > >
>>>>>> > > > > > --
>>>>>> > > > > > Jarek Potiuk
>>>>>> > > > > > Polidea | Principal Software Engineer
>>>>>> > > > > > M: +48 660 796 129
>>>>>> > > > > >
>>>>>> > > > >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> >
>>>>>> > Jarek Potiuk
>>>>>> > Polidea | Principal Software Engineer
>>>>>> >
>>>>>> > M: +48 660 796 129
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Jarek Potiuk
>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>>
>>>>> M: +48 660 796 129 <+48660796129>
>>>>> [image: Polidea] <https://www.polidea.com/>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>> *Greg Neiheisel* / Chief Architect Astronomer.io
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Awesome, thanks for the update!
On Fri, Apr 10, 2020 at 6:57 PM Greg Neiheisel <gr...@astronomer.io> wrote:
> Hey everyone, just wanted to post a quick update on the helm chart. We're
> in the process of removing any references to Astronomer specific things and
> making them more general features. That work is happening on this PR here
> https://github.com/astronomer/airflow-chart/pull/66. Hoping to wrap up
> the last few outstanding items and open up a PR to Airflow next week
> sometime!
>
> On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> Hi Jarek,
>>
>> Congrats on getting the image out! This is going to make a lot of things
>> easier :). We’ve actually been working on separating out KinD as well (and
>> in fact moving it out of travis and into GitHub issues). Let me circle back
>> with Steven (the engineer who’s been working on that) and get a status
>> report from him.
>>
>> I will also take some time today to get our helm chart to work with
>> the airflow master image. It’s been a bit crazy around here in the last few
>> weeks but I am consistently pushing to make sure we make the final changes
>> of ripping out any remaining platform components before we donate :).
>>
>> I agree that docker hub automation will be great. This will also allow us
>> to have a nightly build for any integration/UI tests we build.
>>
>> Thanks again Jarek!
>>
>> Daniel
>>
>> via Newton Mail
>> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>>
>> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>> Hello Greg and Daniel,
>>
>> Production image is merged and it's also going to be released in 1.10.10.
>> For now, I will manually tag the release in DockerHub when we release but
>> we will automate that in the future. Currently, we have *airflow:master*
>> and *airflow:v1-10-test* images (for all supported python versions) that
>> you can use for testing with the Helm chart!
>>
>> The next things from my side:
>>
>> * I will automate DockerHub building for the prod image from master - for
>> now only CI is automated but we need to add prod image as well
>> * I will move kind-cluster creation outside of the container and use
>> production image for our CI Kubernetes tests. I have a WIP branch on it and
>> I think I should have it early next week
>> * The next step will be to switch our CI to Github Actions finally. I
>> think we will run both Travis and GA for a short time to make sure it works
>> * I will ask people for options they would like to have in the image -
>> additional packages etc. so that I can work on the next iteration of the
>> image. Also if you have any needs to see for the Helm integration, I am
>> super happy to incorporate them in the next version. I plan to keep the
>> "master" and "v1-10-test" images updated with any new changes and push
>> images regularly
>> * I am happy to help with the Helm Chart, so let me know if you need
>> anything!
>>
>> J.
>>
>>
>> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Hello Daniel and Greg.
>>>
>>> I think the production image for Airflow is quite ready for the final
>>> review and merge. I did some tests and I think it's quite complete now.
>>>
>>> The PR is running it's Travis build:
>>> https://github.com/apache/airflow/pull/7832
>>>
>>> I already pushed the image to DockerHub - so that the next time it is
>>> merged It will use it as a base and will be rather quickly built using
>>> those images as a cache. Once it is merged, I will add the automated build
>>> in DockerHub and merge it to 1.10. As the next step I also plan to use the
>>> prod image for our Kubernetes tests (that will help with moving to the
>>> GitHub Actions from Travis).
>>>
>>> I added quite a comprehensive documentation - explaining all ins- and
>>> outs- of both CI and PROD images
>>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
>>> .
>>>
>>> You can build it yourself using manual build process described in the
>>> doc or (easier) using Breeze. That includes using the same Dockerfile to
>>> build older Airflow version from the 1.10.* line.
>>>
>>> Feel free to use the new image in the Helm Chart you have - happy to
>>> review the PRs. The image is available at 'apache/airflow:master-python3.6'
>>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
>>> release 1.10.10 we will also have it automatically published for 1.10.10
>>> and beyond.
>>>
>>> Happy to hear your review comments and hope to merge it very soon.
>>> J.
>>>
>>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <Ja...@polidea.com>
>>> wrote:
>>>
>>>> Absolutely! Please do :). The more eyes on this the better!
>>>>
>>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
>>>> daniel.imberman@gmail.com> wrote:
>>>>
>>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
>>>>> actions today. If we can get that working, we can donate the helm chart and
>>>>> possibly even deprecate the old Kubernetes tests in the process.
>>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
>>>>> wrote:
>>>>> > We have A POC for Github Actions done by Tomek, and the blocker there
>>>>> > was precisely Kubernetes tests and specifically running kind cluster
>>>>> > in the Github Actions
>>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the
>>>>> meantime
>>>>> > Jiaje Zhong is trying to make it works on GA we discussed this very
>>>>> > thing today at slack:
>>>>> >
>>>>> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>>>>> >
>>>>> > For me this is the next thing I will look at after merging
>>>>> > requirements PR and Prod image PR (and those two are a bit
>>>>> > pre-requisites to make it stable and fast). I want to switch to the
>>>>> > Production Image to run Kubernetes tests (they will be much faster)
>>>>> > and then we can work out much faster how to run them on Github
>>>>> > Actions. Then we can add Helm charts to run all those different
>>>>> > deployments. I am really looking forward to it!
>>>>> >
>>>>> > I hope - this weekend I move forward all of that and have a PR that
>>>>> we
>>>>> > can start running tests on and replacing the resources with helm
>>>>> > chart.
>>>>> >
>>>>> > J.
>>>>> >
>>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>>>>> > <da...@gmail.com> wrote:
>>>>> > >
>>>>> > > @Steven glad to help you out on that.
>>>>> > >
>>>>> > > For the moment we only use travis, but I believe @jarek has been
>>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first
>>>>> GitHub tests?
>>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>>>>> <st...@astronomer.io.invalid>, wrote:
>>>>> > > > Hey I’m happy to PR in the chart and get the CI working to test
>>>>> it. Where
>>>>> > > > do we want to run it? Do you all just use Travis only?
>>>>> > > >
>>>>> > > > Steven
>>>>> > > >
>>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>>>>> daniel.imberman@gmail.com>
>>>>> > > > wrote:
>>>>> > > >
>>>>> > > > > @Jarek I think with the helm chart + prod image we can go even
>>>>> further
>>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
>>>>> autoscaling, and a
>>>>> > > > > bunch of other configurations.
>>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>>>>> Jarek.Potiuk@polidea.com>,
>>>>> > > > > wrote:
>>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original email
>>>>> :)
>>>>> > > > > >
>>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>>>>> > > > > daniel.imberman@gmail.com> wrote:
>>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We
>>>>> just have
>>>>> > > > > deployment files. If anything having the helm chart as a part
>>>>> of the
>>>>> > > > > airflow repo could mean that the helm chart becomes the
>>>>> defacto system for
>>>>> > > > > testing airflow on kubernetes (we can get rid of all the yams
>>>>> files and run
>>>>> > > > > multiple k8s tests with different settings).
>>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
>>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo.
>>>>> We've also
>>>>> > > > > been
>>>>> > > > > > > > > building up a test suite that currently runs on
>>>>> CircleCI and uses
>>>>> > > > > kind
>>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>>>>> versions with
>>>>> > > > > some
>>>>> > > > > > > > > different settings. Right now we're mostly testing the
>>>>> different
>>>>> > > > > executors
>>>>> > > > > > > > > since that has the biggest impact on what gets
>>>>> deployed, but that
>>>>> > > > > can be
>>>>> > > > > > > > > expanded.
>>>>> > > > > > > > >
>>>>> > > > > > > > > What CRDs are currently being used to run Airflow for
>>>>> the tests?
>>>>> > > > > > > > >
>>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>>>>> > > > > Jarek.Potiuk@polidea.com>
>>>>> > > > > > > > > wrote:
>>>>> > > > > > > > >
>>>>> > > > > > > > > > One thing for the donation.
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > Did you you want to have separate repository Greg ?
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > I think we should simply create a folder in Airflow
>>>>> repo and
>>>>> > > > > keep it
>>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going
>>>>> to switch our
>>>>> > > > > > > > > > Kubernetes Tests to the production image (will make
>>>>> the tests
>>>>> > > > > much
>>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
>>>>> automatically in
>>>>> > > > > CI -
>>>>> > > > > > > > > > for now we are using some custom Resource
>>>>> definitions to start
>>>>> > > > > Airflow
>>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>>>>> switch to
>>>>> > > > > using the
>>>>> > > > > > > > > > helm chart - this way we can test all three things
>>>>> at once:
>>>>> > > > > > > > > > - Kubernetes Executor
>>>>> > > > > > > > > > - Dockerfile
>>>>> > > > > > > > > > - Helm Chart
>>>>> > > > > > > > > > and we could also add more tests - for example
>>>>> testing different
>>>>> > > > > > > > > > deployment options for the helm chart.
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help
>>>>> with that -
>>>>> > > > > > > > > > especially in terms of future changes and testing
>>>>> them
>>>>> > > > > automatically.
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > J.
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat
>>>>> kyzy
>>>>> > > > > > > > > > <ai...@apache.org> wrote:
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > +1 on the donation. Always happy to see more
>>>>> useful stuff for
>>>>> > > > > the
>>>>> > > > > > > > > > > community :)
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>>>>> > > > > greg@astronomer.io>
>>>>> > > > > > > > > > wrote:
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as an
>>>>> optional
>>>>> > > > > Kubernetes
>>>>> > > > > > > > > > > > CronJob (
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > >
>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>>>>> > > > > > > > > > )
>>>>> > > > > > > > > > > > that we have run periodically to clean failed
>>>>> pods up and
>>>>> > > > > could stay
>>>>> > > > > > > > > > > > separate.
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > > The wait_for_migrations script could definitely
>>>>> be pulled
>>>>> > > > > into Airflow.
>>>>> > > > > > > > > > > > For context, we deploy an initContainer on the
>>>>> scheduler (
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > >
>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>>>>> > > > > > > > > > )
>>>>> > > > > > > > > > > > that runs the upgradedb command before booting
>>>>> the
>>>>> > > > > scheduler. This new
>>>>> > > > > > > > > > > > wait_for_migration script runs in an
>>>>> initContainer on the
>>>>> > > > > webserver and
>>>>> > > > > > > > > > > > workers (
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > >
>>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>>>>> > > > > > > > > > )
>>>>> > > > > > > > > > > > so that they don't boot up ahead of a potentially
>>>>> > > > > long-running
>>>>> > > > > > > > > > migration
>>>>> > > > > > > > > > > > and attempt to operate on new or missing
>>>>> columns/tables
>>>>> > > > > before the
>>>>> > > > > > > > > > > > migrations run. This prevents these pods from
>>>>> entering a
>>>>> > > > > CrashLoop.
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>>>>> > > > > > > > > > > > wrote:
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
>>>>> currently
>>>>> > > > > generated from
>>>>> > > > > > > > > > > > > Dockerfiles
>>>>> > > > > > > > > > > > > > in this repo
>>>>> https://github.com/astronomer/ap-airflow
>>>>> > > > > and get
>>>>> > > > > > > > > > > > > published to
>>>>> > > > > > > > > > > > > > DockerHub
>>>>> > > > > > > > > > > > > >
>>>>> > > > >
>>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > For the most part those are typical Airflow
>>>>> images.
>>>>> > > > > There's an
>>>>> > > > > > > > > > > > > entrypoint
>>>>> > > > > > > > > > > > > > script that we include in the image that
>>>>> handles waiting
>>>>> > > > > for the
>>>>> > > > > > > > > > > > > database
>>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
>>>>> pretty generic.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > I already added waiting for the database (both
>>>>> metadata
>>>>> > > > > and celery
>>>>> > > > > > > > > > URL) in
>>>>> > > > > > > > > > > > > the PR:
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > >
>>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>>>> > > > > > > > > > > > > .
>>>>> > > > > > > > > > > > > It's functionally the same but more generic.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > The only other
>>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses would
>>>>> be the
>>>>> > > > > scripts in this
>>>>> > > > > > > > > > repo
>>>>> > > > > > > > > > > > > >
>>>>> https://github.com/astronomer/astronomer-airflow-scripts.
>>>>> > > > > Our
>>>>> > > > > > > > > > > > > Dockerfiles
>>>>> > > > > > > > > > > > > > pull this package in. These scripts are used
>>>>> to
>>>>> > > > > coordinate running
>>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > I see two scripts:
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
>>>>> needed to run in
>>>>> > > > > airflow -
>>>>> > > > > > > > > > this
>>>>> > > > > > > > > > > > > could be run as a separate pod/container?
>>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is a
>>>>> good
>>>>> > > > > candidate to add
>>>>> > > > > > > > > > > > > *airflow
>>>>> > > > > > > > > > > > > db wait_for_migration* command and make it
>>>>> part of airflow
>>>>> > > > > itself.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > I think we also have to agree on the Airflow
>>>>> version
>>>>> > > > > supported by the
>>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>>>>> 1.10.10+ and we
>>>>> > > > > > > > > > incorporate
>>>>> > > > > > > > > > > > > all
>>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>>>>> > > > > wait_for_migration")
>>>>> > > > > > > > > > into 2.0
>>>>> > > > > > > > > > > > > and 1.10 and we support both - image and helm
>>>>> chart for
>>>>> > > > > those versions
>>>>> > > > > > > > > > > > > only. That would help with people migrating to
>>>>> the latest
>>>>> > > > > version.
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > WDYT?
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>>>>> Imberman <
>>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>>>>> > > > > > > > > > > > > > wrote:
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>>>>> pairing an
>>>>> > > > > official helm
>>>>> > > > > > > > > > chart
>>>>> > > > > > > > > > > > > > > with the official image would make for a
>>>>> REALLY
>>>>> > > > > powerful “up and
>>>>> > > > > > > > > > > > > running
>>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have
>>>>> also been
>>>>> > > > > looking into
>>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
>>>>> create custom
>>>>> > > > > controllers
>>>>> > > > > > > > > > from
>>>>> > > > > > > > > > > > > helm
>>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
>>>>> punch from the
>>>>> > > > > same code
>>>>> > > > > > > > > > base
>>>>> > > > > > > > > > > > > :).
>>>>> > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no
>>>>> issues, can
>>>>> > > > > we please
>>>>> > > > > > > > > > start
>>>>> > > > > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > > process of donation?
>>>>> > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
>>>>> > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > Daniel
>>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
>>>>> Potiuk <
>>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>>>>> > > > > > > > > > > > > > > wrote:
>>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>>>>> official image
>>>>> > > > > we have
>>>>> > > > > > > > > > work in
>>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>>>>> Astronomer's
>>>>> > > > > image while
>>>>> > > > > > > > > > > > > preparing
>>>>> > > > > > > > > > > > > > my
>>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>>>>> needed to make
>>>>> > > > > it works
>>>>> > > > > > > > > > with
>>>>> > > > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > > helm
>>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
>>>>> collaborate on that.
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > PR here:
>>>>> https://github.com/apache/airflow/pull/7832
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > J.
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil
>>>>> Naik <
>>>>> > > > > kaxilnaik@gmail.com
>>>>> > > > > > > > > > >
>>>>> > > > > > > > > > > > > > wrote:
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>>>>> tomasz.urbaszek@polidea.com> :
>>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
>>>>> > > > > https://github.com/astronomer/airflow-chart
>>>>> > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz
>>>>> Urbaszek <
>>>>> > > > > > > > > > > > > > turbaszek@apache.org>
>>>>> > > > > > > > > > > > > > > > > wrote:
>>>>> > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > An official helm chart is something
>>>>> our
>>>>> > > > > community needs!
>>>>> > > > > > > > > > Using
>>>>> > > > > > > > > > > > > your
>>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot of
>>>>> sens to me
>>>>> > > > > because as
>>>>> > > > > > > > > > you
>>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>>>>> > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image do
>>>>> you use?
>>>>> > > > > Also, would you
>>>>> > > > > > > > > > > > > mind
>>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>>>>> > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > Tomek
>>>>> > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg
>>>>> Neiheisel
>>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
>>>>> Astronomer, we’ve
>>>>> > > > > created,
>>>>> > > > > > > > > > managed,
>>>>> > > > > > > > > > > > > > and
>>>>> > > > > > > > > > > > > > > > > > hardened
>>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for
>>>>> Airflow (
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> https://github.com/astronomer/airflow-chart)
>>>>> > > > > that is
>>>>> > > > > > > > > > being
>>>>> > > > > > > > > > > > > used
>>>>> > > > > > > > > > > > > > by
>>>>> > > > > > > > > > > > > > > > > both
>>>>> > > > > > > > > > > > > > > > > > our
>>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers.
>>>>> This chart is
>>>>> > > > > > > > > > battle-tested and
>>>>> > > > > > > > > > > > > > > running
>>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of
>>>>> varying
>>>>> > > > > sizes and
>>>>> > > > > > > > > > runtime
>>>>> > > > > > > > > > > > > > > > > > environments.
>>>>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate
>>>>> the issues
>>>>> > > > > that Airflow
>>>>> > > > > > > > > > > > > users
>>>>> > > > > > > > > > > > > > run
>>>>> > > > > > > > > > > > > > > > > into
>>>>> > > > > > > > > > > > > > > > > > in
>>>>> > > > > > > > > > > > > > > > > > > the real world.
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > While this chart was originally
>>>>> developed
>>>>> > > > > internally for
>>>>> > > > > > > > > > our
>>>>> > > > > > > > > > > > > > > Astronomer
>>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled
>>>>> the chart
>>>>> > > > > from the
>>>>> > > > > > > > > > rest of
>>>>> > > > > > > > > > > > > our
>>>>> > > > > > > > > > > > > > > > > > platform
>>>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater
>>>>> Airflow
>>>>> > > > > community. With
>>>>> > > > > > > > > > these
>>>>> > > > > > > > > > > > > > > changes
>>>>> > > > > > > > > > > > > > > > > in
>>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
>>>>> conversation about
>>>>> > > > > donating this
>>>>> > > > > > > > > > > > > chart
>>>>> > > > > > > > > > > > > > to
>>>>> > > > > > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > > > > > > Airflow community.
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > Some of the main features of the
>>>>> chart are:
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With
>>>>> zero
>>>>> > > > > configuration, a user
>>>>> > > > > > > > > > > > > will
>>>>> > > > > > > > > > > > > > get
>>>>> > > > > > > > > > > > > > > > > a
>>>>> > > > > > > > > > > > > > > > > > > postgres database, a default user
>>>>> and the
>>>>> > > > > > > > > > KubernetesExecutor
>>>>> > > > > > > > > > > > > > ready
>>>>> > > > > > > > > > > > > > > > > to
>>>>> > > > > > > > > > > > > > > > > > run
>>>>> > > > > > > > > > > > > > > > > > > DAGs.
>>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>>>>> optional KEDA
>>>>> > > > > > > > > > autoscaling) and
>>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We
>>>>> use this to
>>>>> > > > > share a
>>>>> > > > > > > > > > > > > > configurable
>>>>> > > > > > > > > > > > > > > > > > > connection pool size per
>>>>> deployment. Useful
>>>>> > > > > for limiting
>>>>> > > > > > > > > > > > > > > connections to
>>>>> > > > > > > > > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > > > > > > metadata database.
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A
>>>>> user can push a
>>>>> > > > > newer
>>>>> > > > > > > > > > version
>>>>> > > > > > > > > > > > > of
>>>>> > > > > > > > > > > > > > > > > > Airflow
>>>>> > > > > > > > > > > > > > > > > > > into an existing release and
>>>>> migrations will
>>>>> > > > > > > > > > automatically run
>>>>> > > > > > > > > > > > > > > > > > cleanly.
>>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
>>>>> install and
>>>>> > > > > configure a
>>>>> > > > > > > > > > > > > > > > > > statsd-exporter
>>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
>>>>> expose them to
>>>>> > > > > Prometheus
>>>>> > > > > > > > > > > > > > > > > automatically.
>>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>>>>> control the
>>>>> > > > > ResourceQuotas
>>>>> > > > > > > > > > and
>>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment so
>>>>> that no
>>>>> > > > > deployment can
>>>>> > > > > > > > > > > > > > overload
>>>>> > > > > > > > > > > > > > > a
>>>>> > > > > > > > > > > > > > > > > > > cluster.
>>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>>>>> support.
>>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>>>>> Sometimes
>>>>> > > > > > > > > > KubernetesExecutor and
>>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail
>>>>> for reasons
>>>>> > > > > other than the
>>>>> > > > > > > > > > > > > actual
>>>>> > > > > > > > > > > > > > > > > > task.
>>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things
>>>>> clean in
>>>>> > > > > Kubernetes.
>>>>> > > > > > > > > > > > > > > > > > > - Support for running locally in
>>>>> KIND
>>>>> > > > > (Kubernetes in
>>>>> > > > > > > > > > Docker).
>>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across many
>>>>> Kubernetes
>>>>> > > > > versions
>>>>> > > > > > > > > > with
>>>>> > > > > > > > > > > > > Helm
>>>>> > > > > > > > > > > > > > 2
>>>>> > > > > > > > > > > > > > > > > > and 3
>>>>> > > > > > > > > > > > > > > > > > > support.
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and
>>>>> most
>>>>> > > > > reliable way to
>>>>> > > > > > > > > > deploy
>>>>> > > > > > > > > > > > > > DAGs
>>>>> > > > > > > > > > > > > > > to
>>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at
>>>>> scale is to
>>>>> > > > > package them
>>>>> > > > > > > > > > into
>>>>> > > > > > > > > > > > > the
>>>>> > > > > > > > > > > > > > > actual
>>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared
>>>>> this chart
>>>>> > > > > towards that
>>>>> > > > > > > > > > > > > method of
>>>>> > > > > > > > > > > > > > > > > > > operation, though adding other
>>>>> methods should
>>>>> > > > > be
>>>>> > > > > > > > > > > > > straightforward.
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>>>>> community and
>>>>> > > > > would love
>>>>> > > > > > > > > > to
>>>>> > > > > > > > > > > > > see
>>>>> > > > > > > > > > > > > > > this
>>>>> > > > > > > > > > > > > > > > > > chart
>>>>> > > > > > > > > > > > > > > > > > > help others to get up and running
>>>>> on
>>>>> > > > > Kubernetes!
>>>>> > > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > > > > --
>>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>> > > > > Astronomer.io
>>>>> > > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > --
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > Jarek Potiuk
>>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>> Principal
>>>>> > > > > Software
>>>>> > > > > > > > > > Engineer
>>>>> > > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > > > M: +48 660 796 129
>>>>> <+48%20660%20796%20129>
>>>>> > > > > <+48660796129
>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>> > > > > > > > > > > > > > > > [image: Polidea] <
>>>>> https://www.polidea.com/>
>>>>> > > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > > --
>>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>>> Astronomer.io
>>>>> > > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > --
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > Jarek Potiuk
>>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>>> Principal Software
>>>>> > > > > Engineer
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>>>> <+48660796129
>>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>>> > > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > > > > --
>>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>> > > > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > --
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > Jarek Potiuk
>>>>> > > > > > > > > > Polidea | Principal Software Engineer
>>>>> > > > > > > > > >
>>>>> > > > > > > > > > M: +48 660 796 129
>>>>> > > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > >
>>>>> > > > > > > > > --
>>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>>> > > > > >
>>>>> > > > > >
>>>>> > > > > > --
>>>>> > > > > > Jarek Potiuk
>>>>> > > > > > Polidea | Principal Software Engineer
>>>>> > > > > > M: +48 660 796 129
>>>>> > > > > >
>>>>> > > > >
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> >
>>>>> > Jarek Potiuk
>>>>> > Polidea | Principal Software Engineer
>>>>> >
>>>>> > M: +48 660 796 129
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Jarek Potiuk
>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>>
>>>> M: +48 660 796 129 <+48660796129>
>>>> [image: Polidea] <https://www.polidea.com/>
>>>>
>>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Greg Neiheisel <gr...@astronomer.io.INVALID>.
Hey everyone, just wanted to post a quick update on the helm chart. We're
in the process of removing any references to Astronomer specific things and
making them more general features. That work is happening on this PR here
https://github.com/astronomer/airflow-chart/pull/66. Hoping to wrap up the
last few outstanding items and open up a PR to Airflow next week sometime!
On Fri, Apr 3, 2020 at 10:19 AM Daniel Imberman <da...@gmail.com>
wrote:
> Hi Jarek,
>
> Congrats on getting the image out! This is going to make a lot of things
> easier :). We’ve actually been working on separating out KinD as well (and
> in fact moving it out of travis and into GitHub issues). Let me circle back
> with Steven (the engineer who’s been working on that) and get a status
> report from him.
>
> I will also take some time today to get our helm chart to work with
> the airflow master image. It’s been a bit crazy around here in the last few
> weeks but I am consistently pushing to make sure we make the final changes
> of ripping out any remaining platform components before we donate :).
>
> I agree that docker hub automation will be great. This will also allow us
> to have a nightly build for any integration/UI tests we build.
>
> Thanks again Jarek!
>
> Daniel
>
> via Newton Mail
> <https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2>
>
> On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> Hello Greg and Daniel,
>
> Production image is merged and it's also going to be released in 1.10.10.
> For now, I will manually tag the release in DockerHub when we release but
> we will automate that in the future. Currently, we have *airflow:master*
> and *airflow:v1-10-test* images (for all supported python versions) that
> you can use for testing with the Helm chart!
>
> The next things from my side:
>
> * I will automate DockerHub building for the prod image from master - for
> now only CI is automated but we need to add prod image as well
> * I will move kind-cluster creation outside of the container and use
> production image for our CI Kubernetes tests. I have a WIP branch on it and
> I think I should have it early next week
> * The next step will be to switch our CI to Github Actions finally. I
> think we will run both Travis and GA for a short time to make sure it works
> * I will ask people for options they would like to have in the image -
> additional packages etc. so that I can work on the next iteration of the
> image. Also if you have any needs to see for the Helm integration, I am
> super happy to incorporate them in the next version. I plan to keep the
> "master" and "v1-10-test" images updated with any new changes and push
> images regularly
> * I am happy to help with the Helm Chart, so let me know if you need
> anything!
>
> J.
>
>
> On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Hello Daniel and Greg.
>>
>> I think the production image for Airflow is quite ready for the final
>> review and merge. I did some tests and I think it's quite complete now.
>>
>> The PR is running it's Travis build:
>> https://github.com/apache/airflow/pull/7832
>>
>> I already pushed the image to DockerHub - so that the next time it is
>> merged It will use it as a base and will be rather quickly built using
>> those images as a cache. Once it is merged, I will add the automated build
>> in DockerHub and merge it to 1.10. As the next step I also plan to use the
>> prod image for our Kubernetes tests (that will help with moving to the
>> GitHub Actions from Travis).
>>
>> I added quite a comprehensive documentation - explaining all ins- and
>> outs- of both CI and PROD images
>> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
>> .
>>
>> You can build it yourself using manual build process described in the doc
>> or (easier) using Breeze. That includes using the same Dockerfile to build
>> older Airflow version from the 1.10.* line.
>>
>> Feel free to use the new image in the Helm Chart you have - happy to
>> review the PRs. The image is available at 'apache/airflow:master-python3.6'
>> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
>> release 1.10.10 we will also have it automatically published for 1.10.10
>> and beyond.
>>
>> Happy to hear your review comments and hope to merge it very soon.
>> J.
>>
>> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <Ja...@polidea.com>
>> wrote:
>>
>>> Absolutely! Please do :). The more eyes on this the better!
>>>
>>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
>>> daniel.imberman@gmail.com> wrote:
>>>
>>>> @Jarek Steven and I will look into getting KinD to work on GitHub
>>>> actions today. If we can get that working, we can donate the helm chart and
>>>> possibly even deprecate the old Kubernetes tests in the process.
>>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
>>>> wrote:
>>>> > We have A POC for Github Actions done by Tomek, and the blocker there
>>>> > was precisely Kubernetes tests and specifically running kind cluster
>>>> > in the Github Actions
>>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the meantime
>>>> > Jiaje Zhong is trying to make it works on GA we discussed this very
>>>> > thing today at slack:
>>>> > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>>>> >
>>>> > For me this is the next thing I will look at after merging
>>>> > requirements PR and Prod image PR (and those two are a bit
>>>> > pre-requisites to make it stable and fast). I want to switch to the
>>>> > Production Image to run Kubernetes tests (they will be much faster)
>>>> > and then we can work out much faster how to run them on Github
>>>> > Actions. Then we can add Helm charts to run all those different
>>>> > deployments. I am really looking forward to it!
>>>> >
>>>> > I hope - this weekend I move forward all of that and have a PR that we
>>>> > can start running tests on and replacing the resources with helm
>>>> > chart.
>>>> >
>>>> > J.
>>>> >
>>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>>>> > <da...@gmail.com> wrote:
>>>> > >
>>>> > > @Steven glad to help you out on that.
>>>> > >
>>>> > > For the moment we only use travis, but I believe @jarek has been
>>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first
>>>> GitHub tests?
>>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>>>> <st...@astronomer.io.invalid>, wrote:
>>>> > > > Hey I’m happy to PR in the chart and get the CI working to test
>>>> it. Where
>>>> > > > do we want to run it? Do you all just use Travis only?
>>>> > > >
>>>> > > > Steven
>>>> > > >
>>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>>>> daniel.imberman@gmail.com>
>>>> > > > wrote:
>>>> > > >
>>>> > > > > @Jarek I think with the helm chart + prod image we can go even
>>>> further
>>>> > > > > than that :). We can test CeleryExecutor, with KEDA
>>>> autoscaling, and a
>>>> > > > > bunch of other configurations.
>>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>>>> Jarek.Potiuk@polidea.com>,
>>>> > > > > wrote:
>>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original email
>>>> :)
>>>> > > > > >
>>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>>>> > > > > daniel.imberman@gmail.com> wrote:
>>>> > > > > > > > We’re not using CRDs for the tests at the moment. We just
>>>> have
>>>> > > > > deployment files. If anything having the helm chart as a part
>>>> of the
>>>> > > > > airflow repo could mean that the helm chart becomes the defacto
>>>> system for
>>>> > > > > testing airflow on kubernetes (we can get rid of all the yams
>>>> files and run
>>>> > > > > multiple k8s tests with different settings).
>>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>>>> > > > > <gr...@astronomer.io.invalid>, wrote:
>>>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo.
>>>> We've also
>>>> > > > > been
>>>> > > > > > > > > building up a test suite that currently runs on
>>>> CircleCI and uses
>>>> > > > > kind
>>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>>>> versions with
>>>> > > > > some
>>>> > > > > > > > > different settings. Right now we're mostly testing the
>>>> different
>>>> > > > > executors
>>>> > > > > > > > > since that has the biggest impact on what gets
>>>> deployed, but that
>>>> > > > > can be
>>>> > > > > > > > > expanded.
>>>> > > > > > > > >
>>>> > > > > > > > > What CRDs are currently being used to run Airflow for
>>>> the tests?
>>>> > > > > > > > >
>>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>>>> > > > > Jarek.Potiuk@polidea.com>
>>>> > > > > > > > > wrote:
>>>> > > > > > > > >
>>>> > > > > > > > > > One thing for the donation.
>>>> > > > > > > > > >
>>>> > > > > > > > > > Did you you want to have separate repository Greg ?
>>>> > > > > > > > > >
>>>> > > > > > > > > > I think we should simply create a folder in Airflow
>>>> repo and
>>>> > > > > keep it
>>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going
>>>> to switch our
>>>> > > > > > > > > > Kubernetes Tests to the production image (will make
>>>> the tests
>>>> > > > > much
>>>> > > > > > > > > > faster) and I am going to test the Dockerfile
>>>> automatically in
>>>> > > > > CI -
>>>> > > > > > > > > > for now we are using some custom Resource definitions
>>>> to start
>>>> > > > > Airflow
>>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>>>> switch to
>>>> > > > > using the
>>>> > > > > > > > > > helm chart - this way we can test all three things at
>>>> once:
>>>> > > > > > > > > > - Kubernetes Executor
>>>> > > > > > > > > > - Dockerfile
>>>> > > > > > > > > > - Helm Chart
>>>> > > > > > > > > > and we could also add more tests - for example
>>>> testing different
>>>> > > > > > > > > > deployment options for the helm chart.
>>>> > > > > > > > > >
>>>> > > > > > > > > > Having the Helm chart in Airflow repo would help with
>>>> that -
>>>> > > > > > > > > > especially in terms of future changes and testing them
>>>> > > > > automatically.
>>>> > > > > > > > > >
>>>> > > > > > > > > > J.
>>>> > > > > > > > > >
>>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
>>>> > > > > > > > > > <ai...@apache.org> wrote:
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > +1 on the donation. Always happy to see more useful
>>>> stuff for
>>>> > > > > the
>>>> > > > > > > > > > > community :)
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>>>> > > > > greg@astronomer.io>
>>>> > > > > > > > > > wrote:
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as an
>>>> optional
>>>> > > > > Kubernetes
>>>> > > > > > > > > > > > CronJob (
>>>> > > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > >
>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>>>> > > > > > > > > > )
>>>> > > > > > > > > > > > that we have run periodically to clean failed
>>>> pods up and
>>>> > > > > could stay
>>>> > > > > > > > > > > > separate.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > The wait_for_migrations script could definitely
>>>> be pulled
>>>> > > > > into Airflow.
>>>> > > > > > > > > > > > For context, we deploy an initContainer on the
>>>> scheduler (
>>>> > > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > >
>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>>>> > > > > > > > > > )
>>>> > > > > > > > > > > > that runs the upgradedb command before booting the
>>>> > > > > scheduler. This new
>>>> > > > > > > > > > > > wait_for_migration script runs in an
>>>> initContainer on the
>>>> > > > > webserver and
>>>> > > > > > > > > > > > workers (
>>>> > > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > >
>>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>>>> > > > > > > > > > )
>>>> > > > > > > > > > > > so that they don't boot up ahead of a potentially
>>>> > > > > long-running
>>>> > > > > > > > > > migration
>>>> > > > > > > > > > > > and attempt to operate on new or missing
>>>> columns/tables
>>>> > > > > before the
>>>> > > > > > > > > > > > migrations run. This prevents these pods from
>>>> entering a
>>>> > > > > CrashLoop.
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>>>> > > > > > > > > > > > wrote:
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
>>>> currently
>>>> > > > > generated from
>>>> > > > > > > > > > > > > Dockerfiles
>>>> > > > > > > > > > > > > > in this repo
>>>> https://github.com/astronomer/ap-airflow
>>>> > > > > and get
>>>> > > > > > > > > > > > > published to
>>>> > > > > > > > > > > > > > DockerHub
>>>> > > > > > > > > > > > > >
>>>> > > > >
>>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > For the most part those are typical Airflow
>>>> images.
>>>> > > > > There's an
>>>> > > > > > > > > > > > > entrypoint
>>>> > > > > > > > > > > > > > script that we include in the image that
>>>> handles waiting
>>>> > > > > for the
>>>> > > > > > > > > > > > > database
>>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
>>>> pretty generic.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > I already added waiting for the database (both
>>>> metadata
>>>> > > > > and celery
>>>> > > > > > > > > > URL) in
>>>> > > > > > > > > > > > > the PR:
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > >
>>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>>> > > > > > > > > > > > > .
>>>> > > > > > > > > > > > > It's functionally the same but more generic.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > The only other
>>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses would
>>>> be the
>>>> > > > > scripts in this
>>>> > > > > > > > > > repo
>>>> > > > > > > > > > > > > >
>>>> https://github.com/astronomer/astronomer-airflow-scripts.
>>>> > > > > Our
>>>> > > > > > > > > > > > > Dockerfiles
>>>> > > > > > > > > > > > > > pull this package in. These scripts are used
>>>> to
>>>> > > > > coordinate running
>>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > I see two scripts:
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not
>>>> needed to run in
>>>> > > > > airflow -
>>>> > > > > > > > > > this
>>>> > > > > > > > > > > > > could be run as a separate pod/container?
>>>> > > > > > > > > > > > > * waiting for migrations -> I think this is a
>>>> good
>>>> > > > > candidate to add
>>>> > > > > > > > > > > > > *airflow
>>>> > > > > > > > > > > > > db wait_for_migration* command and make it part
>>>> of airflow
>>>> > > > > itself.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > I think we also have to agree on the Airflow
>>>> version
>>>> > > > > supported by the
>>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>>>> 1.10.10+ and we
>>>> > > > > > > > > > incorporate
>>>> > > > > > > > > > > > > all
>>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>>>> > > > > wait_for_migration")
>>>> > > > > > > > > > into 2.0
>>>> > > > > > > > > > > > > and 1.10 and we support both - image and helm
>>>> chart for
>>>> > > > > those versions
>>>> > > > > > > > > > > > > only. That would help with people migrating to
>>>> the latest
>>>> > > > > version.
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > WDYT?
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>>>> Imberman <
>>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>>>> > > > > > > > > > > > > > wrote:
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>>>> pairing an
>>>> > > > > official helm
>>>> > > > > > > > > > chart
>>>> > > > > > > > > > > > > > > with the official image would make for a
>>>> REALLY
>>>> > > > > powerful “up and
>>>> > > > > > > > > > > > > running
>>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have
>>>> also been
>>>> > > > > looking into
>>>> > > > > > > > > > > > > > > operator-sdk which has the ability to
>>>> create custom
>>>> > > > > controllers
>>>> > > > > > > > > > from
>>>> > > > > > > > > > > > > helm
>>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
>>>> punch from the
>>>> > > > > same code
>>>> > > > > > > > > > base
>>>> > > > > > > > > > > > > :).
>>>> > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no
>>>> issues, can
>>>> > > > > we please
>>>> > > > > > > > > > start
>>>> > > > > > > > > > > > > > the
>>>> > > > > > > > > > > > > > > process of donation?
>>>> > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > +1 on my part, of course :)
>>>> > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > Daniel
>>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek
>>>> Potiuk <
>>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>>>> > > > > > > > > > > > > > > wrote:
>>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>>>> official image
>>>> > > > > we have
>>>> > > > > > > > > > work in
>>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>>>> Astronomer's
>>>> > > > > image while
>>>> > > > > > > > > > > > > preparing
>>>> > > > > > > > > > > > > > my
>>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>>>> needed to make
>>>> > > > > it works
>>>> > > > > > > > > > with
>>>> > > > > > > > > > > > > the
>>>> > > > > > > > > > > > > > > helm
>>>> > > > > > > > > > > > > > > > chart - and I am super happy to
>>>> collaborate on that.
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > PR here:
>>>> https://github.com/apache/airflow/pull/7832
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > J.
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil
>>>> Naik <
>>>> > > > > kaxilnaik@gmail.com
>>>> > > > > > > > > > >
>>>> > > > > > > > > > > > > > wrote:
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>>>> tomasz.urbaszek@polidea.com> :
>>>> > > > > > > > > > > > > > > > > Helm Chart Link:
>>>> > > > > https://github.com/astronomer/airflow-chart
>>>> > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz
>>>> Urbaszek <
>>>> > > > > > > > > > > > > > turbaszek@apache.org>
>>>> > > > > > > > > > > > > > > > > wrote:
>>>> > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > An official helm chart is something
>>>> our
>>>> > > > > community needs!
>>>> > > > > > > > > > Using
>>>> > > > > > > > > > > > > your
>>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot of
>>>> sens to me
>>>> > > > > because as
>>>> > > > > > > > > > you
>>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>>>> > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > One question: what Airflow image do
>>>> you use?
>>>> > > > > Also, would you
>>>> > > > > > > > > > > > > mind
>>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>>>> > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > Tomek
>>>> > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg
>>>> Neiheisel
>>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > Hey everyone,
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > Over the past few years at
>>>> Astronomer, we’ve
>>>> > > > > created,
>>>> > > > > > > > > > managed,
>>>> > > > > > > > > > > > > > and
>>>> > > > > > > > > > > > > > > > > > hardened
>>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for
>>>> Airflow (
>>>> > > > > > > > > > > > > > > > > > >
>>>> https://github.com/astronomer/airflow-chart)
>>>> > > > > that is
>>>> > > > > > > > > > being
>>>> > > > > > > > > > > > > used
>>>> > > > > > > > > > > > > > by
>>>> > > > > > > > > > > > > > > > > both
>>>> > > > > > > > > > > > > > > > > > our
>>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This
>>>> chart is
>>>> > > > > > > > > > battle-tested and
>>>> > > > > > > > > > > > > > > running
>>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of
>>>> varying
>>>> > > > > sizes and
>>>> > > > > > > > > > runtime
>>>> > > > > > > > > > > > > > > > > > environments.
>>>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate
>>>> the issues
>>>> > > > > that Airflow
>>>> > > > > > > > > > > > > users
>>>> > > > > > > > > > > > > > run
>>>> > > > > > > > > > > > > > > > > into
>>>> > > > > > > > > > > > > > > > > > in
>>>> > > > > > > > > > > > > > > > > > > the real world.
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > While this chart was originally
>>>> developed
>>>> > > > > internally for
>>>> > > > > > > > > > our
>>>> > > > > > > > > > > > > > > Astronomer
>>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled
>>>> the chart
>>>> > > > > from the
>>>> > > > > > > > > > rest of
>>>> > > > > > > > > > > > > our
>>>> > > > > > > > > > > > > > > > > > platform
>>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater
>>>> Airflow
>>>> > > > > community. With
>>>> > > > > > > > > > these
>>>> > > > > > > > > > > > > > > changes
>>>> > > > > > > > > > > > > > > > > in
>>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
>>>> conversation about
>>>> > > > > donating this
>>>> > > > > > > > > > > > > chart
>>>> > > > > > > > > > > > > > to
>>>> > > > > > > > > > > > > > > the
>>>> > > > > > > > > > > > > > > > > > > Airflow community.
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > Some of the main features of the
>>>> chart are:
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With zero
>>>> > > > > configuration, a user
>>>> > > > > > > > > > > > > will
>>>> > > > > > > > > > > > > > get
>>>> > > > > > > > > > > > > > > > > a
>>>> > > > > > > > > > > > > > > > > > > postgres database, a default user
>>>> and the
>>>> > > > > > > > > > KubernetesExecutor
>>>> > > > > > > > > > > > > > ready
>>>> > > > > > > > > > > > > > > > > to
>>>> > > > > > > > > > > > > > > > > > run
>>>> > > > > > > > > > > > > > > > > > > DAGs.
>>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>>>> optional KEDA
>>>> > > > > > > > > > autoscaling) and
>>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We
>>>> use this to
>>>> > > > > share a
>>>> > > > > > > > > > > > > > configurable
>>>> > > > > > > > > > > > > > > > > > > connection pool size per
>>>> deployment. Useful
>>>> > > > > for limiting
>>>> > > > > > > > > > > > > > > connections to
>>>> > > > > > > > > > > > > > > > > > the
>>>> > > > > > > > > > > > > > > > > > > metadata database.
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A user
>>>> can push a
>>>> > > > > newer
>>>> > > > > > > > > > version
>>>> > > > > > > > > > > > > of
>>>> > > > > > > > > > > > > > > > > > Airflow
>>>> > > > > > > > > > > > > > > > > > > into an existing release and
>>>> migrations will
>>>> > > > > > > > > > automatically run
>>>> > > > > > > > > > > > > > > > > > cleanly.
>>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
>>>> install and
>>>> > > > > configure a
>>>> > > > > > > > > > > > > > > > > > statsd-exporter
>>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and
>>>> expose them to
>>>> > > > > Prometheus
>>>> > > > > > > > > > > > > > > > > automatically.
>>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>>>> control the
>>>> > > > > ResourceQuotas
>>>> > > > > > > > > > and
>>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment so
>>>> that no
>>>> > > > > deployment can
>>>> > > > > > > > > > > > > > overload
>>>> > > > > > > > > > > > > > > a
>>>> > > > > > > > > > > > > > > > > > > cluster.
>>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>>>> support.
>>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>>>> Sometimes
>>>> > > > > > > > > > KubernetesExecutor and
>>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for
>>>> reasons
>>>> > > > > other than the
>>>> > > > > > > > > > > > > actual
>>>> > > > > > > > > > > > > > > > > > task.
>>>> > > > > > > > > > > > > > > > > > > This feature helps keep things
>>>> clean in
>>>> > > > > Kubernetes.
>>>> > > > > > > > > > > > > > > > > > > - Support for running locally in
>>>> KIND
>>>> > > > > (Kubernetes in
>>>> > > > > > > > > > Docker).
>>>> > > > > > > > > > > > > > > > > > > - Automatically tested across many
>>>> Kubernetes
>>>> > > > > versions
>>>> > > > > > > > > > with
>>>> > > > > > > > > > > > > Helm
>>>> > > > > > > > > > > > > > 2
>>>> > > > > > > > > > > > > > > > > > and 3
>>>> > > > > > > > > > > > > > > > > > > support.
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and
>>>> most
>>>> > > > > reliable way to
>>>> > > > > > > > > > deploy
>>>> > > > > > > > > > > > > > DAGs
>>>> > > > > > > > > > > > > > > to
>>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale
>>>> is to
>>>> > > > > package them
>>>> > > > > > > > > > into
>>>> > > > > > > > > > > > > the
>>>> > > > > > > > > > > > > > > actual
>>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared
>>>> this chart
>>>> > > > > towards that
>>>> > > > > > > > > > > > > method of
>>>> > > > > > > > > > > > > > > > > > > operation, though adding other
>>>> methods should
>>>> > > > > be
>>>> > > > > > > > > > > > > straightforward.
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>>>> community and
>>>> > > > > would love
>>>> > > > > > > > > > to
>>>> > > > > > > > > > > > > see
>>>> > > > > > > > > > > > > > > this
>>>> > > > > > > > > > > > > > > > > > chart
>>>> > > > > > > > > > > > > > > > > > > help others to get up and running on
>>>> > > > > Kubernetes!
>>>> > > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > > > > --
>>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>> > > > > Astronomer.io
>>>> > > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > --
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > Jarek Potiuk
>>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>>> Principal
>>>> > > > > Software
>>>> > > > > > > > > > Engineer
>>>> > > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>>> > > > > <+48660796129
>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>> > > > > > > > > > > > > > > > [image: Polidea] <
>>>> https://www.polidea.com/>
>>>> > > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > > > --
>>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>>> Astronomer.io
>>>> > > > > > > > > > > > > >
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > --
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > Jarek Potiuk
>>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
>>>> Software
>>>> > > > > Engineer
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>>> <+48660796129
>>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>>> > > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > >
>>>> > > > > > > > > > > > --
>>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>> > > > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > >
>>>> > > > > > > > > > --
>>>> > > > > > > > > >
>>>> > > > > > > > > > Jarek Potiuk
>>>> > > > > > > > > > Polidea | Principal Software Engineer
>>>> > > > > > > > > >
>>>> > > > > > > > > > M: +48 660 796 129
>>>> > > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > >
>>>> > > > > > > > > --
>>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>>> > > > > >
>>>> > > > > >
>>>> > > > > > --
>>>> > > > > > Jarek Potiuk
>>>> > > > > > Polidea | Principal Software Engineer
>>>> > > > > > M: +48 660 796 129
>>>> > > > > >
>>>> > > > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Jarek Potiuk
>>>> > Polidea | Principal Software Engineer
>>>> >
>>>> > M: +48 660 796 129
>>>>
>>>
>>>
>>> --
>>>
>>> Jarek Potiuk
>>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>>
>>> M: +48 660 796 129 <+48660796129>
>>> [image: Polidea] <https://www.polidea.com/>
>>>
>>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Daniel Imberman <da...@gmail.com>.
Hi Jarek,
Congrats on getting the image out! This is going to make a lot of things easier :). We’ve actually been working on separating out KinD as well (and in fact moving it out of travis and into GitHub issues). Let me circle back with Steven (the engineer who’s been working on that) and get a status report from him.
I will also take some time today to get our helm chart to work with the airflow master image. It’s been a bit crazy around here in the last few weeks but I am consistently pushing to make sure we make the final changes of ripping out any remaining platform components before we donate :).
I agree that docker hub automation will be great. This will also allow us to have a nightly build for any integration/UI tests we build.
Thanks again Jarek!
Daniel
via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2]
On Fri, Apr 3, 2020 at 1:04 AM, Jarek Potiuk <Ja...@polidea.com> wrote:
Hello Greg and Daniel,
Production image is merged and it's also going to be released in 1.10.10. For now, I will manually tag the release in DockerHub when we release but we will automate that in the future. Currently, we have airflow:master and airflow:v1-10-test images (for all supported python versions) that you can use for testing with the Helm chart!
The next things from my side:
* I will automate DockerHub building for the prod image from master - for now only CI is automated but we need to add prod image as well * I will move kind-cluster creation outside of the container and use production image for our CI Kubernetes tests. I have a WIP branch on it and I think I should have it early next week * The next step will be to switch our CI to Github Actions finally. I think we will run both Travis and GA for a short time to make sure it works * I will ask people for options they would like to have in the image - additional packages etc. so that I can work on the next iteration of the image. Also if you have any needs to see for the Helm integration, I am super happy to incorporate them in the next version. I plan to keep the "master" and "v1-10-test" images updated with any new changes and push images regularly * I am happy to help with the Helm Chart, so let me know if you need anything!
J.
On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk < Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] > wrote:
Hello Daniel and Greg.
I think the production image for Airflow is quite ready for the final review and merge. I did some tests and I think it's quite complete now.
The PR is running it's Travis build: https://github.com/apache/airflow/pull/7832 [https://github.com/apache/airflow/pull/7832]
I already pushed the image to DockerHub - so that the next time it is merged It will use it as a base and will be rather quickly built using those images as a cache. Once it is merged, I will add the automated build in DockerHub and merge it to 1.10. As the next step I also plan to use the prod image for our Kubernetes tests (that will help with moving to the GitHub Actions from Travis).
I added quite a comprehensive documentation - explaining all ins- and outs- of both CI and PROD images https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst [https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst] .
You can build it yourself using manual build process described in the doc or (easier) using Breeze. That includes using the same Dockerfile to build older Airflow version from the 1.10.* line.
Feel free to use the new image in the Helm Chart you have - happy to review the PRs. The image is available at 'apache/airflow:master-python3.6' and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and release 1.10.10 we will also have it automatically published for 1.10.10 and beyond.
Happy to hear your review comments and hope to merge it very soon. J.
On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk < Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] > wrote:
Absolutely! Please do :). The more eyes on this the better!
On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman < daniel.imberman@gmail.com [daniel.imberman@gmail.com] > wrote:
@Jarek Steven and I will look into getting KinD to work on GitHub actions today. If we can get that working, we can donate the helm chart and possibly even deprecate the old Kubernetes tests in the process.
On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk < Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] >, wrote:
> We have A POC for Github Actions done by Tomek, and the blocker there
> was precisely Kubernetes tests and specifically running kind cluster
> in the Github Actions
> https://github.com/PolideaInternal/airflow/pull/542 [https://github.com/PolideaInternal/airflow/pull/542] . In the meantime
> Jiaje Zhong is trying to make it works on GA we discussed this very
> thing today at slack:
> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200 [https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200]
>
> For me this is the next thing I will look at after merging
> requirements PR and Prod image PR (and those two are a bit
> pre-requisites to make it stable and fast). I want to switch to the
> Production Image to run Kubernetes tests (they will be much faster)
> and then we can work out much faster how to run them on Github
> Actions. Then we can add Helm charts to run all those different
> deployments. I am really looking forward to it!
>
> I hope - this weekend I move forward all of that and have a PR that we
> can start running tests on and replacing the resources with helm
> chart.
>
> J.
>
> On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> < daniel.imberman@gmail.com [daniel.imberman@gmail.com] > wrote:
> >
> > @Steven glad to help you out on that.
> >
> > For the moment we only use travis, but I believe @jarek has been looking to move to GitHub CI/CD. Perhaps we can use this as our first GitHub tests?
> > On Mar 26, 2020, 8:17 AM -0700, Steven Miller <st...@astronomer.io.invalid>, wrote:
> > > Hey I’m happy to PR in the chart and get the CI working to test it. Where
> > > do we want to run it? Do you all just use Travis only?
> > >
> > > Steven
> > >
> > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman < daniel.imberman@gmail.com [daniel.imberman@gmail.com] >
> > > wrote:
> > >
> > > > @Jarek I think with the helm chart + prod image we can go even further
> > > > than that :). We can test CeleryExecutor, with KEDA autoscaling, and a
> > > > bunch of other configurations.
> > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk < Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] >,
> > > > wrote:
> > > > > Yeah. I meant Custom Resources not CRDs in my original email :)
> > > > >
> > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > > > daniel.imberman@gmail.com [daniel.imberman@gmail.com] > wrote:
> > > > > > > We’re not using CRDs for the tests at the moment. We just have
> > > > deployment files. If anything having the helm chart as a part of the
> > > > airflow repo could mean that the helm chart becomes the defacto system for
> > > > testing airflow on kubernetes (we can get rid of all the yams files and run
> > > > multiple k8s tests with different settings).
> > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > > > <gr...@astronomer.io.invalid>, wrote:
> > > > > > > > Yep, we can absolutely pull it into the airflow repo. We've also
> > > > been
> > > > > > > > building up a test suite that currently runs on CircleCI and uses
> > > > kind
> > > > > > > > (Kubernetes in Docker) to test several kubernetes versions with
> > > > some
> > > > > > > > different settings. Right now we're mostly testing the different
> > > > executors
> > > > > > > > since that has the biggest impact on what gets deployed, but that
> > > > can be
> > > > > > > > expanded.
> > > > > > > >
> > > > > > > > What CRDs are currently being used to run Airflow for the tests?
> > > > > > > >
> > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > One thing for the donation.
> > > > > > > > >
> > > > > > > > > Did you you want to have separate repository Greg ?
> > > > > > > > >
> > > > > > > > > I think we should simply create a folder in Airflow repo and
> > > > keep it
> > > > > > > > > there (similarly as we keep Dockerfile). I am going to switch our
> > > > > > > > > Kubernetes Tests to the production image (will make the tests
> > > > much
> > > > > > > > > faster) and I am going to test the Dockerfile automatically in
> > > > CI -
> > > > > > > > > for now we are using some custom Resource definitions to start
> > > > Airflow
> > > > > > > > > on Kubernetes Cluster for the tests, but we could switch to
> > > > using the
> > > > > > > > > helm chart - this way we can test all three things at once:
> > > > > > > > > - Kubernetes Executor
> > > > > > > > > - Dockerfile
> > > > > > > > > - Helm Chart
> > > > > > > > > and we could also add more tests - for example testing different
> > > > > > > > > deployment options for the helm chart.
> > > > > > > > >
> > > > > > > > > Having the Helm chart in Airflow repo would help with that -
> > > > > > > > > especially in terms of future changes and testing them
> > > > automatically.
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > > > > > < aizhamal@apache.org [aizhamal@apache.org] > wrote:
> > > > > > > > > >
> > > > > > > > > > +1 on the donation. Always happy to see more useful stuff for
> > > > the
> > > > > > > > > > community :)
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
> > > > greg@astronomer.io [greg@astronomer.io] >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Yep, the cleanup_pods script is set up now as an optional
> > > > Kubernetes
> > > > > > > > > > > CronJob (
> > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml [https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml]
> > > > > > > > > )
> > > > > > > > > > > that we have run periodically to clean failed pods up and
> > > > could stay
> > > > > > > > > > > separate.
> > > > > > > > > > >
> > > > > > > > > > > The wait_for_migrations script could definitely be pulled
> > > > into Airflow.
> > > > > > > > > > > For context, we deploy an initContainer on the scheduler (
> > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84 [https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84]
> > > > > > > > > )
> > > > > > > > > > > that runs the upgradedb command before booting the
> > > > scheduler. This new
> > > > > > > > > > > wait_for_migration script runs in an initContainer on the
> > > > webserver and
> > > > > > > > > > > workers (
> > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65 [https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65]
> > > > > > > > > )
> > > > > > > > > > > so that they don't boot up ahead of a potentially
> > > > long-running
> > > > > > > > > migration
> > > > > > > > > > > and attempt to operate on new or missing columns/tables
> > > > before the
> > > > > > > > > > > migrations run. This prevents these pods from entering a
> > > > CrashLoop.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > > > > > Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Tomasz great question. Our images are currently
> > > > generated from
> > > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > > in this repo https://github.com/astronomer/ap-airflow [https://github.com/astronomer/ap-airflow]
> > > > and get
> > > > > > > > > > > > published to
> > > > > > > > > > > > > DockerHub
> > > > > > > > > > > > >
> > > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow [https://hub.docker.com/repository/docker/astronomerinc/ap-airflow] .
> > > > > > > > > > > > >
> > > > > > > > > > > > > For the most part those are typical Airflow images.
> > > > There's an
> > > > > > > > > > > > entrypoint
> > > > > > > > > > > > > script that we include in the image that handles waiting
> > > > for the
> > > > > > > > > > > > database
> > > > > > > > > > > > > and redis (if used) to come up, which is pretty generic.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I already added waiting for the database (both metadata
> > > > and celery
> > > > > > > > > URL) in
> > > > > > > > > > > > the PR:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741 [https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741]
> > > > > > > > > > > > .
> > > > > > > > > > > > It's functionally the same but more generic.
> > > > > > > > > > > >
> > > > > > > > > > > > The only other
> > > > > > > > > > > > > thing that I think the Helm Chart uses would be the
> > > > scripts in this
> > > > > > > > > repo
> > > > > > > > > > > > > https://github.com/astronomer/astronomer-airflow-scripts [https://github.com/astronomer/astronomer-airflow-scripts] .
> > > > Our
> > > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > > pull this package in. These scripts are used to
> > > > coordinate running
> > > > > > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I see two scripts:
> > > > > > > > > > > >
> > > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed to run in
> > > > airflow -
> > > > > > > > > this
> > > > > > > > > > > > could be run as a separate pod/container?
> > > > > > > > > > > > * waiting for migrations -> I think this is a good
> > > > candidate to add
> > > > > > > > > > > > *airflow
> > > > > > > > > > > > db wait_for_migration* command and make it part of airflow
> > > > itself.
> > > > > > > > > > > >
> > > > > > > > > > > > I think we also have to agree on the Airflow version
> > > > supported by the
> > > > > > > > > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > > > > > > > > incorporate
> > > > > > > > > > > > all
> > > > > > > > > > > > the changes needed to airflow (like the "db
> > > > wait_for_migration")
> > > > > > > > > into 2.0
> > > > > > > > > > > > and 1.10 and we support both - image and helm chart for
> > > > those versions
> > > > > > > > > > > > only. That would help with people migrating to the latest
> > > > version.
> > > > > > > > > > > >
> > > > > > > > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > > > > > > > > daniel.imberman@gmail.com [daniel.imberman@gmail.com] >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > @jarek I agree completely. I think that pairing an
> > > > official helm
> > > > > > > > > chart
> > > > > > > > > > > > > > with the official image would make for a REALLY
> > > > powerful “up and
> > > > > > > > > > > > running
> > > > > > > > > > > > > > with airflow” story :). Tomek and I have also been
> > > > looking into
> > > > > > > > > > > > > > operator-sdk which has the ability to create custom
> > > > controllers
> > > > > > > > > from
> > > > > > > > > > > > helm
> > > > > > > > > > > > > > charts. We might even able to get a 1-2 punch from the
> > > > same code
> > > > > > > > > base
> > > > > > > > > > > > :).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can
> > > > we please
> > > > > > > > > start
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > process of donation?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Daniel
> > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > > > > > Jarek.Potiuk@polidea.com [Jarek.Potiuk@polidea.com] >,
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > +1. And it should be paired with the official image
> > > > we have
> > > > > > > > > work in
> > > > > > > > > > > > > > > progress on. I looked a lot at the Astronomer's
> > > > image while
> > > > > > > > > > > > preparing
> > > > > > > > > > > > > my
> > > > > > > > > > > > > > > draft and we can make any adjustments needed to make
> > > > it works
> > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > helm
> > > > > > > > > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > PR here: https://github.com/apache/airflow/pull/7832 [https://github.com/apache/airflow/pull/7832]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > J.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
> > > > kaxilnaik@gmail.com [kaxilnaik@gmail.com]
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Tomasz Urbaszek < tomasz.urbaszek@polidea.com [tomasz.urbaszek@polidea.com] > :
> > > > > > > > > > > > > > > > Helm Chart Link:
> > > > https://github.com/astronomer/airflow-chart [https://github.com/astronomer/airflow-chart]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > > > > > > > > turbaszek@apache.org [turbaszek@apache.org] >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > An official helm chart is something our
> > > > community needs!
> > > > > > > > > Using
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > chart as the official makes a lot of sens to me
> > > > because as
> > > > > > > > > you
> > > > > > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > One question: what Airflow image do you use?
> > > > Also, would you
> > > > > > > > > > > > mind
> > > > > > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Tomek
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Over the past few years at Astronomer, we’ve
> > > > created,
> > > > > > > > > managed,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > > > > > > > > https://github.com/astronomer/airflow-chart [https://github.com/astronomer/airflow-chart] )
> > > > that is
> > > > > > > > > being
> > > > > > > > > > > > used
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > > > > > > > > battle-tested and
> > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of varying
> > > > sizes and
> > > > > > > > > runtime
> > > > > > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > > > > > It’s been built up to encapsulate the issues
> > > > that Airflow
> > > > > > > > > > > > users
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > While this chart was originally developed
> > > > internally for
> > > > > > > > > our
> > > > > > > > > > > > > > Astronomer
> > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled the chart
> > > > from the
> > > > > > > > > rest of
> > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > platform
> > > > > > > > > > > > > > > > > > to make it usable by the greater Airflow
> > > > community. With
> > > > > > > > > these
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > mind, we want to start a conversation about
> > > > donating this
> > > > > > > > > > > > chart
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - It works out of the box. With zero
> > > > configuration, a user
> > > > > > > > > > > > will
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > postgres database, a default user and the
> > > > > > > > > KubernetesExecutor
> > > > > > > > > > > > > ready
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > > > > > > > > autoscaling) and
> > > > > > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We use this to
> > > > share a
> > > > > > > > > > > > > configurable
> > > > > > > > > > > > > > > > > > connection pool size per deployment. Useful
> > > > for limiting
> > > > > > > > > > > > > > connections to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - Airflow migration support. A user can push a
> > > > newer
> > > > > > > > > version
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > > > > > into an existing release and migrations will
> > > > > > > > > automatically run
> > > > > > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > > > > > - Prometheus support. Optionally install and
> > > > configure a
> > > > > > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose them to
> > > > Prometheus
> > > > > > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > > > > > - Resource control. Optionally control the
> > > > ResourceQuotas
> > > > > > > > > and
> > > > > > > > > > > > > > > > > > LimitRanges for each deployment so that no
> > > > deployment can
> > > > > > > > > > > > > overload
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > > > > > KubernetesExecutor and
> > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for reasons
> > > > other than the
> > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > task.
> > > > > > > > > > > > > > > > > > This feature helps keep things clean in
> > > > Kubernetes.
> > > > > > > > > > > > > > > > > > - Support for running locally in KIND
> > > > (Kubernetes in
> > > > > > > > > Docker).
> > > > > > > > > > > > > > > > > > - Automatically tested across many Kubernetes
> > > > versions
> > > > > > > > > with
> > > > > > > > > > > > Helm
> > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We’ve found that the cleanest and most
> > > > reliable way to
> > > > > > > > > deploy
> > > > > > > > > > > > > DAGs
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale is to
> > > > package them
> > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > > docker image, so we have geared this chart
> > > > towards that
> > > > > > > > > > > > method of
> > > > > > > > > > > > > > > > > > operation, though adding other methods should
> > > > be
> > > > > > > > > > > > straightforward.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We would love thoughts from the community and
> > > > would love
> > > > > > > > > to
> > > > > > > > > > > > see
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > chart
> > > > > > > > > > > > > > > > > > help others to get up and running on
> > > > Kubernetes!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > > Astronomer.io
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > > > > Polidea < https://www.polidea.com/ [https://www.polidea.com/] > | Principal
> > > > Software
> > > > > > > > > Engineer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> > > > <+48660796129
> > > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > > > > [image: Polidea] < https://www.polidea.com/ [https://www.polidea.com/] >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > Polidea < https://www.polidea.com/ [https://www.polidea.com/] > | Principal Software
> > > > Engineer
> > > > > > > > > > > >
> > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > [image: Polidea] < https://www.polidea.com/ [https://www.polidea.com/] >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jarek Potiuk
> > > > > > > > > Polidea | Principal Software Engineer
> > > > > > > > >
> > > > > > > > > M: +48 660 796 129
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > >
> > > > >
> > > > > --
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > > M: +48 660 796 129
> > > > >
> > > >
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
--
Jarek Potiuk
Polidea [https://www.polidea.com/] | Principal Software Engineer
M: +48 660 796 129 [tel:+48660796129]
[https://www.polidea.com/]
--
Jarek Potiuk
Polidea [https://www.polidea.com/] | Principal Software Engineer
M: +48 660 796 129 [tel:+48660796129]
[https://www.polidea.com/]
--
Jarek Potiuk
Polidea [https://www.polidea.com/] | Principal Software Engineer
M: +48 660 796 129 [tel:+48660796129]
[https://www.polidea.com/]
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hello Greg and Daniel,
Production image is merged and it's also going to be released in 1.10.10.
For now, I will manually tag the release in DockerHub when we
release but we will automate that in the future. Currently, we have
*airflow:master* and *airflow:v1-10-test* images (for all supported python
versions) that you can use for testing with the Helm chart!
The next things from my side:
* I will automate DockerHub building for the prod image from master - for
now only CI is automated but we need to add prod image as well
* I will move kind-cluster creation outside of the container and use
production image for our CI Kubernetes tests. I have a WIP branch on it and
I think I should have it early next week
* The next step will be to switch our CI to Github Actions finally. I think
we will run both Travis and GA for a short time to make sure it works
* I will ask people for options they would like to have in the image -
additional packages etc. so that I can work on the next iteration of the
image. Also if you have any needs to see for the Helm integration, I am
super happy to incorporate them in the next version. I plan to keep the
"master" and "v1-10-test" images updated with any new changes and push
images regularly
* I am happy to help with the Helm Chart, so let me know if you need
anything!
J.
On Mon, Mar 30, 2020 at 11:21 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Hello Daniel and Greg.
>
> I think the production image for Airflow is quite ready for the final
> review and merge. I did some tests and I think it's quite complete now.
>
> The PR is running it's Travis build:
> https://github.com/apache/airflow/pull/7832
>
> I already pushed the image to DockerHub - so that the next time it is
> merged It will use it as a base and will be rather quickly built using
> those images as a cache. Once it is merged, I will add the automated build
> in DockerHub and merge it to 1.10. As the next step I also plan to use the
> prod image for our Kubernetes tests (that will help with moving to the
> GitHub Actions from Travis).
>
> I added quite a comprehensive documentation - explaining all ins- and
> outs- of both CI and PROD images
> https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
> .
>
> You can build it yourself using manual build process described in the doc
> or (easier) using Breeze. That includes using the same Dockerfile to
> build older Airflow version from the 1.10.* line.
>
> Feel free to use the new image in the Helm Chart you have - happy to
> review the PRs. The image is available at 'apache/airflow:master-python3.6'
> and 'apache/airflow:master-python3.7` - once we merge it to 1.10 and
> release 1.10.10 we will also have it automatically published for 1.10.10
> and beyond.
>
> Happy to hear your review comments and hope to merge it very soon.
> J.
>
> On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> Absolutely! Please do :). The more eyes on this the better!
>>
>> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <
>> daniel.imberman@gmail.com> wrote:
>>
>>> @Jarek Steven and I will look into getting KinD to work on GitHub
>>> actions today. If we can get that working, we can donate the helm chart and
>>> possibly even deprecate the old Kubernetes tests in the process.
>>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
>>> wrote:
>>> > We have A POC for Github Actions done by Tomek, and the blocker there
>>> > was precisely Kubernetes tests and specifically running kind cluster
>>> > in the Github Actions
>>> > https://github.com/PolideaInternal/airflow/pull/542 . In the meantime
>>> > Jiaje Zhong is trying to make it works on GA we discussed this very
>>> > thing today at slack:
>>> > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>>> >
>>> > For me this is the next thing I will look at after merging
>>> > requirements PR and Prod image PR (and those two are a bit
>>> > pre-requisites to make it stable and fast). I want to switch to the
>>> > Production Image to run Kubernetes tests (they will be much faster)
>>> > and then we can work out much faster how to run them on Github
>>> > Actions. Then we can add Helm charts to run all those different
>>> > deployments. I am really looking forward to it!
>>> >
>>> > I hope - this weekend I move forward all of that and have a PR that we
>>> > can start running tests on and replacing the resources with helm
>>> > chart.
>>> >
>>> > J.
>>> >
>>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>>> > <da...@gmail.com> wrote:
>>> > >
>>> > > @Steven glad to help you out on that.
>>> > >
>>> > > For the moment we only use travis, but I believe @jarek has been
>>> looking to move to GitHub CI/CD. Perhaps we can use this as our first
>>> GitHub tests?
>>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>>> <st...@astronomer.io.invalid>, wrote:
>>> > > > Hey I’m happy to PR in the chart and get the CI working to test
>>> it. Where
>>> > > > do we want to run it? Do you all just use Travis only?
>>> > > >
>>> > > > Steven
>>> > > >
>>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>>> daniel.imberman@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > @Jarek I think with the helm chart + prod image we can go even
>>> further
>>> > > > > than that :). We can test CeleryExecutor, with KEDA autoscaling,
>>> and a
>>> > > > > bunch of other configurations.
>>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>>> Jarek.Potiuk@polidea.com>,
>>> > > > > wrote:
>>> > > > > > Yeah. I meant Custom Resources not CRDs in my original email :)
>>> > > > > >
>>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>>> > > > > daniel.imberman@gmail.com> wrote:
>>> > > > > > > > We’re not using CRDs for the tests at the moment. We just
>>> have
>>> > > > > deployment files. If anything having the helm chart as a part of
>>> the
>>> > > > > airflow repo could mean that the helm chart becomes the defacto
>>> system for
>>> > > > > testing airflow on kubernetes (we can get rid of all the yams
>>> files and run
>>> > > > > multiple k8s tests with different settings).
>>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>>> > > > > <gr...@astronomer.io.invalid>, wrote:
>>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo.
>>> We've also
>>> > > > > been
>>> > > > > > > > > building up a test suite that currently runs on CircleCI
>>> and uses
>>> > > > > kind
>>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>>> versions with
>>> > > > > some
>>> > > > > > > > > different settings. Right now we're mostly testing the
>>> different
>>> > > > > executors
>>> > > > > > > > > since that has the biggest impact on what gets deployed,
>>> but that
>>> > > > > can be
>>> > > > > > > > > expanded.
>>> > > > > > > > >
>>> > > > > > > > > What CRDs are currently being used to run Airflow for
>>> the tests?
>>> > > > > > > > >
>>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>>> > > > > Jarek.Potiuk@polidea.com>
>>> > > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > One thing for the donation.
>>> > > > > > > > > >
>>> > > > > > > > > > Did you you want to have separate repository Greg ?
>>> > > > > > > > > >
>>> > > > > > > > > > I think we should simply create a folder in Airflow
>>> repo and
>>> > > > > keep it
>>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going to
>>> switch our
>>> > > > > > > > > > Kubernetes Tests to the production image (will make
>>> the tests
>>> > > > > much
>>> > > > > > > > > > faster) and I am going to test the Dockerfile
>>> automatically in
>>> > > > > CI -
>>> > > > > > > > > > for now we are using some custom Resource definitions
>>> to start
>>> > > > > Airflow
>>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>>> switch to
>>> > > > > using the
>>> > > > > > > > > > helm chart - this way we can test all three things at
>>> once:
>>> > > > > > > > > > - Kubernetes Executor
>>> > > > > > > > > > - Dockerfile
>>> > > > > > > > > > - Helm Chart
>>> > > > > > > > > > and we could also add more tests - for example testing
>>> different
>>> > > > > > > > > > deployment options for the helm chart.
>>> > > > > > > > > >
>>> > > > > > > > > > Having the Helm chart in Airflow repo would help with
>>> that -
>>> > > > > > > > > > especially in terms of future changes and testing them
>>> > > > > automatically.
>>> > > > > > > > > >
>>> > > > > > > > > > J.
>>> > > > > > > > > >
>>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
>>> > > > > > > > > > <ai...@apache.org> wrote:
>>> > > > > > > > > > >
>>> > > > > > > > > > > +1 on the donation. Always happy to see more useful
>>> stuff for
>>> > > > > the
>>> > > > > > > > > > > community :)
>>> > > > > > > > > > >
>>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>>> > > > > greg@astronomer.io>
>>> > > > > > > > > > wrote:
>>> > > > > > > > > > >
>>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as an
>>> optional
>>> > > > > Kubernetes
>>> > > > > > > > > > > > CronJob (
>>> > > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > >
>>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>>> > > > > > > > > > )
>>> > > > > > > > > > > > that we have run periodically to clean failed pods
>>> up and
>>> > > > > could stay
>>> > > > > > > > > > > > separate.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > The wait_for_migrations script could definitely be
>>> pulled
>>> > > > > into Airflow.
>>> > > > > > > > > > > > For context, we deploy an initContainer on the
>>> scheduler (
>>> > > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > >
>>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>>> > > > > > > > > > )
>>> > > > > > > > > > > > that runs the upgradedb command before booting the
>>> > > > > scheduler. This new
>>> > > > > > > > > > > > wait_for_migration script runs in an initContainer
>>> on the
>>> > > > > webserver and
>>> > > > > > > > > > > > workers (
>>> > > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > >
>>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>>> > > > > > > > > > )
>>> > > > > > > > > > > > so that they don't boot up ahead of a potentially
>>> > > > > long-running
>>> > > > > > > > > > migration
>>> > > > > > > > > > > > and attempt to operate on new or missing
>>> columns/tables
>>> > > > > before the
>>> > > > > > > > > > > > migrations run. This prevents these pods from
>>> entering a
>>> > > > > CrashLoop.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>>> > > > > > > > > > > > wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > @Tomasz great question. Our images are
>>> currently
>>> > > > > generated from
>>> > > > > > > > > > > > > Dockerfiles
>>> > > > > > > > > > > > > > in this repo
>>> https://github.com/astronomer/ap-airflow
>>> > > > > and get
>>> > > > > > > > > > > > > published to
>>> > > > > > > > > > > > > > DockerHub
>>> > > > > > > > > > > > > >
>>> > > > >
>>> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > For the most part those are typical Airflow
>>> images.
>>> > > > > There's an
>>> > > > > > > > > > > > > entrypoint
>>> > > > > > > > > > > > > > script that we include in the image that
>>> handles waiting
>>> > > > > for the
>>> > > > > > > > > > > > > database
>>> > > > > > > > > > > > > > and redis (if used) to come up, which is
>>> pretty generic.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > I already added waiting for the database (both
>>> metadata
>>> > > > > and celery
>>> > > > > > > > > > URL) in
>>> > > > > > > > > > > > > the PR:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > >
>>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>>> > > > > > > > > > > > > .
>>> > > > > > > > > > > > > It's functionally the same but more generic.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > The only other
>>> > > > > > > > > > > > > > thing that I think the Helm Chart uses would
>>> be the
>>> > > > > scripts in this
>>> > > > > > > > > > repo
>>> > > > > > > > > > > > > >
>>> https://github.com/astronomer/astronomer-airflow-scripts.
>>> > > > > Our
>>> > > > > > > > > > > > > Dockerfiles
>>> > > > > > > > > > > > > > pull this package in. These scripts are used to
>>> > > > > coordinate running
>>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > I see two scripts:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed
>>> to run in
>>> > > > > airflow -
>>> > > > > > > > > > this
>>> > > > > > > > > > > > > could be run as a separate pod/container?
>>> > > > > > > > > > > > > * waiting for migrations -> I think this is a
>>> good
>>> > > > > candidate to add
>>> > > > > > > > > > > > > *airflow
>>> > > > > > > > > > > > > db wait_for_migration* command and make it part
>>> of airflow
>>> > > > > itself.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > I think we also have to agree on the Airflow
>>> version
>>> > > > > supported by the
>>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>>> 1.10.10+ and we
>>> > > > > > > > > > incorporate
>>> > > > > > > > > > > > > all
>>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>>> > > > > wait_for_migration")
>>> > > > > > > > > > into 2.0
>>> > > > > > > > > > > > > and 1.10 and we support both - image and helm
>>> chart for
>>> > > > > those versions
>>> > > > > > > > > > > > > only. That would help with people migrating to
>>> the latest
>>> > > > > version.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > WDYT?
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>>> Imberman <
>>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>>> > > > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>>> pairing an
>>> > > > > official helm
>>> > > > > > > > > > chart
>>> > > > > > > > > > > > > > > with the official image would make for a
>>> REALLY
>>> > > > > powerful “up and
>>> > > > > > > > > > > > > running
>>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have
>>> also been
>>> > > > > looking into
>>> > > > > > > > > > > > > > > operator-sdk which has the ability to create
>>> custom
>>> > > > > controllers
>>> > > > > > > > > > from
>>> > > > > > > > > > > > > helm
>>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2
>>> punch from the
>>> > > > > same code
>>> > > > > > > > > > base
>>> > > > > > > > > > > > > :).
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no
>>> issues, can
>>> > > > > we please
>>> > > > > > > > > > start
>>> > > > > > > > > > > > > > the
>>> > > > > > > > > > > > > > > process of donation?
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > +1 on my part, of course :)
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > Daniel
>>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk
>>> <
>>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>>> > > > > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>>> official image
>>> > > > > we have
>>> > > > > > > > > > work in
>>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>>> Astronomer's
>>> > > > > image while
>>> > > > > > > > > > > > > preparing
>>> > > > > > > > > > > > > > my
>>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>>> needed to make
>>> > > > > it works
>>> > > > > > > > > > with
>>> > > > > > > > > > > > > the
>>> > > > > > > > > > > > > > > helm
>>> > > > > > > > > > > > > > > > chart - and I am super happy to
>>> collaborate on that.
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > PR here:
>>> https://github.com/apache/airflow/pull/7832
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > J.
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik
>>> <
>>> > > > > kaxilnaik@gmail.com
>>> > > > > > > > > > >
>>> > > > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>>> tomasz.urbaszek@polidea.com> :
>>> > > > > > > > > > > > > > > > > Helm Chart Link:
>>> > > > > https://github.com/astronomer/airflow-chart
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz
>>> Urbaszek <
>>> > > > > > > > > > > > > > turbaszek@apache.org>
>>> > > > > > > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > An official helm chart is something our
>>> > > > > community needs!
>>> > > > > > > > > > Using
>>> > > > > > > > > > > > > your
>>> > > > > > > > > > > > > > > > > > chart as the official makes a lot of
>>> sens to me
>>> > > > > because as
>>> > > > > > > > > > you
>>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > One question: what Airflow image do
>>> you use?
>>> > > > > Also, would you
>>> > > > > > > > > > > > > mind
>>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > Tomek
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg
>>> Neiheisel
>>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > Hey everyone,
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > Over the past few years at
>>> Astronomer, we’ve
>>> > > > > created,
>>> > > > > > > > > > managed,
>>> > > > > > > > > > > > > > and
>>> > > > > > > > > > > > > > > > > > hardened
>>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for
>>> Airflow (
>>> > > > > > > > > > > > > > > > > > >
>>> https://github.com/astronomer/airflow-chart)
>>> > > > > that is
>>> > > > > > > > > > being
>>> > > > > > > > > > > > > used
>>> > > > > > > > > > > > > > by
>>> > > > > > > > > > > > > > > > > both
>>> > > > > > > > > > > > > > > > > > our
>>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This
>>> chart is
>>> > > > > > > > > > battle-tested and
>>> > > > > > > > > > > > > > > running
>>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of
>>> varying
>>> > > > > sizes and
>>> > > > > > > > > > runtime
>>> > > > > > > > > > > > > > > > > > environments.
>>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate
>>> the issues
>>> > > > > that Airflow
>>> > > > > > > > > > > > > users
>>> > > > > > > > > > > > > > run
>>> > > > > > > > > > > > > > > > > into
>>> > > > > > > > > > > > > > > > > > in
>>> > > > > > > > > > > > > > > > > > > the real world.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > While this chart was originally
>>> developed
>>> > > > > internally for
>>> > > > > > > > > > our
>>> > > > > > > > > > > > > > > Astronomer
>>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled
>>> the chart
>>> > > > > from the
>>> > > > > > > > > > rest of
>>> > > > > > > > > > > > > our
>>> > > > > > > > > > > > > > > > > > platform
>>> > > > > > > > > > > > > > > > > > > to make it usable by the greater
>>> Airflow
>>> > > > > community. With
>>> > > > > > > > > > these
>>> > > > > > > > > > > > > > > changes
>>> > > > > > > > > > > > > > > > > in
>>> > > > > > > > > > > > > > > > > > > mind, we want to start a
>>> conversation about
>>> > > > > donating this
>>> > > > > > > > > > > > > chart
>>> > > > > > > > > > > > > > to
>>> > > > > > > > > > > > > > > the
>>> > > > > > > > > > > > > > > > > > > Airflow community.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > Some of the main features of the
>>> chart are:
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > - It works out of the box. With zero
>>> > > > > configuration, a user
>>> > > > > > > > > > > > > will
>>> > > > > > > > > > > > > > get
>>> > > > > > > > > > > > > > > > > a
>>> > > > > > > > > > > > > > > > > > > postgres database, a default user
>>> and the
>>> > > > > > > > > > KubernetesExecutor
>>> > > > > > > > > > > > > > ready
>>> > > > > > > > > > > > > > > > > to
>>> > > > > > > > > > > > > > > > > > run
>>> > > > > > > > > > > > > > > > > > > DAGs.
>>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>>> optional KEDA
>>> > > > > > > > > > autoscaling) and
>>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We
>>> use this to
>>> > > > > share a
>>> > > > > > > > > > > > > > configurable
>>> > > > > > > > > > > > > > > > > > > connection pool size per deployment.
>>> Useful
>>> > > > > for limiting
>>> > > > > > > > > > > > > > > connections to
>>> > > > > > > > > > > > > > > > > > the
>>> > > > > > > > > > > > > > > > > > > metadata database.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A user
>>> can push a
>>> > > > > newer
>>> > > > > > > > > > version
>>> > > > > > > > > > > > > of
>>> > > > > > > > > > > > > > > > > > Airflow
>>> > > > > > > > > > > > > > > > > > > into an existing release and
>>> migrations will
>>> > > > > > > > > > automatically run
>>> > > > > > > > > > > > > > > > > > cleanly.
>>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
>>> install and
>>> > > > > configure a
>>> > > > > > > > > > > > > > > > > > statsd-exporter
>>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose
>>> them to
>>> > > > > Prometheus
>>> > > > > > > > > > > > > > > > > automatically.
>>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>>> control the
>>> > > > > ResourceQuotas
>>> > > > > > > > > > and
>>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment so
>>> that no
>>> > > > > deployment can
>>> > > > > > > > > > > > > > overload
>>> > > > > > > > > > > > > > > a
>>> > > > > > > > > > > > > > > > > > > cluster.
>>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>>> support.
>>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>>> Sometimes
>>> > > > > > > > > > KubernetesExecutor and
>>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for
>>> reasons
>>> > > > > other than the
>>> > > > > > > > > > > > > actual
>>> > > > > > > > > > > > > > > > > > task.
>>> > > > > > > > > > > > > > > > > > > This feature helps keep things clean
>>> in
>>> > > > > Kubernetes.
>>> > > > > > > > > > > > > > > > > > > - Support for running locally in KIND
>>> > > > > (Kubernetes in
>>> > > > > > > > > > Docker).
>>> > > > > > > > > > > > > > > > > > > - Automatically tested across many
>>> Kubernetes
>>> > > > > versions
>>> > > > > > > > > > with
>>> > > > > > > > > > > > > Helm
>>> > > > > > > > > > > > > > 2
>>> > > > > > > > > > > > > > > > > > and 3
>>> > > > > > > > > > > > > > > > > > > support.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and
>>> most
>>> > > > > reliable way to
>>> > > > > > > > > > deploy
>>> > > > > > > > > > > > > > DAGs
>>> > > > > > > > > > > > > > > to
>>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale
>>> is to
>>> > > > > package them
>>> > > > > > > > > > into
>>> > > > > > > > > > > > > the
>>> > > > > > > > > > > > > > > actual
>>> > > > > > > > > > > > > > > > > > > docker image, so we have geared this
>>> chart
>>> > > > > towards that
>>> > > > > > > > > > > > > method of
>>> > > > > > > > > > > > > > > > > > > operation, though adding other
>>> methods should
>>> > > > > be
>>> > > > > > > > > > > > > straightforward.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>>> community and
>>> > > > > would love
>>> > > > > > > > > > to
>>> > > > > > > > > > > > > see
>>> > > > > > > > > > > > > > > this
>>> > > > > > > > > > > > > > > > > > chart
>>> > > > > > > > > > > > > > > > > > > help others to get up and running on
>>> > > > > Kubernetes!
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > --
>>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>> > > > > Astronomer.io
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > --
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > Jarek Potiuk
>>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>>> Principal
>>> > > > > Software
>>> > > > > > > > > > Engineer
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>> > > > > <+48660796129
>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>> > > > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/
>>> >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > --
>>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>>> Astronomer.io
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > --
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Jarek Potiuk
>>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
>>> Software
>>> > > > > Engineer
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>>> <+48660796129
>>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > --
>>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>> > > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > --
>>> > > > > > > > > >
>>> > > > > > > > > > Jarek Potiuk
>>> > > > > > > > > > Polidea | Principal Software Engineer
>>> > > > > > > > > >
>>> > > > > > > > > > M: +48 660 796 129
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > --
>>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>>> > > > > >
>>> > > > > >
>>> > > > > > --
>>> > > > > > Jarek Potiuk
>>> > > > > > Polidea | Principal Software Engineer
>>> > > > > > M: +48 660 796 129
>>> > > > > >
>>> > > > >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Jarek Potiuk
>>> > Polidea | Principal Software Engineer
>>> >
>>> > M: +48 660 796 129
>>>
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48660796129>
>> [image: Polidea] <https://www.polidea.com/>
>>
>>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Hello Daniel and Greg.
I think the production image for Airflow is quite ready for the final
review and merge. I did some tests and I think it's quite complete now.
The PR is running it's Travis build:
https://github.com/apache/airflow/pull/7832
I already pushed the image to DockerHub - so that the next time it is
merged It will use it as a base and will be rather quickly built using
those images as a cache. Once it is merged, I will add the automated build
in DockerHub and merge it to 1.10. As the next step I also plan to use the
prod image for our Kubernetes tests (that will help with moving to the
GitHub Actions from Travis).
I added quite a comprehensive documentation - explaining all ins- and outs-
of both CI and PROD images
https://github.com/PolideaInternal/airflow/blob/add-production-docker-image/IMAGES.rst
.
You can build it yourself using manual build process described in the doc
or (easier) using Breeze. That includes using the same Dockerfile to
build older Airflow version from the 1.10.* line.
Feel free to use the new image in the Helm Chart you have - happy to review
the PRs. The image is available at 'apache/airflow:master-python3.6' and
'apache/airflow:master-python3.7` - once we merge it to 1.10 and release
1.10.10 we will also have it automatically published for 1.10.10 and beyond.
Happy to hear your review comments and hope to merge it very soon.
J.
On Thu, Mar 26, 2020 at 4:38 PM Jarek Potiuk <Ja...@polidea.com>
wrote:
> Absolutely! Please do :). The more eyes on this the better!
>
> On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <da...@gmail.com>
> wrote:
>
>> @Jarek Steven and I will look into getting KinD to work on GitHub actions
>> today. If we can get that working, we can donate the helm chart and
>> possibly even deprecate the old Kubernetes tests in the process.
>> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
>> wrote:
>> > We have A POC for Github Actions done by Tomek, and the blocker there
>> > was precisely Kubernetes tests and specifically running kind cluster
>> > in the Github Actions
>> > https://github.com/PolideaInternal/airflow/pull/542 . In the meantime
>> > Jiaje Zhong is trying to make it works on GA we discussed this very
>> > thing today at slack:
>> > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>> >
>> > For me this is the next thing I will look at after merging
>> > requirements PR and Prod image PR (and those two are a bit
>> > pre-requisites to make it stable and fast). I want to switch to the
>> > Production Image to run Kubernetes tests (they will be much faster)
>> > and then we can work out much faster how to run them on Github
>> > Actions. Then we can add Helm charts to run all those different
>> > deployments. I am really looking forward to it!
>> >
>> > I hope - this weekend I move forward all of that and have a PR that we
>> > can start running tests on and replacing the resources with helm
>> > chart.
>> >
>> > J.
>> >
>> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
>> > <da...@gmail.com> wrote:
>> > >
>> > > @Steven glad to help you out on that.
>> > >
>> > > For the moment we only use travis, but I believe @jarek has been
>> looking to move to GitHub CI/CD. Perhaps we can use this as our first
>> GitHub tests?
>> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
>> <st...@astronomer.io.invalid>, wrote:
>> > > > Hey I’m happy to PR in the chart and get the CI working to test it.
>> Where
>> > > > do we want to run it? Do you all just use Travis only?
>> > > >
>> > > > Steven
>> > > >
>> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
>> daniel.imberman@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > @Jarek I think with the helm chart + prod image we can go even
>> further
>> > > > > than that :). We can test CeleryExecutor, with KEDA autoscaling,
>> and a
>> > > > > bunch of other configurations.
>> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>,
>> > > > > wrote:
>> > > > > > Yeah. I meant Custom Resources not CRDs in my original email :)
>> > > > > >
>> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
>> > > > > daniel.imberman@gmail.com> wrote:
>> > > > > > > > We’re not using CRDs for the tests at the moment. We just
>> have
>> > > > > deployment files. If anything having the helm chart as a part of
>> the
>> > > > > airflow repo could mean that the helm chart becomes the defacto
>> system for
>> > > > > testing airflow on kubernetes (we can get rid of all the yams
>> files and run
>> > > > > multiple k8s tests with different settings).
>> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
>> > > > > <gr...@astronomer.io.invalid>, wrote:
>> > > > > > > > > Yep, we can absolutely pull it into the airflow repo.
>> We've also
>> > > > > been
>> > > > > > > > > building up a test suite that currently runs on CircleCI
>> and uses
>> > > > > kind
>> > > > > > > > > (Kubernetes in Docker) to test several kubernetes
>> versions with
>> > > > > some
>> > > > > > > > > different settings. Right now we're mostly testing the
>> different
>> > > > > executors
>> > > > > > > > > since that has the biggest impact on what gets deployed,
>> but that
>> > > > > can be
>> > > > > > > > > expanded.
>> > > > > > > > >
>> > > > > > > > > What CRDs are currently being used to run Airflow for the
>> tests?
>> > > > > > > > >
>> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
>> > > > > Jarek.Potiuk@polidea.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > One thing for the donation.
>> > > > > > > > > >
>> > > > > > > > > > Did you you want to have separate repository Greg ?
>> > > > > > > > > >
>> > > > > > > > > > I think we should simply create a folder in Airflow
>> repo and
>> > > > > keep it
>> > > > > > > > > > there (similarly as we keep Dockerfile). I am going to
>> switch our
>> > > > > > > > > > Kubernetes Tests to the production image (will make the
>> tests
>> > > > > much
>> > > > > > > > > > faster) and I am going to test the Dockerfile
>> automatically in
>> > > > > CI -
>> > > > > > > > > > for now we are using some custom Resource definitions
>> to start
>> > > > > Airflow
>> > > > > > > > > > on Kubernetes Cluster for the tests, but we could
>> switch to
>> > > > > using the
>> > > > > > > > > > helm chart - this way we can test all three things at
>> once:
>> > > > > > > > > > - Kubernetes Executor
>> > > > > > > > > > - Dockerfile
>> > > > > > > > > > - Helm Chart
>> > > > > > > > > > and we could also add more tests - for example testing
>> different
>> > > > > > > > > > deployment options for the helm chart.
>> > > > > > > > > >
>> > > > > > > > > > Having the Helm chart in Airflow repo would help with
>> that -
>> > > > > > > > > > especially in terms of future changes and testing them
>> > > > > automatically.
>> > > > > > > > > >
>> > > > > > > > > > J.
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
>> > > > > > > > > > <ai...@apache.org> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > +1 on the donation. Always happy to see more useful
>> stuff for
>> > > > > the
>> > > > > > > > > > > community :)
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
>> > > > > greg@astronomer.io>
>> > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as an
>> optional
>> > > > > Kubernetes
>> > > > > > > > > > > > CronJob (
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > >
>> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
>> > > > > > > > > > )
>> > > > > > > > > > > > that we have run periodically to clean failed pods
>> up and
>> > > > > could stay
>> > > > > > > > > > > > separate.
>> > > > > > > > > > > >
>> > > > > > > > > > > > The wait_for_migrations script could definitely be
>> pulled
>> > > > > into Airflow.
>> > > > > > > > > > > > For context, we deploy an initContainer on the
>> scheduler (
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > >
>> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
>> > > > > > > > > > )
>> > > > > > > > > > > > that runs the upgradedb command before booting the
>> > > > > scheduler. This new
>> > > > > > > > > > > > wait_for_migration script runs in an initContainer
>> on the
>> > > > > webserver and
>> > > > > > > > > > > > workers (
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > >
>> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
>> > > > > > > > > > )
>> > > > > > > > > > > > so that they don't boot up ahead of a potentially
>> > > > > long-running
>> > > > > > > > > > migration
>> > > > > > > > > > > > and attempt to operate on new or missing
>> columns/tables
>> > > > > before the
>> > > > > > > > > > > > migrations run. This prevents these pods from
>> entering a
>> > > > > CrashLoop.
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>> > > > > > > > > > Jarek.Potiuk@polidea.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > @Tomasz great question. Our images are currently
>> > > > > generated from
>> > > > > > > > > > > > > Dockerfiles
>> > > > > > > > > > > > > > in this repo
>> https://github.com/astronomer/ap-airflow
>> > > > > and get
>> > > > > > > > > > > > > published to
>> > > > > > > > > > > > > > DockerHub
>> > > > > > > > > > > > > >
>> > > > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow
>> .
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > For the most part those are typical Airflow
>> images.
>> > > > > There's an
>> > > > > > > > > > > > > entrypoint
>> > > > > > > > > > > > > > script that we include in the image that
>> handles waiting
>> > > > > for the
>> > > > > > > > > > > > > database
>> > > > > > > > > > > > > > and redis (if used) to come up, which is pretty
>> generic.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I already added waiting for the database (both
>> metadata
>> > > > > and celery
>> > > > > > > > > > URL) in
>> > > > > > > > > > > > > the PR:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > >
>> > > > >
>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>> > > > > > > > > > > > > .
>> > > > > > > > > > > > > It's functionally the same but more generic.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The only other
>> > > > > > > > > > > > > > thing that I think the Helm Chart uses would be
>> the
>> > > > > scripts in this
>> > > > > > > > > > repo
>> > > > > > > > > > > > > >
>> https://github.com/astronomer/astronomer-airflow-scripts.
>> > > > > Our
>> > > > > > > > > > > > > Dockerfiles
>> > > > > > > > > > > > > > pull this package in. These scripts are used to
>> > > > > coordinate running
>> > > > > > > > > > > > > > migrations and cleaning up failed pods.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I see two scripts:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed
>> to run in
>> > > > > airflow -
>> > > > > > > > > > this
>> > > > > > > > > > > > > could be run as a separate pod/container?
>> > > > > > > > > > > > > * waiting for migrations -> I think this is a good
>> > > > > candidate to add
>> > > > > > > > > > > > > *airflow
>> > > > > > > > > > > > > db wait_for_migration* command and make it part
>> of airflow
>> > > > > itself.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > I think we also have to agree on the Airflow
>> version
>> > > > > supported by the
>> > > > > > > > > > > > > official helm chart. I'd suggest we support
>> 1.10.10+ and we
>> > > > > > > > > > incorporate
>> > > > > > > > > > > > > all
>> > > > > > > > > > > > > the changes needed to airflow (like the "db
>> > > > > wait_for_migration")
>> > > > > > > > > > into 2.0
>> > > > > > > > > > > > > and 1.10 and we support both - image and helm
>> chart for
>> > > > > those versions
>> > > > > > > > > > > > > only. That would help with people migrating to
>> the latest
>> > > > > version.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > WDYT?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel
>> Imberman <
>> > > > > > > > > > > > > > daniel.imberman@gmail.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > @jarek I agree completely. I think that
>> pairing an
>> > > > > official helm
>> > > > > > > > > > chart
>> > > > > > > > > > > > > > > with the official image would make for a
>> REALLY
>> > > > > powerful “up and
>> > > > > > > > > > > > > running
>> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have also
>> been
>> > > > > looking into
>> > > > > > > > > > > > > > > operator-sdk which has the ability to create
>> custom
>> > > > > controllers
>> > > > > > > > > > from
>> > > > > > > > > > > > > helm
>> > > > > > > > > > > > > > > charts. We might even able to get a 1-2 punch
>> from the
>> > > > > same code
>> > > > > > > > > > base
>> > > > > > > > > > > > > :).
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no
>> issues, can
>> > > > > we please
>> > > > > > > > > > start
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > process of donation?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > +1 on my part, of course :)
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Daniel
>> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
>> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > +1. And it should be paired with the
>> official image
>> > > > > we have
>> > > > > > > > > > work in
>> > > > > > > > > > > > > > > > progress on. I looked a lot at the
>> Astronomer's
>> > > > > image while
>> > > > > > > > > > > > > preparing
>> > > > > > > > > > > > > > my
>> > > > > > > > > > > > > > > > draft and we can make any adjustments
>> needed to make
>> > > > > it works
>> > > > > > > > > > with
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > helm
>> > > > > > > > > > > > > > > > chart - and I am super happy to collaborate
>> on that.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > PR here:
>> https://github.com/apache/airflow/pull/7832
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > J.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
>> > > > > kaxilnaik@gmail.com
>> > > > > > > > > > >
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
>> tomasz.urbaszek@polidea.com> :
>> > > > > > > > > > > > > > > > > Helm Chart Link:
>> > > > > https://github.com/astronomer/airflow-chart
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz
>> Urbaszek <
>> > > > > > > > > > > > > > turbaszek@apache.org>
>> > > > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > An official helm chart is something our
>> > > > > community needs!
>> > > > > > > > > > Using
>> > > > > > > > > > > > > your
>> > > > > > > > > > > > > > > > > > chart as the official makes a lot of
>> sens to me
>> > > > > because as
>> > > > > > > > > > you
>> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > One question: what Airflow image do you
>> use?
>> > > > > Also, would you
>> > > > > > > > > > > > > mind
>> > > > > > > > > > > > > > > > > > sharing a link to the chart?
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > Tomek
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg
>> Neiheisel
>> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Hey everyone,
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Over the past few years at
>> Astronomer, we’ve
>> > > > > created,
>> > > > > > > > > > managed,
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > hardened
>> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for
>> Airflow (
>> > > > > > > > > > > > > > > > > > >
>> https://github.com/astronomer/airflow-chart)
>> > > > > that is
>> > > > > > > > > > being
>> > > > > > > > > > > > > used
>> > > > > > > > > > > > > > by
>> > > > > > > > > > > > > > > > > both
>> > > > > > > > > > > > > > > > > > our
>> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This
>> chart is
>> > > > > > > > > > battle-tested and
>> > > > > > > > > > > > > > > running
>> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of
>> varying
>> > > > > sizes and
>> > > > > > > > > > runtime
>> > > > > > > > > > > > > > > > > > environments.
>> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate the
>> issues
>> > > > > that Airflow
>> > > > > > > > > > > > > users
>> > > > > > > > > > > > > > run
>> > > > > > > > > > > > > > > > > into
>> > > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > the real world.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > While this chart was originally
>> developed
>> > > > > internally for
>> > > > > > > > > > our
>> > > > > > > > > > > > > > > Astronomer
>> > > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled
>> the chart
>> > > > > from the
>> > > > > > > > > > rest of
>> > > > > > > > > > > > > our
>> > > > > > > > > > > > > > > > > > platform
>> > > > > > > > > > > > > > > > > > > to make it usable by the greater
>> Airflow
>> > > > > community. With
>> > > > > > > > > > these
>> > > > > > > > > > > > > > > changes
>> > > > > > > > > > > > > > > > > in
>> > > > > > > > > > > > > > > > > > > mind, we want to start a conversation
>> about
>> > > > > donating this
>> > > > > > > > > > > > > chart
>> > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > Airflow community.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Some of the main features of the
>> chart are:
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > - It works out of the box. With zero
>> > > > > configuration, a user
>> > > > > > > > > > > > > will
>> > > > > > > > > > > > > > get
>> > > > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > postgres database, a default user and
>> the
>> > > > > > > > > > KubernetesExecutor
>> > > > > > > > > > > > > > ready
>> > > > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > run
>> > > > > > > > > > > > > > > > > > > DAGs.
>> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
>> optional KEDA
>> > > > > > > > > > autoscaling) and
>> > > > > > > > > > > > > > > > > > > Kubernetes executors.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We
>> use this to
>> > > > > share a
>> > > > > > > > > > > > > > configurable
>> > > > > > > > > > > > > > > > > > > connection pool size per deployment.
>> Useful
>> > > > > for limiting
>> > > > > > > > > > > > > > > connections to
>> > > > > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > > > > > metadata database.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > - Airflow migration support. A user
>> can push a
>> > > > > newer
>> > > > > > > > > > version
>> > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > > > > Airflow
>> > > > > > > > > > > > > > > > > > > into an existing release and
>> migrations will
>> > > > > > > > > > automatically run
>> > > > > > > > > > > > > > > > > > cleanly.
>> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
>> install and
>> > > > > configure a
>> > > > > > > > > > > > > > > > > > statsd-exporter
>> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose
>> them to
>> > > > > Prometheus
>> > > > > > > > > > > > > > > > > automatically.
>> > > > > > > > > > > > > > > > > > > - Resource control. Optionally
>> control the
>> > > > > ResourceQuotas
>> > > > > > > > > > and
>> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment so
>> that no
>> > > > > deployment can
>> > > > > > > > > > > > > > overload
>> > > > > > > > > > > > > > > a
>> > > > > > > > > > > > > > > > > > > cluster.
>> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
>> support.
>> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup.
>> Sometimes
>> > > > > > > > > > KubernetesExecutor and
>> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for
>> reasons
>> > > > > other than the
>> > > > > > > > > > > > > actual
>> > > > > > > > > > > > > > > > > > task.
>> > > > > > > > > > > > > > > > > > > This feature helps keep things clean
>> in
>> > > > > Kubernetes.
>> > > > > > > > > > > > > > > > > > > - Support for running locally in KIND
>> > > > > (Kubernetes in
>> > > > > > > > > > Docker).
>> > > > > > > > > > > > > > > > > > > - Automatically tested across many
>> Kubernetes
>> > > > > versions
>> > > > > > > > > > with
>> > > > > > > > > > > > > Helm
>> > > > > > > > > > > > > > 2
>> > > > > > > > > > > > > > > > > > and 3
>> > > > > > > > > > > > > > > > > > > support.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and most
>> > > > > reliable way to
>> > > > > > > > > > deploy
>> > > > > > > > > > > > > > DAGs
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale
>> is to
>> > > > > package them
>> > > > > > > > > > into
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > actual
>> > > > > > > > > > > > > > > > > > > docker image, so we have geared this
>> chart
>> > > > > towards that
>> > > > > > > > > > > > > method of
>> > > > > > > > > > > > > > > > > > > operation, though adding other
>> methods should
>> > > > > be
>> > > > > > > > > > > > > straightforward.
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > We would love thoughts from the
>> community and
>> > > > > would love
>> > > > > > > > > > to
>> > > > > > > > > > > > > see
>> > > > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > > > > chart
>> > > > > > > > > > > > > > > > > > > help others to get up and running on
>> > > > > Kubernetes!
>> > > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
>> > > > > Astronomer.io
>> > > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Jarek Potiuk
>> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
>> Principal
>> > > > > Software
>> > > > > > > > > > Engineer
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>> > > > > <+48660796129
>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>> > > > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > --
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Jarek Potiuk
>> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
>> Software
>> > > > > Engineer
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
>> <+48660796129
>> > > > > > > > > > > > > <+48%20660%20796%20129>>
>> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > --
>> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>> > > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > >
>> > > > > > > > > > Jarek Potiuk
>> > > > > > > > > > Polidea | Principal Software Engineer
>> > > > > > > > > >
>> > > > > > > > > > M: +48 660 796 129
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Jarek Potiuk
>> > > > > > Polidea | Principal Software Engineer
>> > > > > > M: +48 660 796 129
>> > > > > >
>> > > > >
>> >
>> >
>> >
>> > --
>> >
>> > Jarek Potiuk
>> > Polidea | Principal Software Engineer
>> >
>> > M: +48 660 796 129
>>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Absolutely! Please do :). The more eyes on this the better!
On Thu, Mar 26, 2020 at 4:32 PM Daniel Imberman <da...@gmail.com>
wrote:
> @Jarek Steven and I will look into getting KinD to work on GitHub actions
> today. If we can get that working, we can donate the helm chart and
> possibly even deprecate the old Kubernetes tests in the process.
> On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> wrote:
> > We have A POC for Github Actions done by Tomek, and the blocker there
> > was precisely Kubernetes tests and specifically running kind cluster
> > in the Github Actions
> > https://github.com/PolideaInternal/airflow/pull/542 . In the meantime
> > Jiaje Zhong is trying to make it works on GA we discussed this very
> > thing today at slack:
> > https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
> >
> > For me this is the next thing I will look at after merging
> > requirements PR and Prod image PR (and those two are a bit
> > pre-requisites to make it stable and fast). I want to switch to the
> > Production Image to run Kubernetes tests (they will be much faster)
> > and then we can work out much faster how to run them on Github
> > Actions. Then we can add Helm charts to run all those different
> > deployments. I am really looking forward to it!
> >
> > I hope - this weekend I move forward all of that and have a PR that we
> > can start running tests on and replacing the resources with helm
> > chart.
> >
> > J.
> >
> > On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> > <da...@gmail.com> wrote:
> > >
> > > @Steven glad to help you out on that.
> > >
> > > For the moment we only use travis, but I believe @jarek has been
> looking to move to GitHub CI/CD. Perhaps we can use this as our first
> GitHub tests?
> > > On Mar 26, 2020, 8:17 AM -0700, Steven Miller
> <st...@astronomer.io.invalid>, wrote:
> > > > Hey I’m happy to PR in the chart and get the CI working to test it.
> Where
> > > > do we want to run it? Do you all just use Travis only?
> > > >
> > > > Steven
> > > >
> > > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <
> daniel.imberman@gmail.com>
> > > > wrote:
> > > >
> > > > > @Jarek I think with the helm chart + prod image we can go even
> further
> > > > > than that :). We can test CeleryExecutor, with KEDA autoscaling,
> and a
> > > > > bunch of other configurations.
> > > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <
> Jarek.Potiuk@polidea.com>,
> > > > > wrote:
> > > > > > Yeah. I meant Custom Resources not CRDs in my original email :)
> > > > > >
> > > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > > > > daniel.imberman@gmail.com> wrote:
> > > > > > > > We’re not using CRDs for the tests at the moment. We just
> have
> > > > > deployment files. If anything having the helm chart as a part of
> the
> > > > > airflow repo could mean that the helm chart becomes the defacto
> system for
> > > > > testing airflow on kubernetes (we can get rid of all the yams
> files and run
> > > > > multiple k8s tests with different settings).
> > > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > > > > <gr...@astronomer.io.invalid>, wrote:
> > > > > > > > > Yep, we can absolutely pull it into the airflow repo.
> We've also
> > > > > been
> > > > > > > > > building up a test suite that currently runs on CircleCI
> and uses
> > > > > kind
> > > > > > > > > (Kubernetes in Docker) to test several kubernetes versions
> with
> > > > > some
> > > > > > > > > different settings. Right now we're mostly testing the
> different
> > > > > executors
> > > > > > > > > since that has the biggest impact on what gets deployed,
> but that
> > > > > can be
> > > > > > > > > expanded.
> > > > > > > > >
> > > > > > > > > What CRDs are currently being used to run Airflow for the
> tests?
> > > > > > > > >
> > > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > One thing for the donation.
> > > > > > > > > >
> > > > > > > > > > Did you you want to have separate repository Greg ?
> > > > > > > > > >
> > > > > > > > > > I think we should simply create a folder in Airflow repo
> and
> > > > > keep it
> > > > > > > > > > there (similarly as we keep Dockerfile). I am going to
> switch our
> > > > > > > > > > Kubernetes Tests to the production image (will make the
> tests
> > > > > much
> > > > > > > > > > faster) and I am going to test the Dockerfile
> automatically in
> > > > > CI -
> > > > > > > > > > for now we are using some custom Resource definitions to
> start
> > > > > Airflow
> > > > > > > > > > on Kubernetes Cluster for the tests, but we could switch
> to
> > > > > using the
> > > > > > > > > > helm chart - this way we can test all three things at
> once:
> > > > > > > > > > - Kubernetes Executor
> > > > > > > > > > - Dockerfile
> > > > > > > > > > - Helm Chart
> > > > > > > > > > and we could also add more tests - for example testing
> different
> > > > > > > > > > deployment options for the helm chart.
> > > > > > > > > >
> > > > > > > > > > Having the Helm chart in Airflow repo would help with
> that -
> > > > > > > > > > especially in terms of future changes and testing them
> > > > > automatically.
> > > > > > > > > >
> > > > > > > > > > J.
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > > > > > > <ai...@apache.org> wrote:
> > > > > > > > > > >
> > > > > > > > > > > +1 on the donation. Always happy to see more useful
> stuff for
> > > > > the
> > > > > > > > > > > community :)
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
> > > > > greg@astronomer.io>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Yep, the cleanup_pods script is set up now as an
> optional
> > > > > Kubernetes
> > > > > > > > > > > > CronJob (
> > > > > > > > > > > >
> > > > > > > > > >
> > > > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > > > > > > > > )
> > > > > > > > > > > > that we have run periodically to clean failed pods
> up and
> > > > > could stay
> > > > > > > > > > > > separate.
> > > > > > > > > > > >
> > > > > > > > > > > > The wait_for_migrations script could definitely be
> pulled
> > > > > into Airflow.
> > > > > > > > > > > > For context, we deploy an initContainer on the
> scheduler (
> > > > > > > > > > > >
> > > > > > > > > >
> > > > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > > > > > > > > )
> > > > > > > > > > > > that runs the upgradedb command before booting the
> > > > > scheduler. This new
> > > > > > > > > > > > wait_for_migration script runs in an initContainer
> on the
> > > > > webserver and
> > > > > > > > > > > > workers (
> > > > > > > > > > > >
> > > > > > > > > >
> > > > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > > > > > > > > )
> > > > > > > > > > > > so that they don't boot up ahead of a potentially
> > > > > long-running
> > > > > > > > > > migration
> > > > > > > > > > > > and attempt to operate on new or missing
> columns/tables
> > > > > before the
> > > > > > > > > > > > migrations run. This prevents these pods from
> entering a
> > > > > CrashLoop.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > > > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Tomasz great question. Our images are currently
> > > > > generated from
> > > > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > > > in this repo
> https://github.com/astronomer/ap-airflow
> > > > > and get
> > > > > > > > > > > > > published to
> > > > > > > > > > > > > > DockerHub
> > > > > > > > > > > > > >
> > > > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For the most part those are typical Airflow
> images.
> > > > > There's an
> > > > > > > > > > > > > entrypoint
> > > > > > > > > > > > > > script that we include in the image that handles
> waiting
> > > > > for the
> > > > > > > > > > > > > database
> > > > > > > > > > > > > > and redis (if used) to come up, which is pretty
> generic.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I already added waiting for the database (both
> metadata
> > > > > and celery
> > > > > > > > > > URL) in
> > > > > > > > > > > > > the PR:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > >
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > > > > > > > > > .
> > > > > > > > > > > > > It's functionally the same but more generic.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The only other
> > > > > > > > > > > > > > thing that I think the Helm Chart uses would be
> the
> > > > > scripts in this
> > > > > > > > > > repo
> > > > > > > > > > > > > >
> https://github.com/astronomer/astronomer-airflow-scripts.
> > > > > Our
> > > > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > > > pull this package in. These scripts are used to
> > > > > coordinate running
> > > > > > > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I see two scripts:
> > > > > > > > > > > > >
> > > > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed
> to run in
> > > > > airflow -
> > > > > > > > > > this
> > > > > > > > > > > > > could be run as a separate pod/container?
> > > > > > > > > > > > > * waiting for migrations -> I think this is a good
> > > > > candidate to add
> > > > > > > > > > > > > *airflow
> > > > > > > > > > > > > db wait_for_migration* command and make it part of
> airflow
> > > > > itself.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think we also have to agree on the Airflow
> version
> > > > > supported by the
> > > > > > > > > > > > > official helm chart. I'd suggest we support
> 1.10.10+ and we
> > > > > > > > > > incorporate
> > > > > > > > > > > > > all
> > > > > > > > > > > > > the changes needed to airflow (like the "db
> > > > > wait_for_migration")
> > > > > > > > > > into 2.0
> > > > > > > > > > > > > and 1.10 and we support both - image and helm
> chart for
> > > > > those versions
> > > > > > > > > > > > > only. That would help with people migrating to the
> latest
> > > > > version.
> > > > > > > > > > > > >
> > > > > > > > > > > > > WDYT?
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman
> <
> > > > > > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @jarek I agree completely. I think that
> pairing an
> > > > > official helm
> > > > > > > > > > chart
> > > > > > > > > > > > > > > with the official image would make for a REALLY
> > > > > powerful “up and
> > > > > > > > > > > > > running
> > > > > > > > > > > > > > > with airflow” story :). Tomek and I have also
> been
> > > > > looking into
> > > > > > > > > > > > > > > operator-sdk which has the ability to create
> custom
> > > > > controllers
> > > > > > > > > > from
> > > > > > > > > > > > > helm
> > > > > > > > > > > > > > > charts. We might even able to get a 1-2 punch
> from the
> > > > > same code
> > > > > > > > > > base
> > > > > > > > > > > > > :).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no
> issues, can
> > > > > we please
> > > > > > > > > > start
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > process of donation?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Daniel
> > > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > +1. And it should be paired with the
> official image
> > > > > we have
> > > > > > > > > > work in
> > > > > > > > > > > > > > > > progress on. I looked a lot at the
> Astronomer's
> > > > > image while
> > > > > > > > > > > > > preparing
> > > > > > > > > > > > > > my
> > > > > > > > > > > > > > > > draft and we can make any adjustments needed
> to make
> > > > > it works
> > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > helm
> > > > > > > > > > > > > > > > chart - and I am super happy to collaborate
> on that.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > PR here:
> https://github.com/apache/airflow/pull/7832
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > J.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
> > > > > kaxilnaik@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Tomasz Urbaszek <
> tomasz.urbaszek@polidea.com> :
> > > > > > > > > > > > > > > > > Helm Chart Link:
> > > > > https://github.com/astronomer/airflow-chart
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz
> Urbaszek <
> > > > > > > > > > > > > > turbaszek@apache.org>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > An official helm chart is something our
> > > > > community needs!
> > > > > > > > > > Using
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > > chart as the official makes a lot of
> sens to me
> > > > > because as
> > > > > > > > > > you
> > > > > > > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > One question: what Airflow image do you
> use?
> > > > > Also, would you
> > > > > > > > > > > > > mind
> > > > > > > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Tomek
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg
> Neiheisel
> > > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Over the past few years at Astronomer,
> we’ve
> > > > > created,
> > > > > > > > > > managed,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > > > > > > a production-ready Helm Chart for
> Airflow (
> > > > > > > > > > > > > > > > > > >
> https://github.com/astronomer/airflow-chart)
> > > > > that is
> > > > > > > > > > being
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This
> chart is
> > > > > > > > > > battle-tested and
> > > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of
> varying
> > > > > sizes and
> > > > > > > > > > runtime
> > > > > > > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > > > > > > It’s been built up to encapsulate the
> issues
> > > > > that Airflow
> > > > > > > > > > > > > users
> > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > While this chart was originally
> developed
> > > > > internally for
> > > > > > > > > > our
> > > > > > > > > > > > > > > Astronomer
> > > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled the
> chart
> > > > > from the
> > > > > > > > > > rest of
> > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > platform
> > > > > > > > > > > > > > > > > > > to make it usable by the greater
> Airflow
> > > > > community. With
> > > > > > > > > > these
> > > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > > mind, we want to start a conversation
> about
> > > > > donating this
> > > > > > > > > > > > > chart
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Some of the main features of the chart
> are:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - It works out of the box. With zero
> > > > > configuration, a user
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > postgres database, a default user and
> the
> > > > > > > > > > KubernetesExecutor
> > > > > > > > > > > > > > ready
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/
> optional KEDA
> > > > > > > > > > autoscaling) and
> > > > > > > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We use
> this to
> > > > > share a
> > > > > > > > > > > > > > configurable
> > > > > > > > > > > > > > > > > > > connection pool size per deployment.
> Useful
> > > > > for limiting
> > > > > > > > > > > > > > > connections to
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > - Airflow migration support. A user
> can push a
> > > > > newer
> > > > > > > > > > version
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > > > > > > into an existing release and
> migrations will
> > > > > > > > > > automatically run
> > > > > > > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > > > > > > - Prometheus support. Optionally
> install and
> > > > > configure a
> > > > > > > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose
> them to
> > > > > Prometheus
> > > > > > > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > > > > > > - Resource control. Optionally control
> the
> > > > > ResourceQuotas
> > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > LimitRanges for each deployment so
> that no
> > > > > deployment can
> > > > > > > > > > > > > > overload
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch
> support.
> > > > > > > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > > > > > > KubernetesExecutor and
> > > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for
> reasons
> > > > > other than the
> > > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > > task.
> > > > > > > > > > > > > > > > > > > This feature helps keep things clean in
> > > > > Kubernetes.
> > > > > > > > > > > > > > > > > > > - Support for running locally in KIND
> > > > > (Kubernetes in
> > > > > > > > > > Docker).
> > > > > > > > > > > > > > > > > > > - Automatically tested across many
> Kubernetes
> > > > > versions
> > > > > > > > > > with
> > > > > > > > > > > > > Helm
> > > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > We’ve found that the cleanest and most
> > > > > reliable way to
> > > > > > > > > > deploy
> > > > > > > > > > > > > > DAGs
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale is
> to
> > > > > package them
> > > > > > > > > > into
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > > > docker image, so we have geared this
> chart
> > > > > towards that
> > > > > > > > > > > > > method of
> > > > > > > > > > > > > > > > > > > operation, though adding other methods
> should
> > > > > be
> > > > > > > > > > > > > straightforward.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > We would love thoughts from the
> community and
> > > > > would love
> > > > > > > > > > to
> > > > > > > > > > > > > see
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > chart
> > > > > > > > > > > > > > > > > > > help others to get up and running on
> > > > > Kubernetes!
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > > > Astronomer.io
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> |
> Principal
> > > > > Software
> > > > > > > > > > Engineer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> > > > > <+48660796129
> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
> Software
> > > > > Engineer
> > > > > > > > > > > > >
> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> <+48660796129
> > > > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Jarek Potiuk
> > > > > > > > > > Polidea | Principal Software Engineer
> > > > > > > > > >
> > > > > > > > > > M: +48 660 796 129
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > > M: +48 660 796 129
> > > > > >
> > > > >
> >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Daniel Imberman <da...@gmail.com>.
@Jarek Steven and I will look into getting KinD to work on GitHub actions today. If we can get that working, we can donate the helm chart and possibly even deprecate the old Kubernetes tests in the process.
On Mar 26, 2020, 8:27 AM -0700, Jarek Potiuk <Ja...@polidea.com>, wrote:
> We have A POC for Github Actions done by Tomek, and the blocker there
> was precisely Kubernetes tests and specifically running kind cluster
> in the Github Actions
> https://github.com/PolideaInternal/airflow/pull/542 . In the meantime
> Jiaje Zhong is trying to make it works on GA we discussed this very
> thing today at slack:
> https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
>
> For me this is the next thing I will look at after merging
> requirements PR and Prod image PR (and those two are a bit
> pre-requisites to make it stable and fast). I want to switch to the
> Production Image to run Kubernetes tests (they will be much faster)
> and then we can work out much faster how to run them on Github
> Actions. Then we can add Helm charts to run all those different
> deployments. I am really looking forward to it!
>
> I hope - this weekend I move forward all of that and have a PR that we
> can start running tests on and replacing the resources with helm
> chart.
>
> J.
>
> On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
> <da...@gmail.com> wrote:
> >
> > @Steven glad to help you out on that.
> >
> > For the moment we only use travis, but I believe @jarek has been looking to move to GitHub CI/CD. Perhaps we can use this as our first GitHub tests?
> > On Mar 26, 2020, 8:17 AM -0700, Steven Miller <st...@astronomer.io.invalid>, wrote:
> > > Hey I’m happy to PR in the chart and get the CI working to test it. Where
> > > do we want to run it? Do you all just use Travis only?
> > >
> > > Steven
> > >
> > > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <da...@gmail.com>
> > > wrote:
> > >
> > > > @Jarek I think with the helm chart + prod image we can go even further
> > > > than that :). We can test CeleryExecutor, with KEDA autoscaling, and a
> > > > bunch of other configurations.
> > > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> > > > wrote:
> > > > > Yeah. I meant Custom Resources not CRDs in my original email :)
> > > > >
> > > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > > > daniel.imberman@gmail.com> wrote:
> > > > > > > We’re not using CRDs for the tests at the moment. We just have
> > > > deployment files. If anything having the helm chart as a part of the
> > > > airflow repo could mean that the helm chart becomes the defacto system for
> > > > testing airflow on kubernetes (we can get rid of all the yams files and run
> > > > multiple k8s tests with different settings).
> > > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > > > <gr...@astronomer.io.invalid>, wrote:
> > > > > > > > Yep, we can absolutely pull it into the airflow repo. We've also
> > > > been
> > > > > > > > building up a test suite that currently runs on CircleCI and uses
> > > > kind
> > > > > > > > (Kubernetes in Docker) to test several kubernetes versions with
> > > > some
> > > > > > > > different settings. Right now we're mostly testing the different
> > > > executors
> > > > > > > > since that has the biggest impact on what gets deployed, but that
> > > > can be
> > > > > > > > expanded.
> > > > > > > >
> > > > > > > > What CRDs are currently being used to run Airflow for the tests?
> > > > > > > >
> > > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > > > Jarek.Potiuk@polidea.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > One thing for the donation.
> > > > > > > > >
> > > > > > > > > Did you you want to have separate repository Greg ?
> > > > > > > > >
> > > > > > > > > I think we should simply create a folder in Airflow repo and
> > > > keep it
> > > > > > > > > there (similarly as we keep Dockerfile). I am going to switch our
> > > > > > > > > Kubernetes Tests to the production image (will make the tests
> > > > much
> > > > > > > > > faster) and I am going to test the Dockerfile automatically in
> > > > CI -
> > > > > > > > > for now we are using some custom Resource definitions to start
> > > > Airflow
> > > > > > > > > on Kubernetes Cluster for the tests, but we could switch to
> > > > using the
> > > > > > > > > helm chart - this way we can test all three things at once:
> > > > > > > > > - Kubernetes Executor
> > > > > > > > > - Dockerfile
> > > > > > > > > - Helm Chart
> > > > > > > > > and we could also add more tests - for example testing different
> > > > > > > > > deployment options for the helm chart.
> > > > > > > > >
> > > > > > > > > Having the Helm chart in Airflow repo would help with that -
> > > > > > > > > especially in terms of future changes and testing them
> > > > automatically.
> > > > > > > > >
> > > > > > > > > J.
> > > > > > > > >
> > > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > > > > > <ai...@apache.org> wrote:
> > > > > > > > > >
> > > > > > > > > > +1 on the donation. Always happy to see more useful stuff for
> > > > the
> > > > > > > > > > community :)
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
> > > > greg@astronomer.io>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Yep, the cleanup_pods script is set up now as an optional
> > > > Kubernetes
> > > > > > > > > > > CronJob (
> > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > > > > > > > )
> > > > > > > > > > > that we have run periodically to clean failed pods up and
> > > > could stay
> > > > > > > > > > > separate.
> > > > > > > > > > >
> > > > > > > > > > > The wait_for_migrations script could definitely be pulled
> > > > into Airflow.
> > > > > > > > > > > For context, we deploy an initContainer on the scheduler (
> > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > > > > > > > )
> > > > > > > > > > > that runs the upgradedb command before booting the
> > > > scheduler. This new
> > > > > > > > > > > wait_for_migration script runs in an initContainer on the
> > > > webserver and
> > > > > > > > > > > workers (
> > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > > > > > > > )
> > > > > > > > > > > so that they don't boot up ahead of a potentially
> > > > long-running
> > > > > > > > > migration
> > > > > > > > > > > and attempt to operate on new or missing columns/tables
> > > > before the
> > > > > > > > > > > migrations run. This prevents these pods from entering a
> > > > CrashLoop.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Tomasz great question. Our images are currently
> > > > generated from
> > > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > > in this repo https://github.com/astronomer/ap-airflow
> > > > and get
> > > > > > > > > > > > published to
> > > > > > > > > > > > > DockerHub
> > > > > > > > > > > > >
> > > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For the most part those are typical Airflow images.
> > > > There's an
> > > > > > > > > > > > entrypoint
> > > > > > > > > > > > > script that we include in the image that handles waiting
> > > > for the
> > > > > > > > > > > > database
> > > > > > > > > > > > > and redis (if used) to come up, which is pretty generic.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I already added waiting for the database (both metadata
> > > > and celery
> > > > > > > > > URL) in
> > > > > > > > > > > > the PR:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > > > > > > > > .
> > > > > > > > > > > > It's functionally the same but more generic.
> > > > > > > > > > > >
> > > > > > > > > > > > The only other
> > > > > > > > > > > > > thing that I think the Helm Chart uses would be the
> > > > scripts in this
> > > > > > > > > repo
> > > > > > > > > > > > > https://github.com/astronomer/astronomer-airflow-scripts.
> > > > Our
> > > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > > pull this package in. These scripts are used to
> > > > coordinate running
> > > > > > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I see two scripts:
> > > > > > > > > > > >
> > > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed to run in
> > > > airflow -
> > > > > > > > > this
> > > > > > > > > > > > could be run as a separate pod/container?
> > > > > > > > > > > > * waiting for migrations -> I think this is a good
> > > > candidate to add
> > > > > > > > > > > > *airflow
> > > > > > > > > > > > db wait_for_migration* command and make it part of airflow
> > > > itself.
> > > > > > > > > > > >
> > > > > > > > > > > > I think we also have to agree on the Airflow version
> > > > supported by the
> > > > > > > > > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > > > > > > > > incorporate
> > > > > > > > > > > > all
> > > > > > > > > > > > the changes needed to airflow (like the "db
> > > > wait_for_migration")
> > > > > > > > > into 2.0
> > > > > > > > > > > > and 1.10 and we support both - image and helm chart for
> > > > those versions
> > > > > > > > > > > > only. That would help with people migrating to the latest
> > > > version.
> > > > > > > > > > > >
> > > > > > > > > > > > WDYT?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > @jarek I agree completely. I think that pairing an
> > > > official helm
> > > > > > > > > chart
> > > > > > > > > > > > > > with the official image would make for a REALLY
> > > > powerful “up and
> > > > > > > > > > > > running
> > > > > > > > > > > > > > with airflow” story :). Tomek and I have also been
> > > > looking into
> > > > > > > > > > > > > > operator-sdk which has the ability to create custom
> > > > controllers
> > > > > > > > > from
> > > > > > > > > > > > helm
> > > > > > > > > > > > > > charts. We might even able to get a 1-2 punch from the
> > > > same code
> > > > > > > > > base
> > > > > > > > > > > > :).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can
> > > > we please
> > > > > > > > > start
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > process of donation?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Daniel
> > > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > +1. And it should be paired with the official image
> > > > we have
> > > > > > > > > work in
> > > > > > > > > > > > > > > progress on. I looked a lot at the Astronomer's
> > > > image while
> > > > > > > > > > > > preparing
> > > > > > > > > > > > > my
> > > > > > > > > > > > > > > draft and we can make any adjustments needed to make
> > > > it works
> > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > helm
> > > > > > > > > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > PR here: https://github.com/apache/airflow/pull/7832
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > J.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
> > > > kaxilnaik@gmail.com
> > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > > > > > > > > > > > > Helm Chart Link:
> > > > https://github.com/astronomer/airflow-chart
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > > > > > > > > turbaszek@apache.org>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > An official helm chart is something our
> > > > community needs!
> > > > > > > > > Using
> > > > > > > > > > > > your
> > > > > > > > > > > > > > > > > chart as the official makes a lot of sens to me
> > > > because as
> > > > > > > > > you
> > > > > > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > One question: what Airflow image do you use?
> > > > Also, would you
> > > > > > > > > > > > mind
> > > > > > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Tomek
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Over the past few years at Astronomer, we’ve
> > > > created,
> > > > > > > > > managed,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > > > > > > > > https://github.com/astronomer/airflow-chart)
> > > > that is
> > > > > > > > > being
> > > > > > > > > > > > used
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > > > > > > > > battle-tested and
> > > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > > hundreds of Airflow deployments of varying
> > > > sizes and
> > > > > > > > > runtime
> > > > > > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > > > > > It’s been built up to encapsulate the issues
> > > > that Airflow
> > > > > > > > > > > > users
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > While this chart was originally developed
> > > > internally for
> > > > > > > > > our
> > > > > > > > > > > > > > Astronomer
> > > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled the chart
> > > > from the
> > > > > > > > > rest of
> > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > platform
> > > > > > > > > > > > > > > > > > to make it usable by the greater Airflow
> > > > community. With
> > > > > > > > > these
> > > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > mind, we want to start a conversation about
> > > > donating this
> > > > > > > > > > > > chart
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - It works out of the box. With zero
> > > > configuration, a user
> > > > > > > > > > > > will
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > postgres database, a default user and the
> > > > > > > > > KubernetesExecutor
> > > > > > > > > > > > > ready
> > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > > > > > > > > autoscaling) and
> > > > > > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Support for optional pgbouncer. We use this to
> > > > share a
> > > > > > > > > > > > > configurable
> > > > > > > > > > > > > > > > > > connection pool size per deployment. Useful
> > > > for limiting
> > > > > > > > > > > > > > connections to
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > - Airflow migration support. A user can push a
> > > > newer
> > > > > > > > > version
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > > > > > into an existing release and migrations will
> > > > > > > > > automatically run
> > > > > > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > > > > > - Prometheus support. Optionally install and
> > > > configure a
> > > > > > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose them to
> > > > Prometheus
> > > > > > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > > > > > - Resource control. Optionally control the
> > > > ResourceQuotas
> > > > > > > > > and
> > > > > > > > > > > > > > > > > > LimitRanges for each deployment so that no
> > > > deployment can
> > > > > > > > > > > > > overload
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > > > > > KubernetesExecutor and
> > > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for reasons
> > > > other than the
> > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > task.
> > > > > > > > > > > > > > > > > > This feature helps keep things clean in
> > > > Kubernetes.
> > > > > > > > > > > > > > > > > > - Support for running locally in KIND
> > > > (Kubernetes in
> > > > > > > > > Docker).
> > > > > > > > > > > > > > > > > > - Automatically tested across many Kubernetes
> > > > versions
> > > > > > > > > with
> > > > > > > > > > > > Helm
> > > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We’ve found that the cleanest and most
> > > > reliable way to
> > > > > > > > > deploy
> > > > > > > > > > > > > DAGs
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > Kubernetes and manage them at scale is to
> > > > package them
> > > > > > > > > into
> > > > > > > > > > > > the
> > > > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > > docker image, so we have geared this chart
> > > > towards that
> > > > > > > > > > > > method of
> > > > > > > > > > > > > > > > > > operation, though adding other methods should
> > > > be
> > > > > > > > > > > > straightforward.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We would love thoughts from the community and
> > > > would love
> > > > > > > > > to
> > > > > > > > > > > > see
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > chart
> > > > > > > > > > > > > > > > > > help others to get up and running on
> > > > Kubernetes!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > > Astronomer.io
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
> > > > Software
> > > > > > > > > Engineer
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> > > > <+48660796129
> > > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> > > > Engineer
> > > > > > > > > > > >
> > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jarek Potiuk
> > > > > > > > > Polidea | Principal Software Engineer
> > > > > > > > >
> > > > > > > > > M: +48 660 796 129
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > >
> > > > >
> > > > > --
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > > M: +48 660 796 129
> > > > >
> > > >
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
We have A POC for Github Actions done by Tomek, and the blocker there
was precisely Kubernetes tests and specifically running kind cluster
in the Github Actions
https://github.com/PolideaInternal/airflow/pull/542 . In the meantime
Jiaje Zhong is trying to make it works on GA we discussed this very
thing today at slack:
https://apache-airflow.slack.com/archives/CCPRP7943/p1585073249374200
For me this is the next thing I will look at after merging
requirements PR and Prod image PR (and those two are a bit
pre-requisites to make it stable and fast). I want to switch to the
Production Image to run Kubernetes tests (they will be much faster)
and then we can work out much faster how to run them on Github
Actions. Then we can add Helm charts to run all those different
deployments. I am really looking forward to it!
I hope - this weekend I move forward all of that and have a PR that we
can start running tests on and replacing the resources with helm
chart.
J.
On Thu, Mar 26, 2020 at 4:20 PM Daniel Imberman
<da...@gmail.com> wrote:
>
> @Steven glad to help you out on that.
>
> For the moment we only use travis, but I believe @jarek has been looking to move to GitHub CI/CD. Perhaps we can use this as our first GitHub tests?
> On Mar 26, 2020, 8:17 AM -0700, Steven Miller <st...@astronomer.io.invalid>, wrote:
> > Hey I’m happy to PR in the chart and get the CI working to test it. Where
> > do we want to run it? Do you all just use Travis only?
> >
> > Steven
> >
> > On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <da...@gmail.com>
> > wrote:
> >
> > > @Jarek I think with the helm chart + prod image we can go even further
> > > than that :). We can test CeleryExecutor, with KEDA autoscaling, and a
> > > bunch of other configurations.
> > > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> > > wrote:
> > > > Yeah. I meant Custom Resources not CRDs in my original email :)
> > > >
> > > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > > daniel.imberman@gmail.com> wrote:
> > > > > > We’re not using CRDs for the tests at the moment. We just have
> > > deployment files. If anything having the helm chart as a part of the
> > > airflow repo could mean that the helm chart becomes the defacto system for
> > > testing airflow on kubernetes (we can get rid of all the yams files and run
> > > multiple k8s tests with different settings).
> > > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > > <gr...@astronomer.io.invalid>, wrote:
> > > > > > > Yep, we can absolutely pull it into the airflow repo. We've also
> > > been
> > > > > > > building up a test suite that currently runs on CircleCI and uses
> > > kind
> > > > > > > (Kubernetes in Docker) to test several kubernetes versions with
> > > some
> > > > > > > different settings. Right now we're mostly testing the different
> > > executors
> > > > > > > since that has the biggest impact on what gets deployed, but that
> > > can be
> > > > > > > expanded.
> > > > > > >
> > > > > > > What CRDs are currently being used to run Airflow for the tests?
> > > > > > >
> > > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > > Jarek.Potiuk@polidea.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > One thing for the donation.
> > > > > > > >
> > > > > > > > Did you you want to have separate repository Greg ?
> > > > > > > >
> > > > > > > > I think we should simply create a folder in Airflow repo and
> > > keep it
> > > > > > > > there (similarly as we keep Dockerfile). I am going to switch our
> > > > > > > > Kubernetes Tests to the production image (will make the tests
> > > much
> > > > > > > > faster) and I am going to test the Dockerfile automatically in
> > > CI -
> > > > > > > > for now we are using some custom Resource definitions to start
> > > Airflow
> > > > > > > > on Kubernetes Cluster for the tests, but we could switch to
> > > using the
> > > > > > > > helm chart - this way we can test all three things at once:
> > > > > > > > - Kubernetes Executor
> > > > > > > > - Dockerfile
> > > > > > > > - Helm Chart
> > > > > > > > and we could also add more tests - for example testing different
> > > > > > > > deployment options for the helm chart.
> > > > > > > >
> > > > > > > > Having the Helm chart in Airflow repo would help with that -
> > > > > > > > especially in terms of future changes and testing them
> > > automatically.
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > > > > <ai...@apache.org> wrote:
> > > > > > > > >
> > > > > > > > > +1 on the donation. Always happy to see more useful stuff for
> > > the
> > > > > > > > > community :)
> > > > > > > > >
> > > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
> > > greg@astronomer.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Yep, the cleanup_pods script is set up now as an optional
> > > Kubernetes
> > > > > > > > > > CronJob (
> > > > > > > > > >
> > > > > > > >
> > > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > > > > > > )
> > > > > > > > > > that we have run periodically to clean failed pods up and
> > > could stay
> > > > > > > > > > separate.
> > > > > > > > > >
> > > > > > > > > > The wait_for_migrations script could definitely be pulled
> > > into Airflow.
> > > > > > > > > > For context, we deploy an initContainer on the scheduler (
> > > > > > > > > >
> > > > > > > >
> > > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > > > > > > )
> > > > > > > > > > that runs the upgradedb command before booting the
> > > scheduler. This new
> > > > > > > > > > wait_for_migration script runs in an initContainer on the
> > > webserver and
> > > > > > > > > > workers (
> > > > > > > > > >
> > > > > > > >
> > > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > > > > > > )
> > > > > > > > > > so that they don't boot up ahead of a potentially
> > > long-running
> > > > > > > > migration
> > > > > > > > > > and attempt to operate on new or missing columns/tables
> > > before the
> > > > > > > > > > migrations run. This prevents these pods from entering a
> > > CrashLoop.
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > @Tomasz great question. Our images are currently
> > > generated from
> > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > in this repo https://github.com/astronomer/ap-airflow
> > > and get
> > > > > > > > > > > published to
> > > > > > > > > > > > DockerHub
> > > > > > > > > > > >
> > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > > > > > > > >
> > > > > > > > > > > > For the most part those are typical Airflow images.
> > > There's an
> > > > > > > > > > > entrypoint
> > > > > > > > > > > > script that we include in the image that handles waiting
> > > for the
> > > > > > > > > > > database
> > > > > > > > > > > > and redis (if used) to come up, which is pretty generic.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I already added waiting for the database (both metadata
> > > and celery
> > > > > > > > URL) in
> > > > > > > > > > > the PR:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > > > > > > > .
> > > > > > > > > > > It's functionally the same but more generic.
> > > > > > > > > > >
> > > > > > > > > > > The only other
> > > > > > > > > > > > thing that I think the Helm Chart uses would be the
> > > scripts in this
> > > > > > > > repo
> > > > > > > > > > > > https://github.com/astronomer/astronomer-airflow-scripts.
> > > Our
> > > > > > > > > > > Dockerfiles
> > > > > > > > > > > > pull this package in. These scripts are used to
> > > coordinate running
> > > > > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I see two scripts:
> > > > > > > > > > >
> > > > > > > > > > > * cleanup_pods -> this is (I believe) not needed to run in
> > > airflow -
> > > > > > > > this
> > > > > > > > > > > could be run as a separate pod/container?
> > > > > > > > > > > * waiting for migrations -> I think this is a good
> > > candidate to add
> > > > > > > > > > > *airflow
> > > > > > > > > > > db wait_for_migration* command and make it part of airflow
> > > itself.
> > > > > > > > > > >
> > > > > > > > > > > I think we also have to agree on the Airflow version
> > > supported by the
> > > > > > > > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > > > > > > > incorporate
> > > > > > > > > > > all
> > > > > > > > > > > the changes needed to airflow (like the "db
> > > wait_for_migration")
> > > > > > > > into 2.0
> > > > > > > > > > > and 1.10 and we support both - image and helm chart for
> > > those versions
> > > > > > > > > > > only. That would help with people migrating to the latest
> > > version.
> > > > > > > > > > >
> > > > > > > > > > > WDYT?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > @jarek I agree completely. I think that pairing an
> > > official helm
> > > > > > > > chart
> > > > > > > > > > > > > with the official image would make for a REALLY
> > > powerful “up and
> > > > > > > > > > > running
> > > > > > > > > > > > > with airflow” story :). Tomek and I have also been
> > > looking into
> > > > > > > > > > > > > operator-sdk which has the ability to create custom
> > > controllers
> > > > > > > > from
> > > > > > > > > > > helm
> > > > > > > > > > > > > charts. We might even able to get a 1-2 punch from the
> > > same code
> > > > > > > > base
> > > > > > > > > > > :).
> > > > > > > > > > > > >
> > > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can
> > > we please
> > > > > > > > start
> > > > > > > > > > > > the
> > > > > > > > > > > > > process of donation?
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Daniel
> > > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > +1. And it should be paired with the official image
> > > we have
> > > > > > > > work in
> > > > > > > > > > > > > > progress on. I looked a lot at the Astronomer's
> > > image while
> > > > > > > > > > > preparing
> > > > > > > > > > > > my
> > > > > > > > > > > > > > draft and we can make any adjustments needed to make
> > > it works
> > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > helm
> > > > > > > > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > PR here: https://github.com/apache/airflow/pull/7832
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > J.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
> > > kaxilnaik@gmail.com
> > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > > > > > > > > > > > Helm Chart Link:
> > > https://github.com/astronomer/airflow-chart
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > > > > > > > turbaszek@apache.org>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > An official helm chart is something our
> > > community needs!
> > > > > > > > Using
> > > > > > > > > > > your
> > > > > > > > > > > > > > > > chart as the official makes a lot of sens to me
> > > because as
> > > > > > > > you
> > > > > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > One question: what Airflow image do you use?
> > > Also, would you
> > > > > > > > > > > mind
> > > > > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Tomek
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Over the past few years at Astronomer, we’ve
> > > created,
> > > > > > > > managed,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > > > > > > > https://github.com/astronomer/airflow-chart)
> > > that is
> > > > > > > > being
> > > > > > > > > > > used
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > > > > > > > battle-tested and
> > > > > > > > > > > > > running
> > > > > > > > > > > > > > > > > hundreds of Airflow deployments of varying
> > > sizes and
> > > > > > > > runtime
> > > > > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > > > > It’s been built up to encapsulate the issues
> > > that Airflow
> > > > > > > > > > > users
> > > > > > > > > > > > run
> > > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > While this chart was originally developed
> > > internally for
> > > > > > > > our
> > > > > > > > > > > > > Astronomer
> > > > > > > > > > > > > > > > > Platform, we’ve recently decoupled the chart
> > > from the
> > > > > > > > rest of
> > > > > > > > > > > our
> > > > > > > > > > > > > > > > platform
> > > > > > > > > > > > > > > > > to make it usable by the greater Airflow
> > > community. With
> > > > > > > > these
> > > > > > > > > > > > > changes
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > mind, we want to start a conversation about
> > > donating this
> > > > > > > > > > > chart
> > > > > > > > > > > > to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - It works out of the box. With zero
> > > configuration, a user
> > > > > > > > > > > will
> > > > > > > > > > > > get
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > postgres database, a default user and the
> > > > > > > > KubernetesExecutor
> > > > > > > > > > > > ready
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > > > > > > > autoscaling) and
> > > > > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Support for optional pgbouncer. We use this to
> > > share a
> > > > > > > > > > > > configurable
> > > > > > > > > > > > > > > > > connection pool size per deployment. Useful
> > > for limiting
> > > > > > > > > > > > > connections to
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > - Airflow migration support. A user can push a
> > > newer
> > > > > > > > version
> > > > > > > > > > > of
> > > > > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > > > > into an existing release and migrations will
> > > > > > > > automatically run
> > > > > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > > > > - Prometheus support. Optionally install and
> > > configure a
> > > > > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > > > > to ingest Airflow metrics and expose them to
> > > Prometheus
> > > > > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > > > > - Resource control. Optionally control the
> > > ResourceQuotas
> > > > > > > > and
> > > > > > > > > > > > > > > > > LimitRanges for each deployment so that no
> > > deployment can
> > > > > > > > > > > > overload
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > > > > KubernetesExecutor and
> > > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for reasons
> > > other than the
> > > > > > > > > > > actual
> > > > > > > > > > > > > > > > task.
> > > > > > > > > > > > > > > > > This feature helps keep things clean in
> > > Kubernetes.
> > > > > > > > > > > > > > > > > - Support for running locally in KIND
> > > (Kubernetes in
> > > > > > > > Docker).
> > > > > > > > > > > > > > > > > - Automatically tested across many Kubernetes
> > > versions
> > > > > > > > with
> > > > > > > > > > > Helm
> > > > > > > > > > > > 2
> > > > > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We’ve found that the cleanest and most
> > > reliable way to
> > > > > > > > deploy
> > > > > > > > > > > > DAGs
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > Kubernetes and manage them at scale is to
> > > package them
> > > > > > > > into
> > > > > > > > > > > the
> > > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > > docker image, so we have geared this chart
> > > towards that
> > > > > > > > > > > method of
> > > > > > > > > > > > > > > > > operation, though adding other methods should
> > > be
> > > > > > > > > > > straightforward.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We would love thoughts from the community and
> > > would love
> > > > > > > > to
> > > > > > > > > > > see
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > chart
> > > > > > > > > > > > > > > > > help others to get up and running on
> > > Kubernetes!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > > Astronomer.io
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
> > > Software
> > > > > > > > Engineer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> > > <+48660796129
> > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> > > Engineer
> > > > > > > > > > >
> > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jarek Potiuk
> > > > > > > > Polidea | Principal Software Engineer
> > > > > > > >
> > > > > > > > M: +48 660 796 129
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > >
> > > >
> > > > --
> > > > Jarek Potiuk
> > > > Polidea | Principal Software Engineer
> > > > M: +48 660 796 129
> > > >
> > >
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: Helm Chart for Airflow
Posted by Daniel Imberman <da...@gmail.com>.
@Steven glad to help you out on that.
For the moment we only use travis, but I believe @jarek has been looking to move to GitHub CI/CD. Perhaps we can use this as our first GitHub tests?
On Mar 26, 2020, 8:17 AM -0700, Steven Miller <st...@astronomer.io.invalid>, wrote:
> Hey I’m happy to PR in the chart and get the CI working to test it. Where
> do we want to run it? Do you all just use Travis only?
>
> Steven
>
> On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <da...@gmail.com>
> wrote:
>
> > @Jarek I think with the helm chart + prod image we can go even further
> > than that :). We can test CeleryExecutor, with KEDA autoscaling, and a
> > bunch of other configurations.
> > On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> > wrote:
> > > Yeah. I meant Custom Resources not CRDs in my original email :)
> > >
> > > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> > daniel.imberman@gmail.com> wrote:
> > > > > We’re not using CRDs for the tests at the moment. We just have
> > deployment files. If anything having the helm chart as a part of the
> > airflow repo could mean that the helm chart becomes the defacto system for
> > testing airflow on kubernetes (we can get rid of all the yams files and run
> > multiple k8s tests with different settings).
> > > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> > <gr...@astronomer.io.invalid>, wrote:
> > > > > > Yep, we can absolutely pull it into the airflow repo. We've also
> > been
> > > > > > building up a test suite that currently runs on CircleCI and uses
> > kind
> > > > > > (Kubernetes in Docker) to test several kubernetes versions with
> > some
> > > > > > different settings. Right now we're mostly testing the different
> > executors
> > > > > > since that has the biggest impact on what gets deployed, but that
> > can be
> > > > > > expanded.
> > > > > >
> > > > > > What CRDs are currently being used to run Airflow for the tests?
> > > > > >
> > > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > > > wrote:
> > > > > >
> > > > > > > One thing for the donation.
> > > > > > >
> > > > > > > Did you you want to have separate repository Greg ?
> > > > > > >
> > > > > > > I think we should simply create a folder in Airflow repo and
> > keep it
> > > > > > > there (similarly as we keep Dockerfile). I am going to switch our
> > > > > > > Kubernetes Tests to the production image (will make the tests
> > much
> > > > > > > faster) and I am going to test the Dockerfile automatically in
> > CI -
> > > > > > > for now we are using some custom Resource definitions to start
> > Airflow
> > > > > > > on Kubernetes Cluster for the tests, but we could switch to
> > using the
> > > > > > > helm chart - this way we can test all three things at once:
> > > > > > > - Kubernetes Executor
> > > > > > > - Dockerfile
> > > > > > > - Helm Chart
> > > > > > > and we could also add more tests - for example testing different
> > > > > > > deployment options for the helm chart.
> > > > > > >
> > > > > > > Having the Helm chart in Airflow repo would help with that -
> > > > > > > especially in terms of future changes and testing them
> > automatically.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > > > <ai...@apache.org> wrote:
> > > > > > > >
> > > > > > > > +1 on the donation. Always happy to see more useful stuff for
> > the
> > > > > > > > community :)
> > > > > > > >
> > > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
> > greg@astronomer.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yep, the cleanup_pods script is set up now as an optional
> > Kubernetes
> > > > > > > > > CronJob (
> > > > > > > > >
> > > > > > >
> > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > > > > > )
> > > > > > > > > that we have run periodically to clean failed pods up and
> > could stay
> > > > > > > > > separate.
> > > > > > > > >
> > > > > > > > > The wait_for_migrations script could definitely be pulled
> > into Airflow.
> > > > > > > > > For context, we deploy an initContainer on the scheduler (
> > > > > > > > >
> > > > > > >
> > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > > > > > )
> > > > > > > > > that runs the upgradedb command before booting the
> > scheduler. This new
> > > > > > > > > wait_for_migration script runs in an initContainer on the
> > webserver and
> > > > > > > > > workers (
> > > > > > > > >
> > > > > > >
> > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > > > > > )
> > > > > > > > > so that they don't boot up ahead of a potentially
> > long-running
> > > > > > > migration
> > > > > > > > > and attempt to operate on new or missing columns/tables
> > before the
> > > > > > > > > migrations run. This prevents these pods from entering a
> > CrashLoop.
> > > > > > > > >
> > > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > @Tomasz great question. Our images are currently
> > generated from
> > > > > > > > > > Dockerfiles
> > > > > > > > > > > in this repo https://github.com/astronomer/ap-airflow
> > and get
> > > > > > > > > > published to
> > > > > > > > > > > DockerHub
> > > > > > > > > > >
> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > > > > > > >
> > > > > > > > > > > For the most part those are typical Airflow images.
> > There's an
> > > > > > > > > > entrypoint
> > > > > > > > > > > script that we include in the image that handles waiting
> > for the
> > > > > > > > > > database
> > > > > > > > > > > and redis (if used) to come up, which is pretty generic.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I already added waiting for the database (both metadata
> > and celery
> > > > > > > URL) in
> > > > > > > > > > the PR:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > > > > > > .
> > > > > > > > > > It's functionally the same but more generic.
> > > > > > > > > >
> > > > > > > > > > The only other
> > > > > > > > > > > thing that I think the Helm Chart uses would be the
> > scripts in this
> > > > > > > repo
> > > > > > > > > > > https://github.com/astronomer/astronomer-airflow-scripts.
> > Our
> > > > > > > > > > Dockerfiles
> > > > > > > > > > > pull this package in. These scripts are used to
> > coordinate running
> > > > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I see two scripts:
> > > > > > > > > >
> > > > > > > > > > * cleanup_pods -> this is (I believe) not needed to run in
> > airflow -
> > > > > > > this
> > > > > > > > > > could be run as a separate pod/container?
> > > > > > > > > > * waiting for migrations -> I think this is a good
> > candidate to add
> > > > > > > > > > *airflow
> > > > > > > > > > db wait_for_migration* command and make it part of airflow
> > itself.
> > > > > > > > > >
> > > > > > > > > > I think we also have to agree on the Airflow version
> > supported by the
> > > > > > > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > > > > > > incorporate
> > > > > > > > > > all
> > > > > > > > > > the changes needed to airflow (like the "db
> > wait_for_migration")
> > > > > > > into 2.0
> > > > > > > > > > and 1.10 and we support both - image and helm chart for
> > those versions
> > > > > > > > > > only. That would help with people migrating to the latest
> > version.
> > > > > > > > > >
> > > > > > > > > > WDYT?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > @jarek I agree completely. I think that pairing an
> > official helm
> > > > > > > chart
> > > > > > > > > > > > with the official image would make for a REALLY
> > powerful “up and
> > > > > > > > > > running
> > > > > > > > > > > > with airflow” story :). Tomek and I have also been
> > looking into
> > > > > > > > > > > > operator-sdk which has the ability to create custom
> > controllers
> > > > > > > from
> > > > > > > > > > helm
> > > > > > > > > > > > charts. We might even able to get a 1-2 punch from the
> > same code
> > > > > > > base
> > > > > > > > > > :).
> > > > > > > > > > > >
> > > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can
> > we please
> > > > > > > start
> > > > > > > > > > > the
> > > > > > > > > > > > process of donation?
> > > > > > > > > > > >
> > > > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Daniel
> > > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > +1. And it should be paired with the official image
> > we have
> > > > > > > work in
> > > > > > > > > > > > > progress on. I looked a lot at the Astronomer's
> > image while
> > > > > > > > > > preparing
> > > > > > > > > > > my
> > > > > > > > > > > > > draft and we can make any adjustments needed to make
> > it works
> > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > helm
> > > > > > > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > PR here: https://github.com/apache/airflow/pull/7832
> > > > > > > > > > > > >
> > > > > > > > > > > > > J.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
> > kaxilnaik@gmail.com
> > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > > > > > > > > > > Helm Chart Link:
> > https://github.com/astronomer/airflow-chart
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > > > > > > turbaszek@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > An official helm chart is something our
> > community needs!
> > > > > > > Using
> > > > > > > > > > your
> > > > > > > > > > > > > > > chart as the official makes a lot of sens to me
> > because as
> > > > > > > you
> > > > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > One question: what Airflow image do you use?
> > Also, would you
> > > > > > > > > > mind
> > > > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Tomek
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Over the past few years at Astronomer, we’ve
> > created,
> > > > > > > managed,
> > > > > > > > > > > and
> > > > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > > > > > > https://github.com/astronomer/airflow-chart)
> > that is
> > > > > > > being
> > > > > > > > > > used
> > > > > > > > > > > by
> > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > > > > > > battle-tested and
> > > > > > > > > > > > running
> > > > > > > > > > > > > > > > hundreds of Airflow deployments of varying
> > sizes and
> > > > > > > runtime
> > > > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > > > It’s been built up to encapsulate the issues
> > that Airflow
> > > > > > > > > > users
> > > > > > > > > > > run
> > > > > > > > > > > > > > into
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > While this chart was originally developed
> > internally for
> > > > > > > our
> > > > > > > > > > > > Astronomer
> > > > > > > > > > > > > > > > Platform, we’ve recently decoupled the chart
> > from the
> > > > > > > rest of
> > > > > > > > > > our
> > > > > > > > > > > > > > > platform
> > > > > > > > > > > > > > > > to make it usable by the greater Airflow
> > community. With
> > > > > > > these
> > > > > > > > > > > > changes
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > mind, we want to start a conversation about
> > donating this
> > > > > > > > > > chart
> > > > > > > > > > > to
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - It works out of the box. With zero
> > configuration, a user
> > > > > > > > > > will
> > > > > > > > > > > get
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > postgres database, a default user and the
> > > > > > > KubernetesExecutor
> > > > > > > > > > > ready
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > > > > > > autoscaling) and
> > > > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Support for optional pgbouncer. We use this to
> > share a
> > > > > > > > > > > configurable
> > > > > > > > > > > > > > > > connection pool size per deployment. Useful
> > for limiting
> > > > > > > > > > > > connections to
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > - Airflow migration support. A user can push a
> > newer
> > > > > > > version
> > > > > > > > > > of
> > > > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > > > into an existing release and migrations will
> > > > > > > automatically run
> > > > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > > > - Prometheus support. Optionally install and
> > configure a
> > > > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > > > to ingest Airflow metrics and expose them to
> > Prometheus
> > > > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > > > - Resource control. Optionally control the
> > ResourceQuotas
> > > > > > > and
> > > > > > > > > > > > > > > > LimitRanges for each deployment so that no
> > deployment can
> > > > > > > > > > > overload
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > > > KubernetesExecutor and
> > > > > > > > > > > > > > > > KubernetesPodOperator pods fail for reasons
> > other than the
> > > > > > > > > > actual
> > > > > > > > > > > > > > > task.
> > > > > > > > > > > > > > > > This feature helps keep things clean in
> > Kubernetes.
> > > > > > > > > > > > > > > > - Support for running locally in KIND
> > (Kubernetes in
> > > > > > > Docker).
> > > > > > > > > > > > > > > > - Automatically tested across many Kubernetes
> > versions
> > > > > > > with
> > > > > > > > > > Helm
> > > > > > > > > > > 2
> > > > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We’ve found that the cleanest and most
> > reliable way to
> > > > > > > deploy
> > > > > > > > > > > DAGs
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > Kubernetes and manage them at scale is to
> > package them
> > > > > > > into
> > > > > > > > > > the
> > > > > > > > > > > > actual
> > > > > > > > > > > > > > > > docker image, so we have geared this chart
> > towards that
> > > > > > > > > > method of
> > > > > > > > > > > > > > > > operation, though adding other methods should
> > be
> > > > > > > > > > straightforward.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We would love thoughts from the community and
> > would love
> > > > > > > to
> > > > > > > > > > see
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > chart
> > > > > > > > > > > > > > > > help others to get up and running on
> > Kubernetes!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> > Astronomer.io
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
> > Software
> > > > > > > Engineer
> > > > > > > > > > > > >
> > > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> > <+48660796129
> > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > > Jarek Potiuk
> > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > > > > > > > >
> > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Jarek Potiuk
> > > > > > > Polidea | Principal Software Engineer
> > > > > > >
> > > > > > > M: +48 660 796 129
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >
> > >
> > > --
> > > Jarek Potiuk
> > > Polidea | Principal Software Engineer
> > > M: +48 660 796 129
> > >
> >
Re: Helm Chart for Airflow
Posted by Steven Miller <st...@astronomer.io.INVALID>.
Hey I’m happy to PR in the chart and get the CI working to test it. Where
do we want to run it? Do you all just use Travis only?
Steven
On Thu, Mar 26, 2020 at 10:59 AM Daniel Imberman <da...@gmail.com>
wrote:
> @Jarek I think with the helm chart + prod image we can go even further
> than that :). We can test CeleryExecutor, with KEDA autoscaling, and a
> bunch of other configurations.
> On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> wrote:
> > Yeah. I meant Custom Resources not CRDs in my original email :)
> >
> > > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <
> daniel.imberman@gmail.com> wrote:
> > > > We’re not using CRDs for the tests at the moment. We just have
> deployment files. If anything having the helm chart as a part of the
> airflow repo could mean that the helm chart becomes the defacto system for
> testing airflow on kubernetes (we can get rid of all the yams files and run
> multiple k8s tests with different settings).
> > > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel
> <gr...@astronomer.io.invalid>, wrote:
> > > > > Yep, we can absolutely pull it into the airflow repo. We've also
> been
> > > > > building up a test suite that currently runs on CircleCI and uses
> kind
> > > > > (Kubernetes in Docker) to test several kubernetes versions with
> some
> > > > > different settings. Right now we're mostly testing the different
> executors
> > > > > since that has the biggest impact on what gets deployed, but that
> can be
> > > > > expanded.
> > > > >
> > > > > What CRDs are currently being used to run Airflow for the tests?
> > > > >
> > > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > > > wrote:
> > > > >
> > > > > > One thing for the donation.
> > > > > >
> > > > > > Did you you want to have separate repository Greg ?
> > > > > >
> > > > > > I think we should simply create a folder in Airflow repo and
> keep it
> > > > > > there (similarly as we keep Dockerfile). I am going to switch our
> > > > > > Kubernetes Tests to the production image (will make the tests
> much
> > > > > > faster) and I am going to test the Dockerfile automatically in
> CI -
> > > > > > for now we are using some custom Resource definitions to start
> Airflow
> > > > > > on Kubernetes Cluster for the tests, but we could switch to
> using the
> > > > > > helm chart - this way we can test all three things at once:
> > > > > > - Kubernetes Executor
> > > > > > - Dockerfile
> > > > > > - Helm Chart
> > > > > > and we could also add more tests - for example testing different
> > > > > > deployment options for the helm chart.
> > > > > >
> > > > > > Having the Helm chart in Airflow repo would help with that -
> > > > > > especially in terms of future changes and testing them
> automatically.
> > > > > >
> > > > > > J.
> > > > > >
> > > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > > <ai...@apache.org> wrote:
> > > > > > >
> > > > > > > +1 on the donation. Always happy to see more useful stuff for
> the
> > > > > > > community :)
> > > > > > >
> > > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <
> greg@astronomer.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Yep, the cleanup_pods script is set up now as an optional
> Kubernetes
> > > > > > > > CronJob (
> > > > > > > >
> > > > > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > > > > )
> > > > > > > > that we have run periodically to clean failed pods up and
> could stay
> > > > > > > > separate.
> > > > > > > >
> > > > > > > > The wait_for_migrations script could definitely be pulled
> into Airflow.
> > > > > > > > For context, we deploy an initContainer on the scheduler (
> > > > > > > >
> > > > > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > > > > )
> > > > > > > > that runs the upgradedb command before booting the
> scheduler. This new
> > > > > > > > wait_for_migration script runs in an initContainer on the
> webserver and
> > > > > > > > workers (
> > > > > > > >
> > > > > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > > > > )
> > > > > > > > so that they don't boot up ahead of a potentially
> long-running
> > > > > > migration
> > > > > > > > and attempt to operate on new or missing columns/tables
> before the
> > > > > > > > migrations run. This prevents these pods from entering a
> CrashLoop.
> > > > > > > >
> > > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > > Jarek.Potiuk@polidea.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > @Tomasz great question. Our images are currently
> generated from
> > > > > > > > > Dockerfiles
> > > > > > > > > > in this repo https://github.com/astronomer/ap-airflow
> and get
> > > > > > > > > published to
> > > > > > > > > > DockerHub
> > > > > > > > > >
> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > > > > > >
> > > > > > > > > > For the most part those are typical Airflow images.
> There's an
> > > > > > > > > entrypoint
> > > > > > > > > > script that we include in the image that handles waiting
> for the
> > > > > > > > > database
> > > > > > > > > > and redis (if used) to come up, which is pretty generic.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I already added waiting for the database (both metadata
> and celery
> > > > > > URL) in
> > > > > > > > > the PR:
> > > > > > > > >
> > > > > > > > >
> > > > > >
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > > > > > .
> > > > > > > > > It's functionally the same but more generic.
> > > > > > > > >
> > > > > > > > > The only other
> > > > > > > > > > thing that I think the Helm Chart uses would be the
> scripts in this
> > > > > > repo
> > > > > > > > > > https://github.com/astronomer/astronomer-airflow-scripts.
> Our
> > > > > > > > > Dockerfiles
> > > > > > > > > > pull this package in. These scripts are used to
> coordinate running
> > > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I see two scripts:
> > > > > > > > >
> > > > > > > > > * cleanup_pods -> this is (I believe) not needed to run in
> airflow -
> > > > > > this
> > > > > > > > > could be run as a separate pod/container?
> > > > > > > > > * waiting for migrations -> I think this is a good
> candidate to add
> > > > > > > > > *airflow
> > > > > > > > > db wait_for_migration* command and make it part of airflow
> itself.
> > > > > > > > >
> > > > > > > > > I think we also have to agree on the Airflow version
> supported by the
> > > > > > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > > > > > incorporate
> > > > > > > > > all
> > > > > > > > > the changes needed to airflow (like the "db
> wait_for_migration")
> > > > > > into 2.0
> > > > > > > > > and 1.10 and we support both - image and helm chart for
> those versions
> > > > > > > > > only. That would help with people migrating to the latest
> version.
> > > > > > > > >
> > > > > > > > > WDYT?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > @jarek I agree completely. I think that pairing an
> official helm
> > > > > > chart
> > > > > > > > > > > with the official image would make for a REALLY
> powerful “up and
> > > > > > > > > running
> > > > > > > > > > > with airflow” story :). Tomek and I have also been
> looking into
> > > > > > > > > > > operator-sdk which has the ability to create custom
> controllers
> > > > > > from
> > > > > > > > > helm
> > > > > > > > > > > charts. We might even able to get a 1-2 punch from the
> same code
> > > > > > base
> > > > > > > > > :).
> > > > > > > > > > >
> > > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can
> we please
> > > > > > start
> > > > > > > > > > the
> > > > > > > > > > > process of donation?
> > > > > > > > > > >
> > > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Daniel
> > > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > > > > > wrote:
> > > > > > > > > > > > +1. And it should be paired with the official image
> we have
> > > > > > work in
> > > > > > > > > > > > progress on. I looked a lot at the Astronomer's
> image while
> > > > > > > > > preparing
> > > > > > > > > > my
> > > > > > > > > > > > draft and we can make any adjustments needed to make
> it works
> > > > > > with
> > > > > > > > > the
> > > > > > > > > > > helm
> > > > > > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > > > > > >
> > > > > > > > > > > > PR here: https://github.com/apache/airflow/pull/7832
> > > > > > > > > > > >
> > > > > > > > > > > > J.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <
> kaxilnaik@gmail.com
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > > > > > > > > > Helm Chart Link:
> https://github.com/astronomer/airflow-chart
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > > > > > turbaszek@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > An official helm chart is something our
> community needs!
> > > > > > Using
> > > > > > > > > your
> > > > > > > > > > > > > > chart as the official makes a lot of sens to me
> because as
> > > > > > you
> > > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > One question: what Airflow image do you use?
> Also, would you
> > > > > > > > > mind
> > > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Tomek
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Over the past few years at Astronomer, we’ve
> created,
> > > > > > managed,
> > > > > > > > > > and
> > > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > > > > > https://github.com/astronomer/airflow-chart)
> that is
> > > > > > being
> > > > > > > > > used
> > > > > > > > > > by
> > > > > > > > > > > > > both
> > > > > > > > > > > > > > our
> > > > > > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > > > > > battle-tested and
> > > > > > > > > > > running
> > > > > > > > > > > > > > > hundreds of Airflow deployments of varying
> sizes and
> > > > > > runtime
> > > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > > It’s been built up to encapsulate the issues
> that Airflow
> > > > > > > > > users
> > > > > > > > > > run
> > > > > > > > > > > > > into
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > While this chart was originally developed
> internally for
> > > > > > our
> > > > > > > > > > > Astronomer
> > > > > > > > > > > > > > > Platform, we’ve recently decoupled the chart
> from the
> > > > > > rest of
> > > > > > > > > our
> > > > > > > > > > > > > > platform
> > > > > > > > > > > > > > > to make it usable by the greater Airflow
> community. With
> > > > > > these
> > > > > > > > > > > changes
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > mind, we want to start a conversation about
> donating this
> > > > > > > > > chart
> > > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - It works out of the box. With zero
> configuration, a user
> > > > > > > > > will
> > > > > > > > > > get
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > postgres database, a default user and the
> > > > > > KubernetesExecutor
> > > > > > > > > > ready
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > run
> > > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > > > > > autoscaling) and
> > > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Support for optional pgbouncer. We use this to
> share a
> > > > > > > > > > configurable
> > > > > > > > > > > > > > > connection pool size per deployment. Useful
> for limiting
> > > > > > > > > > > connections to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > - Airflow migration support. A user can push a
> newer
> > > > > > version
> > > > > > > > > of
> > > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > > into an existing release and migrations will
> > > > > > automatically run
> > > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > > - Prometheus support. Optionally install and
> configure a
> > > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > > to ingest Airflow metrics and expose them to
> Prometheus
> > > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > > - Resource control. Optionally control the
> ResourceQuotas
> > > > > > and
> > > > > > > > > > > > > > > LimitRanges for each deployment so that no
> deployment can
> > > > > > > > > > overload
> > > > > > > > > > > a
> > > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > > KubernetesExecutor and
> > > > > > > > > > > > > > > KubernetesPodOperator pods fail for reasons
> other than the
> > > > > > > > > actual
> > > > > > > > > > > > > > task.
> > > > > > > > > > > > > > > This feature helps keep things clean in
> Kubernetes.
> > > > > > > > > > > > > > > - Support for running locally in KIND
> (Kubernetes in
> > > > > > Docker).
> > > > > > > > > > > > > > > - Automatically tested across many Kubernetes
> versions
> > > > > > with
> > > > > > > > > Helm
> > > > > > > > > > 2
> > > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > > support.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We’ve found that the cleanest and most
> reliable way to
> > > > > > deploy
> > > > > > > > > > DAGs
> > > > > > > > > > > to
> > > > > > > > > > > > > > > Kubernetes and manage them at scale is to
> package them
> > > > > > into
> > > > > > > > > the
> > > > > > > > > > > actual
> > > > > > > > > > > > > > > docker image, so we have geared this chart
> towards that
> > > > > > > > > method of
> > > > > > > > > > > > > > > operation, though adding other methods should
> be
> > > > > > > > > straightforward.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We would love thoughts from the community and
> would love
> > > > > > to
> > > > > > > > > see
> > > > > > > > > > > this
> > > > > > > > > > > > > > chart
> > > > > > > > > > > > > > > help others to get up and running on
> Kubernetes!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect
> Astronomer.io
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal
> Software
> > > > > > Engineer
> > > > > > > > > > > >
> > > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129>
> <+48660796129
> > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Jarek Potiuk
> > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > > > > > > > >
> > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Jarek Potiuk
> > > > > > Polidea | Principal Software Engineer
> > > > > >
> > > > > > M: +48 660 796 129
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> >
> >
> > --
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> > M: +48 660 796 129
> >
>
Re: Helm Chart for Airflow
Posted by Daniel Imberman <da...@gmail.com>.
@Jarek I think with the helm chart + prod image we can go even further than that :). We can test CeleryExecutor, with KEDA autoscaling, and a bunch of other configurations.
On Mar 26, 2020, 7:45 AM -0700, Jarek Potiuk <Ja...@polidea.com>, wrote:
> Yeah. I meant Custom Resources not CRDs in my original email :)
>
> > On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <da...@gmail.com> wrote:
> > > We’re not using CRDs for the tests at the moment. We just have deployment files. If anything having the helm chart as a part of the airflow repo could mean that the helm chart becomes the defacto system for testing airflow on kubernetes (we can get rid of all the yams files and run multiple k8s tests with different settings).
> > > On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel <gr...@astronomer.io.invalid>, wrote:
> > > > Yep, we can absolutely pull it into the airflow repo. We've also been
> > > > building up a test suite that currently runs on CircleCI and uses kind
> > > > (Kubernetes in Docker) to test several kubernetes versions with some
> > > > different settings. Right now we're mostly testing the different executors
> > > > since that has the biggest impact on what gets deployed, but that can be
> > > > expanded.
> > > >
> > > > What CRDs are currently being used to run Airflow for the tests?
> > > >
> > > > On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <Ja...@polidea.com>
> > > > wrote:
> > > >
> > > > > One thing for the donation.
> > > > >
> > > > > Did you you want to have separate repository Greg ?
> > > > >
> > > > > I think we should simply create a folder in Airflow repo and keep it
> > > > > there (similarly as we keep Dockerfile). I am going to switch our
> > > > > Kubernetes Tests to the production image (will make the tests much
> > > > > faster) and I am going to test the Dockerfile automatically in CI -
> > > > > for now we are using some custom Resource definitions to start Airflow
> > > > > on Kubernetes Cluster for the tests, but we could switch to using the
> > > > > helm chart - this way we can test all three things at once:
> > > > > - Kubernetes Executor
> > > > > - Dockerfile
> > > > > - Helm Chart
> > > > > and we could also add more tests - for example testing different
> > > > > deployment options for the helm chart.
> > > > >
> > > > > Having the Helm chart in Airflow repo would help with that -
> > > > > especially in terms of future changes and testing them automatically.
> > > > >
> > > > > J.
> > > > >
> > > > > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > > > > <ai...@apache.org> wrote:
> > > > > >
> > > > > > +1 on the donation. Always happy to see more useful stuff for the
> > > > > > community :)
> > > > > >
> > > > > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io>
> > > > > wrote:
> > > > > >
> > > > > > > Yep, the cleanup_pods script is set up now as an optional Kubernetes
> > > > > > > CronJob (
> > > > > > >
> > > > > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > > > > )
> > > > > > > that we have run periodically to clean failed pods up and could stay
> > > > > > > separate.
> > > > > > >
> > > > > > > The wait_for_migrations script could definitely be pulled into Airflow.
> > > > > > > For context, we deploy an initContainer on the scheduler (
> > > > > > >
> > > > > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > > > > )
> > > > > > > that runs the upgradedb command before booting the scheduler. This new
> > > > > > > wait_for_migration script runs in an initContainer on the webserver and
> > > > > > > workers (
> > > > > > >
> > > > > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > > > > )
> > > > > > > so that they don't boot up ahead of a potentially long-running
> > > > > migration
> > > > > > > and attempt to operate on new or missing columns/tables before the
> > > > > > > migrations run. This prevents these pods from entering a CrashLoop.
> > > > > > >
> > > > > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > >
> > > > > > > > > @Tomasz great question. Our images are currently generated from
> > > > > > > > Dockerfiles
> > > > > > > > > in this repo https://github.com/astronomer/ap-airflow and get
> > > > > > > > published to
> > > > > > > > > DockerHub
> > > > > > > > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > > > > >
> > > > > > > > > For the most part those are typical Airflow images. There's an
> > > > > > > > entrypoint
> > > > > > > > > script that we include in the image that handles waiting for the
> > > > > > > > database
> > > > > > > > > and redis (if used) to come up, which is pretty generic.
> > > > > > > >
> > > > > > > >
> > > > > > > > I already added waiting for the database (both metadata and celery
> > > > > URL) in
> > > > > > > > the PR:
> > > > > > > >
> > > > > > > >
> > > > > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > > > > .
> > > > > > > > It's functionally the same but more generic.
> > > > > > > >
> > > > > > > > The only other
> > > > > > > > > thing that I think the Helm Chart uses would be the scripts in this
> > > > > repo
> > > > > > > > > https://github.com/astronomer/astronomer-airflow-scripts. Our
> > > > > > > > Dockerfiles
> > > > > > > > > pull this package in. These scripts are used to coordinate running
> > > > > > > > > migrations and cleaning up failed pods.
> > > > > > > > >
> > > > > > > >
> > > > > > > > I see two scripts:
> > > > > > > >
> > > > > > > > * cleanup_pods -> this is (I believe) not needed to run in airflow -
> > > > > this
> > > > > > > > could be run as a separate pod/container?
> > > > > > > > * waiting for migrations -> I think this is a good candidate to add
> > > > > > > > *airflow
> > > > > > > > db wait_for_migration* command and make it part of airflow itself.
> > > > > > > >
> > > > > > > > I think we also have to agree on the Airflow version supported by the
> > > > > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > > > > incorporate
> > > > > > > > all
> > > > > > > > the changes needed to airflow (like the "db wait_for_migration")
> > > > > into 2.0
> > > > > > > > and 1.10 and we support both - image and helm chart for those versions
> > > > > > > > only. That would help with people migrating to the latest version.
> > > > > > > >
> > > > > > > > WDYT?
> > > > > > > >
> > > > > > > >
> > > > > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > > > > daniel.imberman@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > @jarek I agree completely. I think that pairing an official helm
> > > > > chart
> > > > > > > > > > with the official image would make for a REALLY powerful “up and
> > > > > > > > running
> > > > > > > > > > with airflow” story :). Tomek and I have also been looking into
> > > > > > > > > > operator-sdk which has the ability to create custom controllers
> > > > > from
> > > > > > > > helm
> > > > > > > > > > charts. We might even able to get a 1-2 punch from the same code
> > > > > base
> > > > > > > > :).
> > > > > > > > > >
> > > > > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please
> > > > > start
> > > > > > > > > the
> > > > > > > > > > process of donation?
> > > > > > > > > >
> > > > > > > > > > +1 on my part, of course :)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Daniel
> > > > > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > > > > wrote:
> > > > > > > > > > > +1. And it should be paired with the official image we have
> > > > > work in
> > > > > > > > > > > progress on. I looked a lot at the Astronomer's image while
> > > > > > > > preparing
> > > > > > > > > my
> > > > > > > > > > > draft and we can make any adjustments needed to make it works
> > > > > with
> > > > > > > > the
> > > > > > > > > > helm
> > > > > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > > > > >
> > > > > > > > > > > PR here: https://github.com/apache/airflow/pull/7832
> > > > > > > > > > >
> > > > > > > > > > > J.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <kaxilnaik@gmail.com
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > > > > > > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > > > > turbaszek@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > An official helm chart is something our community needs!
> > > > > Using
> > > > > > > > your
> > > > > > > > > > > > > chart as the official makes a lot of sens to me because as
> > > > > you
> > > > > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One question: what Airflow image do you use? Also, would you
> > > > > > > > mind
> > > > > > > > > > > > > sharing a link to the chart?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Tomek
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hey everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Over the past few years at Astronomer, we’ve created,
> > > > > managed,
> > > > > > > > > and
> > > > > > > > > > > > > hardened
> > > > > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > > > > https://github.com/astronomer/airflow-chart) that is
> > > > > being
> > > > > > > > used
> > > > > > > > > by
> > > > > > > > > > > > both
> > > > > > > > > > > > > our
> > > > > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > > > > battle-tested and
> > > > > > > > > > running
> > > > > > > > > > > > > > hundreds of Airflow deployments of varying sizes and
> > > > > runtime
> > > > > > > > > > > > > environments.
> > > > > > > > > > > > > > It’s been built up to encapsulate the issues that Airflow
> > > > > > > > users
> > > > > > > > > run
> > > > > > > > > > > > into
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > the real world.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > While this chart was originally developed internally for
> > > > > our
> > > > > > > > > > Astronomer
> > > > > > > > > > > > > > Platform, we’ve recently decoupled the chart from the
> > > > > rest of
> > > > > > > > our
> > > > > > > > > > > > > platform
> > > > > > > > > > > > > > to make it usable by the greater Airflow community. With
> > > > > these
> > > > > > > > > > changes
> > > > > > > > > > > > in
> > > > > > > > > > > > > > mind, we want to start a conversation about donating this
> > > > > > > > chart
> > > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > Airflow community.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - It works out of the box. With zero configuration, a user
> > > > > > > > will
> > > > > > > > > get
> > > > > > > > > > > > a
> > > > > > > > > > > > > > postgres database, a default user and the
> > > > > KubernetesExecutor
> > > > > > > > > ready
> > > > > > > > > > > > to
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > DAGs.
> > > > > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > > > > autoscaling) and
> > > > > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Support for optional pgbouncer. We use this to share a
> > > > > > > > > configurable
> > > > > > > > > > > > > > connection pool size per deployment. Useful for limiting
> > > > > > > > > > connections to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > metadata database.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > - Airflow migration support. A user can push a newer
> > > > > version
> > > > > > > > of
> > > > > > > > > > > > > Airflow
> > > > > > > > > > > > > > into an existing release and migrations will
> > > > > automatically run
> > > > > > > > > > > > > cleanly.
> > > > > > > > > > > > > > - Prometheus support. Optionally install and configure a
> > > > > > > > > > > > > statsd-exporter
> > > > > > > > > > > > > > to ingest Airflow metrics and expose them to Prometheus
> > > > > > > > > > > > automatically.
> > > > > > > > > > > > > > - Resource control. Optionally control the ResourceQuotas
> > > > > and
> > > > > > > > > > > > > > LimitRanges for each deployment so that no deployment can
> > > > > > > > > overload
> > > > > > > > > > a
> > > > > > > > > > > > > > cluster.
> > > > > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > > > > KubernetesExecutor and
> > > > > > > > > > > > > > KubernetesPodOperator pods fail for reasons other than the
> > > > > > > > actual
> > > > > > > > > > > > > task.
> > > > > > > > > > > > > > This feature helps keep things clean in Kubernetes.
> > > > > > > > > > > > > > - Support for running locally in KIND (Kubernetes in
> > > > > Docker).
> > > > > > > > > > > > > > - Automatically tested across many Kubernetes versions
> > > > > with
> > > > > > > > Helm
> > > > > > > > > 2
> > > > > > > > > > > > > and 3
> > > > > > > > > > > > > > support.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We’ve found that the cleanest and most reliable way to
> > > > > deploy
> > > > > > > > > DAGs
> > > > > > > > > > to
> > > > > > > > > > > > > > Kubernetes and manage them at scale is to package them
> > > > > into
> > > > > > > > the
> > > > > > > > > > actual
> > > > > > > > > > > > > > docker image, so we have geared this chart towards that
> > > > > > > > method of
> > > > > > > > > > > > > > operation, though adding other methods should be
> > > > > > > > straightforward.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We would love thoughts from the community and would love
> > > > > to
> > > > > > > > see
> > > > > > > > > > this
> > > > > > > > > > > > > chart
> > > > > > > > > > > > > > help others to get up and running on Kubernetes!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > > Jarek Potiuk
> > > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> > > > > Engineer
> > > > > > > > > > >
> > > > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jarek Potiuk
> > > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > > > > >
> > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > > > > <+48%20660%20796%20129>>
> > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129
> > > > >
> > > >
> > > >
> > > > --
> > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>
>
> --
> Jarek Potiuk
> Polidea | Principal Software Engineer
> M: +48 660 796 129
>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
Yeah. I meant Custom Resources not CRDs in my original email :)
On Thu, Mar 26, 2020 at 3:38 PM Daniel Imberman <da...@gmail.com>
wrote:
> We’re not using CRDs for the tests at the moment. We just have deployment
> files. If anything having the helm chart as a part of the airflow repo
> could mean that the helm chart becomes the defacto system for testing
> airflow on kubernetes (we can get rid of all the yams files and run
> multiple k8s tests with different settings).
> On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel <gr...@astronomer.io.invalid>,
> wrote:
>
> Yep, we can absolutely pull it into the airflow repo. We've also been
> building up a test suite that currently runs on CircleCI and uses kind
> (Kubernetes in Docker) to test several kubernetes versions with some
> different settings. Right now we're mostly testing the different executors
> since that has the biggest impact on what gets deployed, but that can be
> expanded.
>
> What CRDs are currently being used to run Airflow for the tests?
>
> On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> One thing for the donation.
>
> Did you you want to have separate repository Greg ?
>
> I think we should simply create a folder in Airflow repo and keep it
> there (similarly as we keep Dockerfile). I am going to switch our
> Kubernetes Tests to the production image (will make the tests much
> faster) and I am going to test the Dockerfile automatically in CI -
> for now we are using some custom Resource definitions to start Airflow
> on Kubernetes Cluster for the tests, but we could switch to using the
> helm chart - this way we can test all three things at once:
> - Kubernetes Executor
> - Dockerfile
> - Helm Chart
> and we could also add more tests - for example testing different
> deployment options for the helm chart.
>
> Having the Helm chart in Airflow repo would help with that -
> especially in terms of future changes and testing them automatically.
>
> J.
>
> On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> <ai...@apache.org> wrote:
>
>
> +1 on the donation. Always happy to see more useful stuff for the
> community :)
>
> On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io>
>
> wrote:
>
>
> Yep, the cleanup_pods script is set up now as an optional Kubernetes
> CronJob (
>
>
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> )
>
> that we have run periodically to clean failed pods up and could stay
> separate.
>
> The wait_for_migrations script could definitely be pulled into Airflow.
> For context, we deploy an initContainer on the scheduler (
>
>
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> )
>
> that runs the upgradedb command before booting the scheduler. This new
> wait_for_migration script runs in an initContainer on the webserver and
> workers (
>
>
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> )
>
> so that they don't boot up ahead of a potentially long-running
>
> migration
>
> and attempt to operate on new or missing columns/tables before the
> migrations run. This prevents these pods from entering a CrashLoop.
>
> On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
>
> Jarek.Potiuk@polidea.com>
>
> wrote:
>
>
> @Tomasz great question. Our images are currently generated from
>
> Dockerfiles
>
> in this repo https://github.com/astronomer/ap-airflow and get
>
> published to
>
> DockerHub
> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>
> For the most part those are typical Airflow images. There's an
>
> entrypoint
>
> script that we include in the image that handles waiting for the
>
> database
>
> and redis (if used) to come up, which is pretty generic.
>
>
>
> I already added waiting for the database (both metadata and celery
>
> URL) in
>
> the PR:
>
>
>
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>
> .
> It's functionally the same but more generic.
>
> The only other
>
> thing that I think the Helm Chart uses would be the scripts in this
>
> repo
>
> https://github.com/astronomer/astronomer-airflow-scripts. Our
>
> Dockerfiles
>
> pull this package in. These scripts are used to coordinate running
> migrations and cleaning up failed pods.
>
>
> I see two scripts:
>
> * cleanup_pods -> this is (I believe) not needed to run in airflow -
>
> this
>
> could be run as a separate pod/container?
> * waiting for migrations -> I think this is a good candidate to add
> *airflow
> db wait_for_migration* command and make it part of airflow itself.
>
> I think we also have to agree on the Airflow version supported by the
> official helm chart. I'd suggest we support 1.10.10+ and we
>
> incorporate
>
> all
> the changes needed to airflow (like the "db wait_for_migration")
>
> into 2.0
>
> and 1.10 and we support both - image and helm chart for those versions
> only. That would help with people migrating to the latest version.
>
> WDYT?
>
>
> On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> @jarek I agree completely. I think that pairing an official helm
>
> chart
>
> with the official image would make for a REALLY powerful “up and
>
> running
>
> with airflow” story :). Tomek and I have also been looking into
> operator-sdk which has the ability to create custom controllers
>
> from
>
> helm
>
> charts. We might even able to get a 1-2 punch from the same code
>
> base
>
> :).
>
>
> @kaxil @jarek @aizhamal @ash if there’s no issues, can we please
>
> start
>
> the
>
> process of donation?
>
> +1 on my part, of course :)
>
>
>
> Daniel
> On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
>
> Jarek.Potiuk@polidea.com>,
>
> wrote:
>
> +1. And it should be paired with the official image we have
>
> work in
>
> progress on. I looked a lot at the Astronomer's image while
>
> preparing
>
> my
>
> draft and we can make any adjustments needed to make it works
>
> with
>
> the
>
> helm
>
> chart - and I am super happy to collaborate on that.
>
> PR here: https://github.com/apache/airflow/pull/7832
>
> J.
>
>
> On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <kaxilnaik@gmail.com
>
>
> wrote:
>
>
> @Tomasz Urbaszek <to...@polidea.com> :
> Helm Chart Link: https://github.com/astronomer/airflow-chart
>
> On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
>
> turbaszek@apache.org>
>
> wrote:
>
> An official helm chart is something our community needs!
>
> Using
>
> your
>
> chart as the official makes a lot of sens to me because as
>
> you
>
> mentioned - it's battle tested.
>
> One question: what Airflow image do you use? Also, would you
>
> mind
>
> sharing a link to the chart?
>
> Tomek
>
>
> On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> <gr...@astronomer.io.invalid> wrote:
>
>
> Hey everyone,
>
> Over the past few years at Astronomer, we’ve created,
>
> managed,
>
> and
>
> hardened
>
> a production-ready Helm Chart for Airflow (
> https://github.com/astronomer/airflow-chart) that is
>
> being
>
> used
>
> by
>
> both
>
> our
>
> SaaS and Enterprise customers. This chart is
>
> battle-tested and
>
> running
>
> hundreds of Airflow deployments of varying sizes and
>
> runtime
>
> environments.
>
> It’s been built up to encapsulate the issues that Airflow
>
> users
>
> run
>
> into
>
> in
>
> the real world.
>
> While this chart was originally developed internally for
>
> our
>
> Astronomer
>
> Platform, we’ve recently decoupled the chart from the
>
> rest of
>
> our
>
> platform
>
> to make it usable by the greater Airflow community. With
>
> these
>
> changes
>
> in
>
> mind, we want to start a conversation about donating this
>
> chart
>
> to
>
> the
>
> Airflow community.
>
> Some of the main features of the chart are:
>
> - It works out of the box. With zero configuration, a user
>
> will
>
> get
>
> a
>
> postgres database, a default user and the
>
> KubernetesExecutor
>
> ready
>
> to
>
> run
>
> DAGs.
> - Support for Local, Celery (w/ optional KEDA
>
> autoscaling) and
>
> Kubernetes executors.
>
> Support for optional pgbouncer. We use this to share a
>
> configurable
>
> connection pool size per deployment. Useful for limiting
>
> connections to
>
> the
>
> metadata database.
>
> - Airflow migration support. A user can push a newer
>
> version
>
> of
>
> Airflow
>
> into an existing release and migrations will
>
> automatically run
>
> cleanly.
>
> - Prometheus support. Optionally install and configure a
>
> statsd-exporter
>
> to ingest Airflow metrics and expose them to Prometheus
>
> automatically.
>
> - Resource control. Optionally control the ResourceQuotas
>
> and
>
> LimitRanges for each deployment so that no deployment can
>
> overload
>
> a
>
> cluster.
> - Simple optional Elasticsearch support.
> - Optional namespace cleanup. Sometimes
>
> KubernetesExecutor and
>
> KubernetesPodOperator pods fail for reasons other than the
>
> actual
>
> task.
>
> This feature helps keep things clean in Kubernetes.
> - Support for running locally in KIND (Kubernetes in
>
> Docker).
>
> - Automatically tested across many Kubernetes versions
>
> with
>
> Helm
>
> 2
>
> and 3
>
> support.
>
> We’ve found that the cleanest and most reliable way to
>
> deploy
>
> DAGs
>
> to
>
> Kubernetes and manage them at scale is to package them
>
> into
>
> the
>
> actual
>
> docker image, so we have geared this chart towards that
>
> method of
>
> operation, though adding other methods should be
>
> straightforward.
>
>
> We would love thoughts from the community and would love
>
> to
>
> see
>
> this
>
> chart
>
> help others to get up and running on Kubernetes!
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
>
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software
>
> Engineer
>
>
> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
>
> <+48%20660%20796%20129>>
>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> <+48%20660%20796%20129>>
> [image: Polidea] <https://www.polidea.com/>
>
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Daniel Imberman <da...@gmail.com>.
We’re not using CRDs for the tests at the moment. We just have deployment files. If anything having the helm chart as a part of the airflow repo could mean that the helm chart becomes the defacto system for testing airflow on kubernetes (we can get rid of all the yams files and run multiple k8s tests with different settings).
On Mar 26, 2020, 7:20 AM -0700, Greg Neiheisel <gr...@astronomer.io.invalid>, wrote:
> Yep, we can absolutely pull it into the airflow repo. We've also been
> building up a test suite that currently runs on CircleCI and uses kind
> (Kubernetes in Docker) to test several kubernetes versions with some
> different settings. Right now we're mostly testing the different executors
> since that has the biggest impact on what gets deployed, but that can be
> expanded.
>
> What CRDs are currently being used to run Airflow for the tests?
>
> On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
> > One thing for the donation.
> >
> > Did you you want to have separate repository Greg ?
> >
> > I think we should simply create a folder in Airflow repo and keep it
> > there (similarly as we keep Dockerfile). I am going to switch our
> > Kubernetes Tests to the production image (will make the tests much
> > faster) and I am going to test the Dockerfile automatically in CI -
> > for now we are using some custom Resource definitions to start Airflow
> > on Kubernetes Cluster for the tests, but we could switch to using the
> > helm chart - this way we can test all three things at once:
> > - Kubernetes Executor
> > - Dockerfile
> > - Helm Chart
> > and we could also add more tests - for example testing different
> > deployment options for the helm chart.
> >
> > Having the Helm chart in Airflow repo would help with that -
> > especially in terms of future changes and testing them automatically.
> >
> > J.
> >
> > On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> > <ai...@apache.org> wrote:
> > >
> > > +1 on the donation. Always happy to see more useful stuff for the
> > > community :)
> > >
> > > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io>
> > wrote:
> > >
> > > > Yep, the cleanup_pods script is set up now as an optional Kubernetes
> > > > CronJob (
> > > >
> > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> > )
> > > > that we have run periodically to clean failed pods up and could stay
> > > > separate.
> > > >
> > > > The wait_for_migrations script could definitely be pulled into Airflow.
> > > > For context, we deploy an initContainer on the scheduler (
> > > >
> > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> > )
> > > > that runs the upgradedb command before booting the scheduler. This new
> > > > wait_for_migration script runs in an initContainer on the webserver and
> > > > workers (
> > > >
> > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> > )
> > > > so that they don't boot up ahead of a potentially long-running
> > migration
> > > > and attempt to operate on new or missing columns/tables before the
> > > > migrations run. This prevents these pods from entering a CrashLoop.
> > > >
> > > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> > Jarek.Potiuk@polidea.com>
> > > > wrote:
> > > >
> > > > > >
> > > > > > @Tomasz great question. Our images are currently generated from
> > > > > Dockerfiles
> > > > > > in this repo https://github.com/astronomer/ap-airflow and get
> > > > > published to
> > > > > > DockerHub
> > > > > > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > > > > >
> > > > > > For the most part those are typical Airflow images. There's an
> > > > > entrypoint
> > > > > > script that we include in the image that handles waiting for the
> > > > > database
> > > > > > and redis (if used) to come up, which is pretty generic.
> > > > >
> > > > >
> > > > > I already added waiting for the database (both metadata and celery
> > URL) in
> > > > > the PR:
> > > > >
> > > > >
> > https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > > > > .
> > > > > It's functionally the same but more generic.
> > > > >
> > > > > The only other
> > > > > > thing that I think the Helm Chart uses would be the scripts in this
> > repo
> > > > > > https://github.com/astronomer/astronomer-airflow-scripts. Our
> > > > > Dockerfiles
> > > > > > pull this package in. These scripts are used to coordinate running
> > > > > > migrations and cleaning up failed pods.
> > > > > >
> > > > >
> > > > > I see two scripts:
> > > > >
> > > > > * cleanup_pods -> this is (I believe) not needed to run in airflow -
> > this
> > > > > could be run as a separate pod/container?
> > > > > * waiting for migrations -> I think this is a good candidate to add
> > > > > *airflow
> > > > > db wait_for_migration* command and make it part of airflow itself.
> > > > >
> > > > > I think we also have to agree on the Airflow version supported by the
> > > > > official helm chart. I'd suggest we support 1.10.10+ and we
> > incorporate
> > > > > all
> > > > > the changes needed to airflow (like the "db wait_for_migration")
> > into 2.0
> > > > > and 1.10 and we support both - image and helm chart for those versions
> > > > > only. That would help with people migrating to the latest version.
> > > > >
> > > > > WDYT?
> > > > >
> > > > >
> > > > > > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > > > > > daniel.imberman@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > @jarek I agree completely. I think that pairing an official helm
> > chart
> > > > > > > with the official image would make for a REALLY powerful “up and
> > > > > running
> > > > > > > with airflow” story :). Tomek and I have also been looking into
> > > > > > > operator-sdk which has the ability to create custom controllers
> > from
> > > > > helm
> > > > > > > charts. We might even able to get a 1-2 punch from the same code
> > base
> > > > > :).
> > > > > > >
> > > > > > > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please
> > start
> > > > > > the
> > > > > > > process of donation?
> > > > > > >
> > > > > > > +1 on my part, of course :)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Daniel
> > > > > > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > > > > Jarek.Potiuk@polidea.com>,
> > > > > > > wrote:
> > > > > > > > +1. And it should be paired with the official image we have
> > work in
> > > > > > > > progress on. I looked a lot at the Astronomer's image while
> > > > > preparing
> > > > > > my
> > > > > > > > draft and we can make any adjustments needed to make it works
> > with
> > > > > the
> > > > > > > helm
> > > > > > > > chart - and I am super happy to collaborate on that.
> > > > > > > >
> > > > > > > > PR here: https://github.com/apache/airflow/pull/7832
> > > > > > > >
> > > > > > > > J.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <kaxilnaik@gmail.com
> > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > > > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> > > > > > > > >
> > > > > > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > > > > > turbaszek@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > An official helm chart is something our community needs!
> > Using
> > > > > your
> > > > > > > > > > chart as the official makes a lot of sens to me because as
> > you
> > > > > > > > > > mentioned - it's battle tested.
> > > > > > > > > >
> > > > > > > > > > One question: what Airflow image do you use? Also, would you
> > > > > mind
> > > > > > > > > > sharing a link to the chart?
> > > > > > > > > >
> > > > > > > > > > Tomek
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hey everyone,
> > > > > > > > > > >
> > > > > > > > > > > Over the past few years at Astronomer, we’ve created,
> > managed,
> > > > > > and
> > > > > > > > > > hardened
> > > > > > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > > > > > https://github.com/astronomer/airflow-chart) that is
> > being
> > > > > used
> > > > > > by
> > > > > > > > > both
> > > > > > > > > > our
> > > > > > > > > > > SaaS and Enterprise customers. This chart is
> > battle-tested and
> > > > > > > running
> > > > > > > > > > > hundreds of Airflow deployments of varying sizes and
> > runtime
> > > > > > > > > > environments.
> > > > > > > > > > > It’s been built up to encapsulate the issues that Airflow
> > > > > users
> > > > > > run
> > > > > > > > > into
> > > > > > > > > > in
> > > > > > > > > > > the real world.
> > > > > > > > > > >
> > > > > > > > > > > While this chart was originally developed internally for
> > our
> > > > > > > Astronomer
> > > > > > > > > > > Platform, we’ve recently decoupled the chart from the
> > rest of
> > > > > our
> > > > > > > > > > platform
> > > > > > > > > > > to make it usable by the greater Airflow community. With
> > these
> > > > > > > changes
> > > > > > > > > in
> > > > > > > > > > > mind, we want to start a conversation about donating this
> > > > > chart
> > > > > > to
> > > > > > > the
> > > > > > > > > > > Airflow community.
> > > > > > > > > > >
> > > > > > > > > > > Some of the main features of the chart are:
> > > > > > > > > > >
> > > > > > > > > > > - It works out of the box. With zero configuration, a user
> > > > > will
> > > > > > get
> > > > > > > > > a
> > > > > > > > > > > postgres database, a default user and the
> > KubernetesExecutor
> > > > > > ready
> > > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > DAGs.
> > > > > > > > > > > - Support for Local, Celery (w/ optional KEDA
> > autoscaling) and
> > > > > > > > > > > Kubernetes executors.
> > > > > > > > > > >
> > > > > > > > > > > Support for optional pgbouncer. We use this to share a
> > > > > > configurable
> > > > > > > > > > > connection pool size per deployment. Useful for limiting
> > > > > > > connections to
> > > > > > > > > > the
> > > > > > > > > > > metadata database.
> > > > > > > > > > >
> > > > > > > > > > > - Airflow migration support. A user can push a newer
> > version
> > > > > of
> > > > > > > > > > Airflow
> > > > > > > > > > > into an existing release and migrations will
> > automatically run
> > > > > > > > > > cleanly.
> > > > > > > > > > > - Prometheus support. Optionally install and configure a
> > > > > > > > > > statsd-exporter
> > > > > > > > > > > to ingest Airflow metrics and expose them to Prometheus
> > > > > > > > > automatically.
> > > > > > > > > > > - Resource control. Optionally control the ResourceQuotas
> > and
> > > > > > > > > > > LimitRanges for each deployment so that no deployment can
> > > > > > overload
> > > > > > > a
> > > > > > > > > > > cluster.
> > > > > > > > > > > - Simple optional Elasticsearch support.
> > > > > > > > > > > - Optional namespace cleanup. Sometimes
> > KubernetesExecutor and
> > > > > > > > > > > KubernetesPodOperator pods fail for reasons other than the
> > > > > actual
> > > > > > > > > > task.
> > > > > > > > > > > This feature helps keep things clean in Kubernetes.
> > > > > > > > > > > - Support for running locally in KIND (Kubernetes in
> > Docker).
> > > > > > > > > > > - Automatically tested across many Kubernetes versions
> > with
> > > > > Helm
> > > > > > 2
> > > > > > > > > > and 3
> > > > > > > > > > > support.
> > > > > > > > > > >
> > > > > > > > > > > We’ve found that the cleanest and most reliable way to
> > deploy
> > > > > > DAGs
> > > > > > > to
> > > > > > > > > > > Kubernetes and manage them at scale is to package them
> > into
> > > > > the
> > > > > > > actual
> > > > > > > > > > > docker image, so we have geared this chart towards that
> > > > > method of
> > > > > > > > > > > operation, though adding other methods should be
> > > > > straightforward.
> > > > > > > > > > >
> > > > > > > > > > > We would love thoughts from the community and would love
> > to
> > > > > see
> > > > > > > this
> > > > > > > > > > chart
> > > > > > > > > > > help others to get up and running on Kubernetes!
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Jarek Potiuk
> > > > > > > > Polidea <https://www.polidea.com/> | Principal Software
> > Engineer
> > > > > > > >
> > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > <+48%20660%20796%20129>>
> > > > > > > > [image: Polidea] <https://www.polidea.com/>
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Jarek Potiuk
> > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > > >
> > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > > > > <+48%20660%20796%20129>>
> > > > > [image: Polidea] <https://www.polidea.com/>
> > > > >
> > > >
> > > >
> > > > --
> > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > >
> >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea | Principal Software Engineer
> >
> > M: +48 660 796 129
> >
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Greg Neiheisel <gr...@astronomer.io.INVALID>.
Yep, we can absolutely pull it into the airflow repo. We've also been
building up a test suite that currently runs on CircleCI and uses kind
(Kubernetes in Docker) to test several kubernetes versions with some
different settings. Right now we're mostly testing the different executors
since that has the biggest impact on what gets deployed, but that can be
expanded.
What CRDs are currently being used to run Airflow for the tests?
On Wed, Mar 25, 2020 at 11:06 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> One thing for the donation.
>
> Did you you want to have separate repository Greg ?
>
> I think we should simply create a folder in Airflow repo and keep it
> there (similarly as we keep Dockerfile). I am going to switch our
> Kubernetes Tests to the production image (will make the tests much
> faster) and I am going to test the Dockerfile automatically in CI -
> for now we are using some custom Resource definitions to start Airflow
> on Kubernetes Cluster for the tests, but we could switch to using the
> helm chart - this way we can test all three things at once:
> - Kubernetes Executor
> - Dockerfile
> - Helm Chart
> and we could also add more tests - for example testing different
> deployment options for the helm chart.
>
> Having the Helm chart in Airflow repo would help with that -
> especially in terms of future changes and testing them automatically.
>
> J.
>
> On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
> <ai...@apache.org> wrote:
> >
> > +1 on the donation. Always happy to see more useful stuff for the
> > community :)
> >
> > On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io>
> wrote:
> >
> > > Yep, the cleanup_pods script is set up now as an optional Kubernetes
> > > CronJob (
> > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml
> )
> > > that we have run periodically to clean failed pods up and could stay
> > > separate.
> > >
> > > The wait_for_migrations script could definitely be pulled into Airflow.
> > > For context, we deploy an initContainer on the scheduler (
> > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84
> )
> > > that runs the upgradedb command before booting the scheduler. This new
> > > wait_for_migration script runs in an initContainer on the webserver and
> > > workers (
> > >
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65
> )
> > > so that they don't boot up ahead of a potentially long-running
> migration
> > > and attempt to operate on new or missing columns/tables before the
> > > migrations run. This prevents these pods from entering a CrashLoop.
> > >
> > > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <
> Jarek.Potiuk@polidea.com>
> > > wrote:
> > >
> > >> >
> > >> > @Tomasz great question. Our images are currently generated from
> > >> Dockerfiles
> > >> > in this repo https://github.com/astronomer/ap-airflow and get
> > >> published to
> > >> > DockerHub
> > >> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> > >> >
> > >> > For the most part those are typical Airflow images. There's an
> > >> entrypoint
> > >> > script that we include in the image that handles waiting for the
> > >> database
> > >> > and redis (if used) to come up, which is pretty generic.
> > >>
> > >>
> > >> I already added waiting for the database (both metadata and celery
> URL) in
> > >> the PR:
> > >>
> > >>
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> > >> .
> > >> It's functionally the same but more generic.
> > >>
> > >> The only other
> > >> > thing that I think the Helm Chart uses would be the scripts in this
> repo
> > >> > https://github.com/astronomer/astronomer-airflow-scripts. Our
> > >> Dockerfiles
> > >> > pull this package in. These scripts are used to coordinate running
> > >> > migrations and cleaning up failed pods.
> > >> >
> > >>
> > >> I see two scripts:
> > >>
> > >> * cleanup_pods -> this is (I believe) not needed to run in airflow -
> this
> > >> could be run as a separate pod/container?
> > >> * waiting for migrations -> I think this is a good candidate to add
> > >> *airflow
> > >> db wait_for_migration* command and make it part of airflow itself.
> > >>
> > >> I think we also have to agree on the Airflow version supported by the
> > >> official helm chart. I'd suggest we support 1.10.10+ and we
> incorporate
> > >> all
> > >> the changes needed to airflow (like the "db wait_for_migration")
> into 2.0
> > >> and 1.10 and we support both - image and helm chart for those versions
> > >> only. That would help with people migrating to the latest version.
> > >>
> > >> WDYT?
> > >>
> > >>
> > >> > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > >> > daniel.imberman@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > @jarek I agree completely. I think that pairing an official helm
> chart
> > >> > > with the official image would make for a REALLY powerful “up and
> > >> running
> > >> > > with airflow” story :). Tomek and I have also been looking into
> > >> > > operator-sdk which has the ability to create custom controllers
> from
> > >> helm
> > >> > > charts. We might even able to get a 1-2 punch from the same code
> base
> > >> :).
> > >> > >
> > >> > > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please
> start
> > >> > the
> > >> > > process of donation?
> > >> > >
> > >> > > +1 on my part, of course :)
> > >> > >
> > >> > >
> > >> > >
> > >> > > Daniel
> > >> > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> > >> Jarek.Potiuk@polidea.com>,
> > >> > > wrote:
> > >> > > > +1. And it should be paired with the official image we have
> work in
> > >> > > > progress on. I looked a lot at the Astronomer's image while
> > >> preparing
> > >> > my
> > >> > > > draft and we can make any adjustments needed to make it works
> with
> > >> the
> > >> > > helm
> > >> > > > chart - and I am super happy to collaborate on that.
> > >> > > >
> > >> > > > PR here: https://github.com/apache/airflow/pull/7832
> > >> > > >
> > >> > > > J.
> > >> > > >
> > >> > > >
> > >> > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <kaxilnaik@gmail.com
> >
> > >> > wrote:
> > >> > > >
> > >> > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > >> > > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> > >> > > > >
> > >> > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > >> > turbaszek@apache.org>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > An official helm chart is something our community needs!
> Using
> > >> your
> > >> > > > > > chart as the official makes a lot of sens to me because as
> you
> > >> > > > > > mentioned - it's battle tested.
> > >> > > > > >
> > >> > > > > > One question: what Airflow image do you use? Also, would you
> > >> mind
> > >> > > > > > sharing a link to the chart?
> > >> > > > > >
> > >> > > > > > Tomek
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > >> > > > > > <gr...@astronomer.io.invalid> wrote:
> > >> > > > > > >
> > >> > > > > > > Hey everyone,
> > >> > > > > > >
> > >> > > > > > > Over the past few years at Astronomer, we’ve created,
> managed,
> > >> > and
> > >> > > > > > hardened
> > >> > > > > > > a production-ready Helm Chart for Airflow (
> > >> > > > > > > https://github.com/astronomer/airflow-chart) that is
> being
> > >> used
> > >> > by
> > >> > > > > both
> > >> > > > > > our
> > >> > > > > > > SaaS and Enterprise customers. This chart is
> battle-tested and
> > >> > > running
> > >> > > > > > > hundreds of Airflow deployments of varying sizes and
> runtime
> > >> > > > > > environments.
> > >> > > > > > > It’s been built up to encapsulate the issues that Airflow
> > >> users
> > >> > run
> > >> > > > > into
> > >> > > > > > in
> > >> > > > > > > the real world.
> > >> > > > > > >
> > >> > > > > > > While this chart was originally developed internally for
> our
> > >> > > Astronomer
> > >> > > > > > > Platform, we’ve recently decoupled the chart from the
> rest of
> > >> our
> > >> > > > > > platform
> > >> > > > > > > to make it usable by the greater Airflow community. With
> these
> > >> > > changes
> > >> > > > > in
> > >> > > > > > > mind, we want to start a conversation about donating this
> > >> chart
> > >> > to
> > >> > > the
> > >> > > > > > > Airflow community.
> > >> > > > > > >
> > >> > > > > > > Some of the main features of the chart are:
> > >> > > > > > >
> > >> > > > > > > - It works out of the box. With zero configuration, a user
> > >> will
> > >> > get
> > >> > > > > a
> > >> > > > > > > postgres database, a default user and the
> KubernetesExecutor
> > >> > ready
> > >> > > > > to
> > >> > > > > > run
> > >> > > > > > > DAGs.
> > >> > > > > > > - Support for Local, Celery (w/ optional KEDA
> autoscaling) and
> > >> > > > > > > Kubernetes executors.
> > >> > > > > > >
> > >> > > > > > > Support for optional pgbouncer. We use this to share a
> > >> > configurable
> > >> > > > > > > connection pool size per deployment. Useful for limiting
> > >> > > connections to
> > >> > > > > > the
> > >> > > > > > > metadata database.
> > >> > > > > > >
> > >> > > > > > > - Airflow migration support. A user can push a newer
> version
> > >> of
> > >> > > > > > Airflow
> > >> > > > > > > into an existing release and migrations will
> automatically run
> > >> > > > > > cleanly.
> > >> > > > > > > - Prometheus support. Optionally install and configure a
> > >> > > > > > statsd-exporter
> > >> > > > > > > to ingest Airflow metrics and expose them to Prometheus
> > >> > > > > automatically.
> > >> > > > > > > - Resource control. Optionally control the ResourceQuotas
> and
> > >> > > > > > > LimitRanges for each deployment so that no deployment can
> > >> > overload
> > >> > > a
> > >> > > > > > > cluster.
> > >> > > > > > > - Simple optional Elasticsearch support.
> > >> > > > > > > - Optional namespace cleanup. Sometimes
> KubernetesExecutor and
> > >> > > > > > > KubernetesPodOperator pods fail for reasons other than the
> > >> actual
> > >> > > > > > task.
> > >> > > > > > > This feature helps keep things clean in Kubernetes.
> > >> > > > > > > - Support for running locally in KIND (Kubernetes in
> Docker).
> > >> > > > > > > - Automatically tested across many Kubernetes versions
> with
> > >> Helm
> > >> > 2
> > >> > > > > > and 3
> > >> > > > > > > support.
> > >> > > > > > >
> > >> > > > > > > We’ve found that the cleanest and most reliable way to
> deploy
> > >> > DAGs
> > >> > > to
> > >> > > > > > > Kubernetes and manage them at scale is to package them
> into
> > >> the
> > >> > > actual
> > >> > > > > > > docker image, so we have geared this chart towards that
> > >> method of
> > >> > > > > > > operation, though adding other methods should be
> > >> straightforward.
> > >> > > > > > >
> > >> > > > > > > We would love thoughts from the community and would love
> to
> > >> see
> > >> > > this
> > >> > > > > > chart
> > >> > > > > > > help others to get up and running on Kubernetes!
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > >
> > >> > > > Jarek Potiuk
> > >> > > > Polidea <https://www.polidea.com/> | Principal Software
> Engineer
> > >> > > >
> > >> > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > >> <+48%20660%20796%20129>>
> > >> > > > [image: Polidea] <https://www.polidea.com/>
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >> >
> > >>
> > >>
> > >> --
> > >>
> > >> Jarek Potiuk
> > >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >>
> > >> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> > >> <+48%20660%20796%20129>>
> > >> [image: Polidea] <https://www.polidea.com/>
> > >>
> > >
> > >
> > > --
> > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >
>
>
>
> --
>
> Jarek Potiuk
> Polidea | Principal Software Engineer
>
> M: +48 660 796 129
>
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
One thing for the donation.
Did you you want to have separate repository Greg ?
I think we should simply create a folder in Airflow repo and keep it
there (similarly as we keep Dockerfile). I am going to switch our
Kubernetes Tests to the production image (will make the tests much
faster) and I am going to test the Dockerfile automatically in CI -
for now we are using some custom Resource definitions to start Airflow
on Kubernetes Cluster for the tests, but we could switch to using the
helm chart - this way we can test all three things at once:
- Kubernetes Executor
- Dockerfile
- Helm Chart
and we could also add more tests - for example testing different
deployment options for the helm chart.
Having the Helm chart in Airflow repo would help with that -
especially in terms of future changes and testing them automatically.
J.
On Tue, Mar 24, 2020 at 9:09 PM Aizhamal Nurmamat kyzy
<ai...@apache.org> wrote:
>
> +1 on the donation. Always happy to see more useful stuff for the
> community :)
>
> On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io> wrote:
>
> > Yep, the cleanup_pods script is set up now as an optional Kubernetes
> > CronJob (
> > https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml)
> > that we have run periodically to clean failed pods up and could stay
> > separate.
> >
> > The wait_for_migrations script could definitely be pulled into Airflow.
> > For context, we deploy an initContainer on the scheduler (
> > https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84)
> > that runs the upgradedb command before booting the scheduler. This new
> > wait_for_migration script runs in an initContainer on the webserver and
> > workers (
> > https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65)
> > so that they don't boot up ahead of a potentially long-running migration
> > and attempt to operate on new or missing columns/tables before the
> > migrations run. This prevents these pods from entering a CrashLoop.
> >
> > On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <Ja...@polidea.com>
> > wrote:
> >
> >> >
> >> > @Tomasz great question. Our images are currently generated from
> >> Dockerfiles
> >> > in this repo https://github.com/astronomer/ap-airflow and get
> >> published to
> >> > DockerHub
> >> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> >> >
> >> > For the most part those are typical Airflow images. There's an
> >> entrypoint
> >> > script that we include in the image that handles waiting for the
> >> database
> >> > and redis (if used) to come up, which is pretty generic.
> >>
> >>
> >> I already added waiting for the database (both metadata and celery URL) in
> >> the PR:
> >>
> >> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> >> .
> >> It's functionally the same but more generic.
> >>
> >> The only other
> >> > thing that I think the Helm Chart uses would be the scripts in this repo
> >> > https://github.com/astronomer/astronomer-airflow-scripts. Our
> >> Dockerfiles
> >> > pull this package in. These scripts are used to coordinate running
> >> > migrations and cleaning up failed pods.
> >> >
> >>
> >> I see two scripts:
> >>
> >> * cleanup_pods -> this is (I believe) not needed to run in airflow - this
> >> could be run as a separate pod/container?
> >> * waiting for migrations -> I think this is a good candidate to add
> >> *airflow
> >> db wait_for_migration* command and make it part of airflow itself.
> >>
> >> I think we also have to agree on the Airflow version supported by the
> >> official helm chart. I'd suggest we support 1.10.10+ and we incorporate
> >> all
> >> the changes needed to airflow (like the "db wait_for_migration") into 2.0
> >> and 1.10 and we support both - image and helm chart for those versions
> >> only. That would help with people migrating to the latest version.
> >>
> >> WDYT?
> >>
> >>
> >> > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> >> > daniel.imberman@gmail.com>
> >> > wrote:
> >> >
> >> > > @jarek I agree completely. I think that pairing an official helm chart
> >> > > with the official image would make for a REALLY powerful “up and
> >> running
> >> > > with airflow” story :). Tomek and I have also been looking into
> >> > > operator-sdk which has the ability to create custom controllers from
> >> helm
> >> > > charts. We might even able to get a 1-2 punch from the same code base
> >> :).
> >> > >
> >> > > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please start
> >> > the
> >> > > process of donation?
> >> > >
> >> > > +1 on my part, of course :)
> >> > >
> >> > >
> >> > >
> >> > > Daniel
> >> > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
> >> Jarek.Potiuk@polidea.com>,
> >> > > wrote:
> >> > > > +1. And it should be paired with the official image we have work in
> >> > > > progress on. I looked a lot at the Astronomer's image while
> >> preparing
> >> > my
> >> > > > draft and we can make any adjustments needed to make it works with
> >> the
> >> > > helm
> >> > > > chart - and I am super happy to collaborate on that.
> >> > > >
> >> > > > PR here: https://github.com/apache/airflow/pull/7832
> >> > > >
> >> > > > J.
> >> > > >
> >> > > >
> >> > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com>
> >> > wrote:
> >> > > >
> >> > > > > @Tomasz Urbaszek <to...@polidea.com> :
> >> > > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> >> > > > >
> >> > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> >> > turbaszek@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > An official helm chart is something our community needs! Using
> >> your
> >> > > > > > chart as the official makes a lot of sens to me because as you
> >> > > > > > mentioned - it's battle tested.
> >> > > > > >
> >> > > > > > One question: what Airflow image do you use? Also, would you
> >> mind
> >> > > > > > sharing a link to the chart?
> >> > > > > >
> >> > > > > > Tomek
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> >> > > > > > <gr...@astronomer.io.invalid> wrote:
> >> > > > > > >
> >> > > > > > > Hey everyone,
> >> > > > > > >
> >> > > > > > > Over the past few years at Astronomer, we’ve created, managed,
> >> > and
> >> > > > > > hardened
> >> > > > > > > a production-ready Helm Chart for Airflow (
> >> > > > > > > https://github.com/astronomer/airflow-chart) that is being
> >> used
> >> > by
> >> > > > > both
> >> > > > > > our
> >> > > > > > > SaaS and Enterprise customers. This chart is battle-tested and
> >> > > running
> >> > > > > > > hundreds of Airflow deployments of varying sizes and runtime
> >> > > > > > environments.
> >> > > > > > > It’s been built up to encapsulate the issues that Airflow
> >> users
> >> > run
> >> > > > > into
> >> > > > > > in
> >> > > > > > > the real world.
> >> > > > > > >
> >> > > > > > > While this chart was originally developed internally for our
> >> > > Astronomer
> >> > > > > > > Platform, we’ve recently decoupled the chart from the rest of
> >> our
> >> > > > > > platform
> >> > > > > > > to make it usable by the greater Airflow community. With these
> >> > > changes
> >> > > > > in
> >> > > > > > > mind, we want to start a conversation about donating this
> >> chart
> >> > to
> >> > > the
> >> > > > > > > Airflow community.
> >> > > > > > >
> >> > > > > > > Some of the main features of the chart are:
> >> > > > > > >
> >> > > > > > > - It works out of the box. With zero configuration, a user
> >> will
> >> > get
> >> > > > > a
> >> > > > > > > postgres database, a default user and the KubernetesExecutor
> >> > ready
> >> > > > > to
> >> > > > > > run
> >> > > > > > > DAGs.
> >> > > > > > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> >> > > > > > > Kubernetes executors.
> >> > > > > > >
> >> > > > > > > Support for optional pgbouncer. We use this to share a
> >> > configurable
> >> > > > > > > connection pool size per deployment. Useful for limiting
> >> > > connections to
> >> > > > > > the
> >> > > > > > > metadata database.
> >> > > > > > >
> >> > > > > > > - Airflow migration support. A user can push a newer version
> >> of
> >> > > > > > Airflow
> >> > > > > > > into an existing release and migrations will automatically run
> >> > > > > > cleanly.
> >> > > > > > > - Prometheus support. Optionally install and configure a
> >> > > > > > statsd-exporter
> >> > > > > > > to ingest Airflow metrics and expose them to Prometheus
> >> > > > > automatically.
> >> > > > > > > - Resource control. Optionally control the ResourceQuotas and
> >> > > > > > > LimitRanges for each deployment so that no deployment can
> >> > overload
> >> > > a
> >> > > > > > > cluster.
> >> > > > > > > - Simple optional Elasticsearch support.
> >> > > > > > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> >> > > > > > > KubernetesPodOperator pods fail for reasons other than the
> >> actual
> >> > > > > > task.
> >> > > > > > > This feature helps keep things clean in Kubernetes.
> >> > > > > > > - Support for running locally in KIND (Kubernetes in Docker).
> >> > > > > > > - Automatically tested across many Kubernetes versions with
> >> Helm
> >> > 2
> >> > > > > > and 3
> >> > > > > > > support.
> >> > > > > > >
> >> > > > > > > We’ve found that the cleanest and most reliable way to deploy
> >> > DAGs
> >> > > to
> >> > > > > > > Kubernetes and manage them at scale is to package them into
> >> the
> >> > > actual
> >> > > > > > > docker image, so we have geared this chart towards that
> >> method of
> >> > > > > > > operation, though adding other methods should be
> >> straightforward.
> >> > > > > > >
> >> > > > > > > We would love thoughts from the community and would love to
> >> see
> >> > > this
> >> > > > > > chart
> >> > > > > > > help others to get up and running on Kubernetes!
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Jarek Potiuk
> >> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >> > > >
> >> > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> >> <+48%20660%20796%20129>>
> >> > > > [image: Polidea] <https://www.polidea.com/>
> >> > >
> >> >
> >> >
> >> > --
> >> > *Greg Neiheisel* / Chief Architect Astronomer.io
> >> >
> >>
> >>
> >> --
> >>
> >> Jarek Potiuk
> >> Polidea <https://www.polidea.com/> | Principal Software Engineer
> >>
> >> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
> >> <+48%20660%20796%20129>>
> >> [image: Polidea] <https://www.polidea.com/>
> >>
> >
> >
> > --
> > *Greg Neiheisel* / Chief Architect Astronomer.io
> >
--
Jarek Potiuk
Polidea | Principal Software Engineer
M: +48 660 796 129
Re: Helm Chart for Airflow
Posted by Aizhamal Nurmamat kyzy <ai...@apache.org>.
+1 on the donation. Always happy to see more useful stuff for the
community :)
On Tue, Mar 24, 2020 at 9:20 AM Greg Neiheisel <gr...@astronomer.io> wrote:
> Yep, the cleanup_pods script is set up now as an optional Kubernetes
> CronJob (
> https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml)
> that we have run periodically to clean failed pods up and could stay
> separate.
>
> The wait_for_migrations script could definitely be pulled into Airflow.
> For context, we deploy an initContainer on the scheduler (
> https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84)
> that runs the upgradedb command before booting the scheduler. This new
> wait_for_migration script runs in an initContainer on the webserver and
> workers (
> https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65)
> so that they don't boot up ahead of a potentially long-running migration
> and attempt to operate on new or missing columns/tables before the
> migrations run. This prevents these pods from entering a CrashLoop.
>
> On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <Ja...@polidea.com>
> wrote:
>
>> >
>> > @Tomasz great question. Our images are currently generated from
>> Dockerfiles
>> > in this repo https://github.com/astronomer/ap-airflow and get
>> published to
>> > DockerHub
>> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>> >
>> > For the most part those are typical Airflow images. There's an
>> entrypoint
>> > script that we include in the image that handles waiting for the
>> database
>> > and redis (if used) to come up, which is pretty generic.
>>
>>
>> I already added waiting for the database (both metadata and celery URL) in
>> the PR:
>>
>> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
>> .
>> It's functionally the same but more generic.
>>
>> The only other
>> > thing that I think the Helm Chart uses would be the scripts in this repo
>> > https://github.com/astronomer/astronomer-airflow-scripts. Our
>> Dockerfiles
>> > pull this package in. These scripts are used to coordinate running
>> > migrations and cleaning up failed pods.
>> >
>>
>> I see two scripts:
>>
>> * cleanup_pods -> this is (I believe) not needed to run in airflow - this
>> could be run as a separate pod/container?
>> * waiting for migrations -> I think this is a good candidate to add
>> *airflow
>> db wait_for_migration* command and make it part of airflow itself.
>>
>> I think we also have to agree on the Airflow version supported by the
>> official helm chart. I'd suggest we support 1.10.10+ and we incorporate
>> all
>> the changes needed to airflow (like the "db wait_for_migration") into 2.0
>> and 1.10 and we support both - image and helm chart for those versions
>> only. That would help with people migrating to the latest version.
>>
>> WDYT?
>>
>>
>> > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
>> > daniel.imberman@gmail.com>
>> > wrote:
>> >
>> > > @jarek I agree completely. I think that pairing an official helm chart
>> > > with the official image would make for a REALLY powerful “up and
>> running
>> > > with airflow” story :). Tomek and I have also been looking into
>> > > operator-sdk which has the ability to create custom controllers from
>> helm
>> > > charts. We might even able to get a 1-2 punch from the same code base
>> :).
>> > >
>> > > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please start
>> > the
>> > > process of donation?
>> > >
>> > > +1 on my part, of course :)
>> > >
>> > >
>> > >
>> > > Daniel
>> > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <
>> Jarek.Potiuk@polidea.com>,
>> > > wrote:
>> > > > +1. And it should be paired with the official image we have work in
>> > > > progress on. I looked a lot at the Astronomer's image while
>> preparing
>> > my
>> > > > draft and we can make any adjustments needed to make it works with
>> the
>> > > helm
>> > > > chart - and I am super happy to collaborate on that.
>> > > >
>> > > > PR here: https://github.com/apache/airflow/pull/7832
>> > > >
>> > > > J.
>> > > >
>> > > >
>> > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com>
>> > wrote:
>> > > >
>> > > > > @Tomasz Urbaszek <to...@polidea.com> :
>> > > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
>> > > > >
>> > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
>> > turbaszek@apache.org>
>> > > > > wrote:
>> > > > >
>> > > > > > An official helm chart is something our community needs! Using
>> your
>> > > > > > chart as the official makes a lot of sens to me because as you
>> > > > > > mentioned - it's battle tested.
>> > > > > >
>> > > > > > One question: what Airflow image do you use? Also, would you
>> mind
>> > > > > > sharing a link to the chart?
>> > > > > >
>> > > > > > Tomek
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
>> > > > > > <gr...@astronomer.io.invalid> wrote:
>> > > > > > >
>> > > > > > > Hey everyone,
>> > > > > > >
>> > > > > > > Over the past few years at Astronomer, we’ve created, managed,
>> > and
>> > > > > > hardened
>> > > > > > > a production-ready Helm Chart for Airflow (
>> > > > > > > https://github.com/astronomer/airflow-chart) that is being
>> used
>> > by
>> > > > > both
>> > > > > > our
>> > > > > > > SaaS and Enterprise customers. This chart is battle-tested and
>> > > running
>> > > > > > > hundreds of Airflow deployments of varying sizes and runtime
>> > > > > > environments.
>> > > > > > > It’s been built up to encapsulate the issues that Airflow
>> users
>> > run
>> > > > > into
>> > > > > > in
>> > > > > > > the real world.
>> > > > > > >
>> > > > > > > While this chart was originally developed internally for our
>> > > Astronomer
>> > > > > > > Platform, we’ve recently decoupled the chart from the rest of
>> our
>> > > > > > platform
>> > > > > > > to make it usable by the greater Airflow community. With these
>> > > changes
>> > > > > in
>> > > > > > > mind, we want to start a conversation about donating this
>> chart
>> > to
>> > > the
>> > > > > > > Airflow community.
>> > > > > > >
>> > > > > > > Some of the main features of the chart are:
>> > > > > > >
>> > > > > > > - It works out of the box. With zero configuration, a user
>> will
>> > get
>> > > > > a
>> > > > > > > postgres database, a default user and the KubernetesExecutor
>> > ready
>> > > > > to
>> > > > > > run
>> > > > > > > DAGs.
>> > > > > > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
>> > > > > > > Kubernetes executors.
>> > > > > > >
>> > > > > > > Support for optional pgbouncer. We use this to share a
>> > configurable
>> > > > > > > connection pool size per deployment. Useful for limiting
>> > > connections to
>> > > > > > the
>> > > > > > > metadata database.
>> > > > > > >
>> > > > > > > - Airflow migration support. A user can push a newer version
>> of
>> > > > > > Airflow
>> > > > > > > into an existing release and migrations will automatically run
>> > > > > > cleanly.
>> > > > > > > - Prometheus support. Optionally install and configure a
>> > > > > > statsd-exporter
>> > > > > > > to ingest Airflow metrics and expose them to Prometheus
>> > > > > automatically.
>> > > > > > > - Resource control. Optionally control the ResourceQuotas and
>> > > > > > > LimitRanges for each deployment so that no deployment can
>> > overload
>> > > a
>> > > > > > > cluster.
>> > > > > > > - Simple optional Elasticsearch support.
>> > > > > > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
>> > > > > > > KubernetesPodOperator pods fail for reasons other than the
>> actual
>> > > > > > task.
>> > > > > > > This feature helps keep things clean in Kubernetes.
>> > > > > > > - Support for running locally in KIND (Kubernetes in Docker).
>> > > > > > > - Automatically tested across many Kubernetes versions with
>> Helm
>> > 2
>> > > > > > and 3
>> > > > > > > support.
>> > > > > > >
>> > > > > > > We’ve found that the cleanest and most reliable way to deploy
>> > DAGs
>> > > to
>> > > > > > > Kubernetes and manage them at scale is to package them into
>> the
>> > > actual
>> > > > > > > docker image, so we have geared this chart towards that
>> method of
>> > > > > > > operation, though adding other methods should be
>> straightforward.
>> > > > > > >
>> > > > > > > We would love thoughts from the community and would love to
>> see
>> > > this
>> > > > > > chart
>> > > > > > > help others to get up and running on Kubernetes!
>> > > > > > >
>> > > > > > > --
>> > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > >
>> > > > Jarek Potiuk
>> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
>> > > >
>> > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
>> <+48%20660%20796%20129>>
>> > > > [image: Polidea] <https://www.polidea.com/>
>> > >
>> >
>> >
>> > --
>> > *Greg Neiheisel* / Chief Architect Astronomer.io
>> >
>>
>>
>> --
>>
>> Jarek Potiuk
>> Polidea <https://www.polidea.com/> | Principal Software Engineer
>>
>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129
>> <+48%20660%20796%20129>>
>> [image: Polidea] <https://www.polidea.com/>
>>
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
Re: Helm Chart for Airflow
Posted by Greg Neiheisel <gr...@astronomer.io.INVALID>.
Yep, the cleanup_pods script is set up now as an optional Kubernetes
CronJob (
https://github.com/astronomer/airflow-chart/blob/master/templates/cleanup/cleanup-cronjob.yaml)
that we have run periodically to clean failed pods up and could stay
separate.
The wait_for_migrations script could definitely be pulled into Airflow. For
context, we deploy an initContainer on the scheduler (
https://github.com/astronomer/airflow-chart/blob/master/templates/scheduler/scheduler-deployment.yaml#L77-L84)
that runs the upgradedb command before booting the scheduler. This new
wait_for_migration script runs in an initContainer on the webserver and
workers (
https://github.com/astronomer/airflow-chart/blob/master/templates/webserver/webserver-deployment.yaml#L58-L65)
so that they don't boot up ahead of a potentially long-running migration
and attempt to operate on new or missing columns/tables before the
migrations run. This prevents these pods from entering a CrashLoop.
On Tue, Mar 24, 2020 at 11:48 AM Jarek Potiuk <Ja...@polidea.com>
wrote:
> >
> > @Tomasz great question. Our images are currently generated from
> Dockerfiles
> > in this repo https://github.com/astronomer/ap-airflow and get published
> to
> > DockerHub
> > https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
> >
> > For the most part those are typical Airflow images. There's an entrypoint
> > script that we include in the image that handles waiting for the database
> > and redis (if used) to come up, which is pretty generic.
>
>
> I already added waiting for the database (both metadata and celery URL) in
> the PR:
>
> https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
> .
> It's functionally the same but more generic.
>
> The only other
> > thing that I think the Helm Chart uses would be the scripts in this repo
> > https://github.com/astronomer/astronomer-airflow-scripts. Our
> Dockerfiles
> > pull this package in. These scripts are used to coordinate running
> > migrations and cleaning up failed pods.
> >
>
> I see two scripts:
>
> * cleanup_pods -> this is (I believe) not needed to run in airflow - this
> could be run as a separate pod/container?
> * waiting for migrations -> I think this is a good candidate to add
> *airflow
> db wait_for_migration* command and make it part of airflow itself.
>
> I think we also have to agree on the Airflow version supported by the
> official helm chart. I'd suggest we support 1.10.10+ and we incorporate all
> the changes needed to airflow (like the "db wait_for_migration") into 2.0
> and 1.10 and we support both - image and helm chart for those versions
> only. That would help with people migrating to the latest version.
>
> WDYT?
>
>
> > On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> > daniel.imberman@gmail.com>
> > wrote:
> >
> > > @jarek I agree completely. I think that pairing an official helm chart
> > > with the official image would make for a REALLY powerful “up and
> running
> > > with airflow” story :). Tomek and I have also been looking into
> > > operator-sdk which has the ability to create custom controllers from
> helm
> > > charts. We might even able to get a 1-2 punch from the same code base
> :).
> > >
> > > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please start
> > the
> > > process of donation?
> > >
> > > +1 on my part, of course :)
> > >
> > >
> > >
> > > Daniel
> > > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <Jarek.Potiuk@polidea.com
> >,
> > > wrote:
> > > > +1. And it should be paired with the official image we have work in
> > > > progress on. I looked a lot at the Astronomer's image while preparing
> > my
> > > > draft and we can make any adjustments needed to make it works with
> the
> > > helm
> > > > chart - and I am super happy to collaborate on that.
> > > >
> > > > PR here: https://github.com/apache/airflow/pull/7832
> > > >
> > > > J.
> > > >
> > > >
> > > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com>
> > wrote:
> > > >
> > > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> > > > >
> > > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> > turbaszek@apache.org>
> > > > > wrote:
> > > > >
> > > > > > An official helm chart is something our community needs! Using
> your
> > > > > > chart as the official makes a lot of sens to me because as you
> > > > > > mentioned - it's battle tested.
> > > > > >
> > > > > > One question: what Airflow image do you use? Also, would you mind
> > > > > > sharing a link to the chart?
> > > > > >
> > > > > > Tomek
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > > >
> > > > > > > Hey everyone,
> > > > > > >
> > > > > > > Over the past few years at Astronomer, we’ve created, managed,
> > and
> > > > > > hardened
> > > > > > > a production-ready Helm Chart for Airflow (
> > > > > > > https://github.com/astronomer/airflow-chart) that is being
> used
> > by
> > > > > both
> > > > > > our
> > > > > > > SaaS and Enterprise customers. This chart is battle-tested and
> > > running
> > > > > > > hundreds of Airflow deployments of varying sizes and runtime
> > > > > > environments.
> > > > > > > It’s been built up to encapsulate the issues that Airflow users
> > run
> > > > > into
> > > > > > in
> > > > > > > the real world.
> > > > > > >
> > > > > > > While this chart was originally developed internally for our
> > > Astronomer
> > > > > > > Platform, we’ve recently decoupled the chart from the rest of
> our
> > > > > > platform
> > > > > > > to make it usable by the greater Airflow community. With these
> > > changes
> > > > > in
> > > > > > > mind, we want to start a conversation about donating this chart
> > to
> > > the
> > > > > > > Airflow community.
> > > > > > >
> > > > > > > Some of the main features of the chart are:
> > > > > > >
> > > > > > > - It works out of the box. With zero configuration, a user will
> > get
> > > > > a
> > > > > > > postgres database, a default user and the KubernetesExecutor
> > ready
> > > > > to
> > > > > > run
> > > > > > > DAGs.
> > > > > > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> > > > > > > Kubernetes executors.
> > > > > > >
> > > > > > > Support for optional pgbouncer. We use this to share a
> > configurable
> > > > > > > connection pool size per deployment. Useful for limiting
> > > connections to
> > > > > > the
> > > > > > > metadata database.
> > > > > > >
> > > > > > > - Airflow migration support. A user can push a newer version of
> > > > > > Airflow
> > > > > > > into an existing release and migrations will automatically run
> > > > > > cleanly.
> > > > > > > - Prometheus support. Optionally install and configure a
> > > > > > statsd-exporter
> > > > > > > to ingest Airflow metrics and expose them to Prometheus
> > > > > automatically.
> > > > > > > - Resource control. Optionally control the ResourceQuotas and
> > > > > > > LimitRanges for each deployment so that no deployment can
> > overload
> > > a
> > > > > > > cluster.
> > > > > > > - Simple optional Elasticsearch support.
> > > > > > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> > > > > > > KubernetesPodOperator pods fail for reasons other than the
> actual
> > > > > > task.
> > > > > > > This feature helps keep things clean in Kubernetes.
> > > > > > > - Support for running locally in KIND (Kubernetes in Docker).
> > > > > > > - Automatically tested across many Kubernetes versions with
> Helm
> > 2
> > > > > > and 3
> > > > > > > support.
> > > > > > >
> > > > > > > We’ve found that the cleanest and most reliable way to deploy
> > DAGs
> > > to
> > > > > > > Kubernetes and manage them at scale is to package them into the
> > > actual
> > > > > > > docker image, so we have geared this chart towards that method
> of
> > > > > > > operation, though adding other methods should be
> straightforward.
> > > > > > >
> > > > > > > We would love thoughts from the community and would love to see
> > > this
> > > > > > chart
> > > > > > > help others to get up and running on Kubernetes!
> > > > > > >
> > > > > > > --
> > > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Jarek Potiuk
> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > > >
> > > > M: +48 660 796 129 <+48660796129>
> > > > [image: Polidea] <https://www.polidea.com/>
> > >
> >
> >
> > --
> > *Greg Neiheisel* / Chief Architect Astronomer.io
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
>
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
>
> @Tomasz great question. Our images are currently generated from Dockerfiles
> in this repo https://github.com/astronomer/ap-airflow and get published to
> DockerHub
> https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
>
> For the most part those are typical Airflow images. There's an entrypoint
> script that we include in the image that handles waiting for the database
> and redis (if used) to come up, which is pretty generic.
I already added waiting for the database (both metadata and celery URL) in
the PR:
https://github.com/apache/airflow/pull/7832/files#diff-3759f40d4e8ba0c0e82e82b66d376741
.
It's functionally the same but more generic.
The only other
> thing that I think the Helm Chart uses would be the scripts in this repo
> https://github.com/astronomer/astronomer-airflow-scripts. Our Dockerfiles
> pull this package in. These scripts are used to coordinate running
> migrations and cleaning up failed pods.
>
I see two scripts:
* cleanup_pods -> this is (I believe) not needed to run in airflow - this
could be run as a separate pod/container?
* waiting for migrations -> I think this is a good candidate to add *airflow
db wait_for_migration* command and make it part of airflow itself.
I think we also have to agree on the Airflow version supported by the
official helm chart. I'd suggest we support 1.10.10+ and we incorporate all
the changes needed to airflow (like the "db wait_for_migration") into 2.0
and 1.10 and we support both - image and helm chart for those versions
only. That would help with people migrating to the latest version.
WDYT?
> On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <
> daniel.imberman@gmail.com>
> wrote:
>
> > @jarek I agree completely. I think that pairing an official helm chart
> > with the official image would make for a REALLY powerful “up and running
> > with airflow” story :). Tomek and I have also been looking into
> > operator-sdk which has the ability to create custom controllers from helm
> > charts. We might even able to get a 1-2 punch from the same code base :).
> >
> > @kaxil @jarek @aizhamal @ash if there’s no issues, can we please start
> the
> > process of donation?
> >
> > +1 on my part, of course :)
> >
> >
> >
> > Daniel
> > On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> > wrote:
> > > +1. And it should be paired with the official image we have work in
> > > progress on. I looked a lot at the Astronomer's image while preparing
> my
> > > draft and we can make any adjustments needed to make it works with the
> > helm
> > > chart - and I am super happy to collaborate on that.
> > >
> > > PR here: https://github.com/apache/airflow/pull/7832
> > >
> > > J.
> > >
> > >
> > > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com>
> wrote:
> > >
> > > > @Tomasz Urbaszek <to...@polidea.com> :
> > > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> > > >
> > > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <
> turbaszek@apache.org>
> > > > wrote:
> > > >
> > > > > An official helm chart is something our community needs! Using your
> > > > > chart as the official makes a lot of sens to me because as you
> > > > > mentioned - it's battle tested.
> > > > >
> > > > > One question: what Airflow image do you use? Also, would you mind
> > > > > sharing a link to the chart?
> > > > >
> > > > > Tomek
> > > > >
> > > > >
> > > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > > <gr...@astronomer.io.invalid> wrote:
> > > > > >
> > > > > > Hey everyone,
> > > > > >
> > > > > > Over the past few years at Astronomer, we’ve created, managed,
> and
> > > > > hardened
> > > > > > a production-ready Helm Chart for Airflow (
> > > > > > https://github.com/astronomer/airflow-chart) that is being used
> by
> > > > both
> > > > > our
> > > > > > SaaS and Enterprise customers. This chart is battle-tested and
> > running
> > > > > > hundreds of Airflow deployments of varying sizes and runtime
> > > > > environments.
> > > > > > It’s been built up to encapsulate the issues that Airflow users
> run
> > > > into
> > > > > in
> > > > > > the real world.
> > > > > >
> > > > > > While this chart was originally developed internally for our
> > Astronomer
> > > > > > Platform, we’ve recently decoupled the chart from the rest of our
> > > > > platform
> > > > > > to make it usable by the greater Airflow community. With these
> > changes
> > > > in
> > > > > > mind, we want to start a conversation about donating this chart
> to
> > the
> > > > > > Airflow community.
> > > > > >
> > > > > > Some of the main features of the chart are:
> > > > > >
> > > > > > - It works out of the box. With zero configuration, a user will
> get
> > > > a
> > > > > > postgres database, a default user and the KubernetesExecutor
> ready
> > > > to
> > > > > run
> > > > > > DAGs.
> > > > > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> > > > > > Kubernetes executors.
> > > > > >
> > > > > > Support for optional pgbouncer. We use this to share a
> configurable
> > > > > > connection pool size per deployment. Useful for limiting
> > connections to
> > > > > the
> > > > > > metadata database.
> > > > > >
> > > > > > - Airflow migration support. A user can push a newer version of
> > > > > Airflow
> > > > > > into an existing release and migrations will automatically run
> > > > > cleanly.
> > > > > > - Prometheus support. Optionally install and configure a
> > > > > statsd-exporter
> > > > > > to ingest Airflow metrics and expose them to Prometheus
> > > > automatically.
> > > > > > - Resource control. Optionally control the ResourceQuotas and
> > > > > > LimitRanges for each deployment so that no deployment can
> overload
> > a
> > > > > > cluster.
> > > > > > - Simple optional Elasticsearch support.
> > > > > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> > > > > > KubernetesPodOperator pods fail for reasons other than the actual
> > > > > task.
> > > > > > This feature helps keep things clean in Kubernetes.
> > > > > > - Support for running locally in KIND (Kubernetes in Docker).
> > > > > > - Automatically tested across many Kubernetes versions with Helm
> 2
> > > > > and 3
> > > > > > support.
> > > > > >
> > > > > > We’ve found that the cleanest and most reliable way to deploy
> DAGs
> > to
> > > > > > Kubernetes and manage them at scale is to package them into the
> > actual
> > > > > > docker image, so we have geared this chart towards that method of
> > > > > > operation, though adding other methods should be straightforward.
> > > > > >
> > > > > > We would love thoughts from the community and would love to see
> > this
> > > > > chart
> > > > > > help others to get up and running on Kubernetes!
> > > > > >
> > > > > > --
> > > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Jarek Potiuk
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer
> > >
> > > M: +48 660 796 129 <+48660796129>
> > > [image: Polidea] <https://www.polidea.com/>
> >
>
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Greg Neiheisel <gr...@astronomer.io.INVALID>.
@Tomasz great question. Our images are currently generated from Dockerfiles
in this repo https://github.com/astronomer/ap-airflow and get published to
DockerHub https://hub.docker.com/repository/docker/astronomerinc/ap-airflow.
For the most part those are typical Airflow images. There's an entrypoint
script that we include in the image that handles waiting for the database
and redis (if used) to come up, which is pretty generic. The only other
thing that I think the Helm Chart uses would be the scripts in this repo
https://github.com/astronomer/astronomer-airflow-scripts. Our Dockerfiles
pull this package in. These scripts are used to coordinate running
migrations and cleaning up failed pods.
On Tue, Mar 24, 2020 at 10:49 AM Daniel Imberman <da...@gmail.com>
wrote:
> @jarek I agree completely. I think that pairing an official helm chart
> with the official image would make for a REALLY powerful “up and running
> with airflow” story :). Tomek and I have also been looking into
> operator-sdk which has the ability to create custom controllers from helm
> charts. We might even able to get a 1-2 punch from the same code base :).
>
> @kaxil @jarek @aizhamal @ash if there’s no issues, can we please start the
> process of donation?
>
> +1 on my part, of course :)
>
>
>
> Daniel
> On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <Ja...@polidea.com>,
> wrote:
> > +1. And it should be paired with the official image we have work in
> > progress on. I looked a lot at the Astronomer's image while preparing my
> > draft and we can make any adjustments needed to make it works with the
> helm
> > chart - and I am super happy to collaborate on that.
> >
> > PR here: https://github.com/apache/airflow/pull/7832
> >
> > J.
> >
> >
> > On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com> wrote:
> >
> > > @Tomasz Urbaszek <to...@polidea.com> :
> > > Helm Chart Link: https://github.com/astronomer/airflow-chart
> > >
> > > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <tu...@apache.org>
> > > wrote:
> > >
> > > > An official helm chart is something our community needs! Using your
> > > > chart as the official makes a lot of sens to me because as you
> > > > mentioned - it's battle tested.
> > > >
> > > > One question: what Airflow image do you use? Also, would you mind
> > > > sharing a link to the chart?
> > > >
> > > > Tomek
> > > >
> > > >
> > > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > > <gr...@astronomer.io.invalid> wrote:
> > > > >
> > > > > Hey everyone,
> > > > >
> > > > > Over the past few years at Astronomer, we’ve created, managed, and
> > > > hardened
> > > > > a production-ready Helm Chart for Airflow (
> > > > > https://github.com/astronomer/airflow-chart) that is being used by
> > > both
> > > > our
> > > > > SaaS and Enterprise customers. This chart is battle-tested and
> running
> > > > > hundreds of Airflow deployments of varying sizes and runtime
> > > > environments.
> > > > > It’s been built up to encapsulate the issues that Airflow users run
> > > into
> > > > in
> > > > > the real world.
> > > > >
> > > > > While this chart was originally developed internally for our
> Astronomer
> > > > > Platform, we’ve recently decoupled the chart from the rest of our
> > > > platform
> > > > > to make it usable by the greater Airflow community. With these
> changes
> > > in
> > > > > mind, we want to start a conversation about donating this chart to
> the
> > > > > Airflow community.
> > > > >
> > > > > Some of the main features of the chart are:
> > > > >
> > > > > - It works out of the box. With zero configuration, a user will get
> > > a
> > > > > postgres database, a default user and the KubernetesExecutor ready
> > > to
> > > > run
> > > > > DAGs.
> > > > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> > > > > Kubernetes executors.
> > > > >
> > > > > Support for optional pgbouncer. We use this to share a configurable
> > > > > connection pool size per deployment. Useful for limiting
> connections to
> > > > the
> > > > > metadata database.
> > > > >
> > > > > - Airflow migration support. A user can push a newer version of
> > > > Airflow
> > > > > into an existing release and migrations will automatically run
> > > > cleanly.
> > > > > - Prometheus support. Optionally install and configure a
> > > > statsd-exporter
> > > > > to ingest Airflow metrics and expose them to Prometheus
> > > automatically.
> > > > > - Resource control. Optionally control the ResourceQuotas and
> > > > > LimitRanges for each deployment so that no deployment can overload
> a
> > > > > cluster.
> > > > > - Simple optional Elasticsearch support.
> > > > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> > > > > KubernetesPodOperator pods fail for reasons other than the actual
> > > > task.
> > > > > This feature helps keep things clean in Kubernetes.
> > > > > - Support for running locally in KIND (Kubernetes in Docker).
> > > > > - Automatically tested across many Kubernetes versions with Helm 2
> > > > and 3
> > > > > support.
> > > > >
> > > > > We’ve found that the cleanest and most reliable way to deploy DAGs
> to
> > > > > Kubernetes and manage them at scale is to package them into the
> actual
> > > > > docker image, so we have geared this chart towards that method of
> > > > > operation, though adding other methods should be straightforward.
> > > > >
> > > > > We would love thoughts from the community and would love to see
> this
> > > > chart
> > > > > help others to get up and running on Kubernetes!
> > > > >
> > > > > --
> > > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > > >
> > >
> >
> >
> > --
> >
> > Jarek Potiuk
> > Polidea <https://www.polidea.com/> | Principal Software Engineer
> >
> > M: +48 660 796 129 <+48660796129>
> > [image: Polidea] <https://www.polidea.com/>
>
--
*Greg Neiheisel* / Chief Architect Astronomer.io
Re: Helm Chart for Airflow
Posted by Daniel Imberman <da...@gmail.com>.
@jarek I agree completely. I think that pairing an official helm chart with the official image would make for a REALLY powerful “up and running with airflow” story :). Tomek and I have also been looking into operator-sdk which has the ability to create custom controllers from helm charts. We might even able to get a 1-2 punch from the same code base :).
@kaxil @jarek @aizhamal @ash if there’s no issues, can we please start the process of donation?
+1 on my part, of course :)
Daniel
On Mar 24, 2020, 7:40 AM -0700, Jarek Potiuk <Ja...@polidea.com>, wrote:
> +1. And it should be paired with the official image we have work in
> progress on. I looked a lot at the Astronomer's image while preparing my
> draft and we can make any adjustments needed to make it works with the helm
> chart - and I am super happy to collaborate on that.
>
> PR here: https://github.com/apache/airflow/pull/7832
>
> J.
>
>
> On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com> wrote:
>
> > @Tomasz Urbaszek <to...@polidea.com> :
> > Helm Chart Link: https://github.com/astronomer/airflow-chart
> >
> > On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <tu...@apache.org>
> > wrote:
> >
> > > An official helm chart is something our community needs! Using your
> > > chart as the official makes a lot of sens to me because as you
> > > mentioned - it's battle tested.
> > >
> > > One question: what Airflow image do you use? Also, would you mind
> > > sharing a link to the chart?
> > >
> > > Tomek
> > >
> > >
> > > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > > <gr...@astronomer.io.invalid> wrote:
> > > >
> > > > Hey everyone,
> > > >
> > > > Over the past few years at Astronomer, we’ve created, managed, and
> > > hardened
> > > > a production-ready Helm Chart for Airflow (
> > > > https://github.com/astronomer/airflow-chart) that is being used by
> > both
> > > our
> > > > SaaS and Enterprise customers. This chart is battle-tested and running
> > > > hundreds of Airflow deployments of varying sizes and runtime
> > > environments.
> > > > It’s been built up to encapsulate the issues that Airflow users run
> > into
> > > in
> > > > the real world.
> > > >
> > > > While this chart was originally developed internally for our Astronomer
> > > > Platform, we’ve recently decoupled the chart from the rest of our
> > > platform
> > > > to make it usable by the greater Airflow community. With these changes
> > in
> > > > mind, we want to start a conversation about donating this chart to the
> > > > Airflow community.
> > > >
> > > > Some of the main features of the chart are:
> > > >
> > > > - It works out of the box. With zero configuration, a user will get
> > a
> > > > postgres database, a default user and the KubernetesExecutor ready
> > to
> > > run
> > > > DAGs.
> > > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> > > > Kubernetes executors.
> > > >
> > > > Support for optional pgbouncer. We use this to share a configurable
> > > > connection pool size per deployment. Useful for limiting connections to
> > > the
> > > > metadata database.
> > > >
> > > > - Airflow migration support. A user can push a newer version of
> > > Airflow
> > > > into an existing release and migrations will automatically run
> > > cleanly.
> > > > - Prometheus support. Optionally install and configure a
> > > statsd-exporter
> > > > to ingest Airflow metrics and expose them to Prometheus
> > automatically.
> > > > - Resource control. Optionally control the ResourceQuotas and
> > > > LimitRanges for each deployment so that no deployment can overload a
> > > > cluster.
> > > > - Simple optional Elasticsearch support.
> > > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> > > > KubernetesPodOperator pods fail for reasons other than the actual
> > > task.
> > > > This feature helps keep things clean in Kubernetes.
> > > > - Support for running locally in KIND (Kubernetes in Docker).
> > > > - Automatically tested across many Kubernetes versions with Helm 2
> > > and 3
> > > > support.
> > > >
> > > > We’ve found that the cleanest and most reliable way to deploy DAGs to
> > > > Kubernetes and manage them at scale is to package them into the actual
> > > > docker image, so we have geared this chart towards that method of
> > > > operation, though adding other methods should be straightforward.
> > > >
> > > > We would love thoughts from the community and would love to see this
> > > chart
> > > > help others to get up and running on Kubernetes!
> > > >
> > > > --
> > > > *Greg Neiheisel* / Chief Architect Astronomer.io
> > >
> >
>
>
> --
>
> Jarek Potiuk
> Polidea <https://www.polidea.com/> | Principal Software Engineer
>
> M: +48 660 796 129 <+48660796129>
> [image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Jarek Potiuk <Ja...@polidea.com>.
+1. And it should be paired with the official image we have work in
progress on. I looked a lot at the Astronomer's image while preparing my
draft and we can make any adjustments needed to make it works with the helm
chart - and I am super happy to collaborate on that.
PR here: https://github.com/apache/airflow/pull/7832
J.
On Tue, Mar 24, 2020 at 3:15 PM Kaxil Naik <ka...@gmail.com> wrote:
> @Tomasz Urbaszek <to...@polidea.com> :
> Helm Chart Link: https://github.com/astronomer/airflow-chart
>
> On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <tu...@apache.org>
> wrote:
>
> > An official helm chart is something our community needs! Using your
> > chart as the official makes a lot of sens to me because as you
> > mentioned - it's battle tested.
> >
> > One question: what Airflow image do you use? Also, would you mind
> > sharing a link to the chart?
> >
> > Tomek
> >
> >
> > On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> > <gr...@astronomer.io.invalid> wrote:
> > >
> > > Hey everyone,
> > >
> > > Over the past few years at Astronomer, we’ve created, managed, and
> > hardened
> > > a production-ready Helm Chart for Airflow (
> > > https://github.com/astronomer/airflow-chart) that is being used by
> both
> > our
> > > SaaS and Enterprise customers. This chart is battle-tested and running
> > > hundreds of Airflow deployments of varying sizes and runtime
> > environments.
> > > It’s been built up to encapsulate the issues that Airflow users run
> into
> > in
> > > the real world.
> > >
> > > While this chart was originally developed internally for our Astronomer
> > > Platform, we’ve recently decoupled the chart from the rest of our
> > platform
> > > to make it usable by the greater Airflow community. With these changes
> in
> > > mind, we want to start a conversation about donating this chart to the
> > > Airflow community.
> > >
> > > Some of the main features of the chart are:
> > >
> > > - It works out of the box. With zero configuration, a user will get
> a
> > > postgres database, a default user and the KubernetesExecutor ready
> to
> > run
> > > DAGs.
> > > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> > > Kubernetes executors.
> > >
> > > Support for optional pgbouncer. We use this to share a configurable
> > > connection pool size per deployment. Useful for limiting connections to
> > the
> > > metadata database.
> > >
> > > - Airflow migration support. A user can push a newer version of
> > Airflow
> > > into an existing release and migrations will automatically run
> > cleanly.
> > > - Prometheus support. Optionally install and configure a
> > statsd-exporter
> > > to ingest Airflow metrics and expose them to Prometheus
> automatically.
> > > - Resource control. Optionally control the ResourceQuotas and
> > > LimitRanges for each deployment so that no deployment can overload a
> > > cluster.
> > > - Simple optional Elasticsearch support.
> > > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> > > KubernetesPodOperator pods fail for reasons other than the actual
> > task.
> > > This feature helps keep things clean in Kubernetes.
> > > - Support for running locally in KIND (Kubernetes in Docker).
> > > - Automatically tested across many Kubernetes versions with Helm 2
> > and 3
> > > support.
> > >
> > > We’ve found that the cleanest and most reliable way to deploy DAGs to
> > > Kubernetes and manage them at scale is to package them into the actual
> > > docker image, so we have geared this chart towards that method of
> > > operation, though adding other methods should be straightforward.
> > >
> > > We would love thoughts from the community and would love to see this
> > chart
> > > help others to get up and running on Kubernetes!
> > >
> > > --
> > > *Greg Neiheisel* / Chief Architect Astronomer.io
> >
>
--
Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer
M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>
Re: Helm Chart for Airflow
Posted by Kaxil Naik <ka...@gmail.com>.
@Tomasz Urbaszek <to...@polidea.com> :
Helm Chart Link: https://github.com/astronomer/airflow-chart
On Tue, Mar 24, 2020 at 2:13 PM Tomasz Urbaszek <tu...@apache.org>
wrote:
> An official helm chart is something our community needs! Using your
> chart as the official makes a lot of sens to me because as you
> mentioned - it's battle tested.
>
> One question: what Airflow image do you use? Also, would you mind
> sharing a link to the chart?
>
> Tomek
>
>
> On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
> <gr...@astronomer.io.invalid> wrote:
> >
> > Hey everyone,
> >
> > Over the past few years at Astronomer, we’ve created, managed, and
> hardened
> > a production-ready Helm Chart for Airflow (
> > https://github.com/astronomer/airflow-chart) that is being used by both
> our
> > SaaS and Enterprise customers. This chart is battle-tested and running
> > hundreds of Airflow deployments of varying sizes and runtime
> environments.
> > It’s been built up to encapsulate the issues that Airflow users run into
> in
> > the real world.
> >
> > While this chart was originally developed internally for our Astronomer
> > Platform, we’ve recently decoupled the chart from the rest of our
> platform
> > to make it usable by the greater Airflow community. With these changes in
> > mind, we want to start a conversation about donating this chart to the
> > Airflow community.
> >
> > Some of the main features of the chart are:
> >
> > - It works out of the box. With zero configuration, a user will get a
> > postgres database, a default user and the KubernetesExecutor ready to
> run
> > DAGs.
> > - Support for Local, Celery (w/ optional KEDA autoscaling) and
> > Kubernetes executors.
> >
> > Support for optional pgbouncer. We use this to share a configurable
> > connection pool size per deployment. Useful for limiting connections to
> the
> > metadata database.
> >
> > - Airflow migration support. A user can push a newer version of
> Airflow
> > into an existing release and migrations will automatically run
> cleanly.
> > - Prometheus support. Optionally install and configure a
> statsd-exporter
> > to ingest Airflow metrics and expose them to Prometheus automatically.
> > - Resource control. Optionally control the ResourceQuotas and
> > LimitRanges for each deployment so that no deployment can overload a
> > cluster.
> > - Simple optional Elasticsearch support.
> > - Optional namespace cleanup. Sometimes KubernetesExecutor and
> > KubernetesPodOperator pods fail for reasons other than the actual
> task.
> > This feature helps keep things clean in Kubernetes.
> > - Support for running locally in KIND (Kubernetes in Docker).
> > - Automatically tested across many Kubernetes versions with Helm 2
> and 3
> > support.
> >
> > We’ve found that the cleanest and most reliable way to deploy DAGs to
> > Kubernetes and manage them at scale is to package them into the actual
> > docker image, so we have geared this chart towards that method of
> > operation, though adding other methods should be straightforward.
> >
> > We would love thoughts from the community and would love to see this
> chart
> > help others to get up and running on Kubernetes!
> >
> > --
> > *Greg Neiheisel* / Chief Architect Astronomer.io
>
Re: Helm Chart for Airflow
Posted by Tomasz Urbaszek <tu...@apache.org>.
An official helm chart is something our community needs! Using your
chart as the official makes a lot of sens to me because as you
mentioned - it's battle tested.
One question: what Airflow image do you use? Also, would you mind
sharing a link to the chart?
Tomek
On Tue, Mar 24, 2020 at 2:07 PM Greg Neiheisel
<gr...@astronomer.io.invalid> wrote:
>
> Hey everyone,
>
> Over the past few years at Astronomer, we’ve created, managed, and hardened
> a production-ready Helm Chart for Airflow (
> https://github.com/astronomer/airflow-chart) that is being used by both our
> SaaS and Enterprise customers. This chart is battle-tested and running
> hundreds of Airflow deployments of varying sizes and runtime environments.
> It’s been built up to encapsulate the issues that Airflow users run into in
> the real world.
>
> While this chart was originally developed internally for our Astronomer
> Platform, we’ve recently decoupled the chart from the rest of our platform
> to make it usable by the greater Airflow community. With these changes in
> mind, we want to start a conversation about donating this chart to the
> Airflow community.
>
> Some of the main features of the chart are:
>
> - It works out of the box. With zero configuration, a user will get a
> postgres database, a default user and the KubernetesExecutor ready to run
> DAGs.
> - Support for Local, Celery (w/ optional KEDA autoscaling) and
> Kubernetes executors.
>
> Support for optional pgbouncer. We use this to share a configurable
> connection pool size per deployment. Useful for limiting connections to the
> metadata database.
>
> - Airflow migration support. A user can push a newer version of Airflow
> into an existing release and migrations will automatically run cleanly.
> - Prometheus support. Optionally install and configure a statsd-exporter
> to ingest Airflow metrics and expose them to Prometheus automatically.
> - Resource control. Optionally control the ResourceQuotas and
> LimitRanges for each deployment so that no deployment can overload a
> cluster.
> - Simple optional Elasticsearch support.
> - Optional namespace cleanup. Sometimes KubernetesExecutor and
> KubernetesPodOperator pods fail for reasons other than the actual task.
> This feature helps keep things clean in Kubernetes.
> - Support for running locally in KIND (Kubernetes in Docker).
> - Automatically tested across many Kubernetes versions with Helm 2 and 3
> support.
>
> We’ve found that the cleanest and most reliable way to deploy DAGs to
> Kubernetes and manage them at scale is to package them into the actual
> docker image, so we have geared this chart towards that method of
> operation, though adding other methods should be straightforward.
>
> We would love thoughts from the community and would love to see this chart
> help others to get up and running on Kubernetes!
>
> --
> *Greg Neiheisel* / Chief Architect Astronomer.io