10/06 Airflow Contributors Meeting Notes

[Gurer Kiratli <gu...@airbnb.com.INVALID>]

Hi all ,

Here are the meeting notes. Please add/modify as needed.

I also have recording of the meeting. It's in two files due to some Webex
issue that required a restart.

Video 1 <https://airbnb.box.com/s/8b2c5691ux9qmkujum813xyscyol6rod> , Video
2 <https://airbnb.box.com/s/m1uxmkzcedd79f9fwd0s50hcda6hirmc>

Cheers,

Gurer

*Attendees: *

   -

   Gurer Kiratli
   -

   Paul Yang
   -

   Maxime Beauchemin
   -

   Ben Tallman
   -

   George Leslie-Waksman
   -

   Vijay Bhat
   -

   Rob Froetscher
   -

   Xuanji Li
   -

   Bolke de Bruin
   -

   Catherine Wong
   -

   Sumit Maheshwari
   -

   Joe Schmid
   -

   Julia Hsieh
   -

   Sam


*Agenda:*


   -

   Airbnb Update
   -

   Get an apache 1.7.3 release out
   -

   Eliminating cold-case PRs - So far, only Jeremiah and I have added to
   the list below. We need help from all committers:
   https://cwiki.apache.org/confluence/display/AIRFLOW/Whittling+down+PR+List
   -

   Roadmap



*Meeting Notes *



   -

   Airflow
   -

      60 companies, 200 contributors, a lot of interest for meetups.
      -

      Growing faster than other projects like Luigi, Oozie, Azkaban.
      -

   Airbnb Update
   -

      Cgroups for containment. Bad tasks don’t take down workers.
      -

      Impersonation. All jobs run under service account called
      airflow_user. This creates accounting and ownership problems.
      -

      Creating a new cluster in a different data center in the world.
      -

   ING update
   -

      Moving towards cross data center availability. Airflow might play a
      critical role in here. ReAir might be useful.
      -

      Cgroups is interesting.
      -

      Integration with legacy technologies with APIs is key.
      -

   Apache Release
   -

      Next week we are going to cut a release candidate.
      -

      Bolke, Max and some folks were working 1.7.1.3 to 1.7.2 having just
      cherry picking PRs that is related to Apache compliance. This is most to
      test out to process. 1.7.2 will be essentially just 1.7.1.3 not really a
      need to install this.
      -

      ETA is end of next week.
      -

      We will have the RC for 1.8.0. We will inform thru the Apache mail
      list. We should have other folks test as well.
      -

      We need to release every month.
      -

      Each organization has to have a pre-prod/staging environment.
      -

      We can have a Apache staging environment.
      -

      There are plugins and niche operators for every company. How are we
      going to handle?
      -

         Maybe we can modularize scheduler, UI etc.
         -

         If people want certain functionality they can be introduced as
         plugins or module.
         -

         Need to decouple components.
         -

         We need to design this plugin architecture easier said than done.
         There is already JIRAs for this JIRA1
         <https://issues.apache.org/jira/browse/AIRFLOW-299> , JIRA2
         <https://issues.apache.org/jira/browse/AIRFLOW-226> for this.
         -

      Airflow 2.0?
      -

         We might want to have a major fork out.
         -

         This might mean breaking backwards compatibility, repackaging.
         -

         This can prevent moving features for 1 and slate them for 2.0.
         -

         We can break the operators into sub-packages
         -

         Stateless Webservers could be in this.
         -

         DSL for defining pipelines should be backwards compatible. Old
         DAGs should work.
         -

         We can use Git hashes to see get the versions of the versions of
         DAGs.
         -

         We can have a field in the DAG that specify which Airflow version
         this DAG was designed for.
         -

   Cold PRs
   -

      We want more committers. More interaction gets you closer to being a
      committer.
      -

      It will be easier to review more PRs.
      -

      Lots of PRs need rebasing and would have conflicts.
      -

      IF you send a PR that is touching the core hence “dangerous” this
      would require much more scrutiny. Get buy in from the committers
      beforehand. Your work might not be committed at all. Having a
design doc is
      a good idea. Or your PR will be treated as a design doc. : ) If we trust
      you, if you have already done committed PRs before there is more
confidence
      so more chance to be reviewed. All the PRs has to be linted and testing
      needs to converged.
      -

      Testing is unstable. Travis is flapping. Some tests have some
      randomness. This has to be fixed. Cos your PR might be good but some test
      will fail, this is misleading.
      -

      We have a very limited control on the GitHub repo. Apache owns it.
      They will not give us the admin. We can’t introduce 3rd party services as
      we are not admins of the repo.
      -

      [Andrew Phillips] In Jcloud <https://github.com/jclouds/jclouds/> we
      have a GitHub-hosted mirror. We can do whatever we want in this
repo. We do
      CI thru this. We can consider this in Airflow too. "Read-only
mirror of ASF
      Git Repo for jclouds http://jclouds.apache.org/"
      -

      Maybe make a GitHub organization like Airflow-Airflow or something.
      -

      Can we have a policy on open PRs that has been open for n weeks? It
      will be closed. Maybe we can have a policy if it hits a certain
age, we ask
      for a rebase and if we don’t hear for n days we should then close.
      -

      We might be able to automate this. Rails does it in a good way. But
      at this stage we might do it manually.
      -

      Let’s have a policy and put it out in the wiki. Also clarify
      ownership of components.
      -

      For different requests how do we handle communication?
      -

         Kill JIRA. Use GitHub Issues. Apache allows it. GitHub issues is
         easier to search.
         -

         Gitter is not super helpful. It’s more for ad hoc communication
         like Slack.
         -

         Dev mailing list would be for generic questions, requests. We have
         to have this. Apache enforces this. All the decision has to
be given thru
         the Apache mailing list for legal reasons. Apache doesn’t want secret
         decision. We can have a mirror Google groups that is
subscribed to this. We
         can possibly multiple Apache mailing list.
         -

         Mail is slower but mitigates being spread around different
         timezones issue.
         -

      We can break out tickets to Newbie tickets and Projects. Projects
      will be sponsored by
      -

      Differentiate the level of interest and level of desire to commitment.


Q4, 2017 Vision / Roadmap

We will create a wiki page with these and see interest.


Possible Deliverables

   -

   Integration Testing Environment(s)
   -

   Modularizing Airflow
   -

   Containers (Docker, Kubernetes, ECS)
   -

      How do we package management for the DAG and its dependencies, the
      environment?
      -

      Running on your laptop is an important thing to keep. This helped the
      project a lot. This shouldn’t be let go.
      -

   Multitenancy
   -

   Security Improvements
   -

      UI and CLI level roles.
      -

      Kerberos support.
      -

      Putting the password information in a Vault. Currently if you have
      the key you have access to the whole vault.
      -

      Managing connection pools better.
      -

      Defining the difference between a hook and connection.
      -

   Stateless Webservers
   -

      This can help UI improvements.
      -

   UI improvements
   -

      Performance.. like better assets caching, compression and use of cdn..
      -

      Reactify.
      -

      Support for large DAGs.
      -

      This might be dependent on the API.
      -

   Managing logs
   -

   Tighter Dev, test, deploy story
   -

      The local environment and production environment need to be in line.
      -

      DAG.validate method.
      -

   Rest API
   -

      Outside interface where other applications integrates. Like to
      trigger a task.
      -

      Internal service APIs to get execution details. To abstract the
      Database.
      -

      It’s good to see who is interested in these features(care about them)
      and driving these features.
      -

   Hardening the scheduler. Stabilizing.
   -

      Documenting how the scheduler operates.
      -

      Running a single task continuously.
      -

      More visibility into Why do we need to restart scheduler?
      -

      Clarifying the contract between the scheduler and the workers(???)
      -

      Stuck scheduler issues.
      -

      Backfill has a separate code path. It should be a flavor.
      -

   Better documentation
   -

      Onboarding documentation
      -

      Runbook
      -

      How the system works?
      -

   Event driven scheduler
   -

      First task snowballing and kicking other tasks. This reduces the
      white space between when task is runnable and the task is actually is run.
      -

   Revamp the SubDAG operator
   -

      So many special cases in the code base.
      -

      It should be handled by the scheduler.
      -

   Having a task name, separate from task id. Same for the DAG name and DAG
   id.
   - Remove pickling.