Heron Sync up Meeting Notes - 01/30/18

[Karthik Ramasamy <ka...@streaml.io>]

Heron Meeting Minutes - 01/30/18


   -

   Updates
   -

      Basic Streamlet aggregators are available in heron now and ready to
      be used. More could be introduced with the same framework.
      -

      Scala support has been approved and PRs are coming in.
      -

      Progress have been made to k8s support and single docker image. There
      will be more work but the features are available to users now.
      -

      Dhalion is still under testing in Twitter. Only killing container is
      being used. Scaling up/down is not enabled because of aurora permission
      restriction
      -

      Java 9 supported is requested. Currently users are mostly using Java
      8 but there are users using 9. We will start from proposal and evaluation.
      -

      Python support is ready to be used.
      -
      -

   Discussion
   -

      New thread safe support PR is under review. API wise we have reached
      the agreement to make the API threadsafe. There are concerns:
      -

         Clarify user requirements.
         -

         Don’t break stateful support
         -

         Evaluate any performance implications
         -

      Dynamic config.
      -

         Doc:
         https://docs.google.com/document/d/1VWVmnGouZMyQriKWwXn25fDeiHKd7nYCIcbAyb2jcak/edit#
         -

         It is required by the Rate Limiter feature, but could be used for
         other configs.
         -

         Try to add to the existing “heron update” command first instead of
         introducing new command
         -

         The existing PhysicalPlan propagation mechanism will be used.
         -

      Current StreamManager quitting logic seems to be risky.
      -

         Doc with more details to be created. TODO.
         -

      Slack digest support. Sree, Matteo, Dave are looking