Apache Beam Newsletter - December 2018

[Rose Nguyen <rt...@google.com>]

[image: Beam.png]

December 2018 | Newsletter

     What’s been done

------------------------------

KafkaIO (by: Alexey Romanenko, Raghu Angadi)

   -

   Added writing support with ProducerRecord
   -

   See BEAM-6063 <https://issues.apache.org/jira/browse/BEAM-6063> for more
   details


HadoopFormatIO (by: Alexey Romanenko, David Moravek, David Hrbacek)

   -

   Added writing support (batching/streaming) for HadoopFormatIO.
   -

   See BEAM-5310 <https://issues.apache.org/jira/browse/BEAM-5310> for more
   details


Flink Runner (by: Ankur Goenka, Maximilian Michels, Thomas Weise, Ryan
Williams, Robert Bradshaw)

   -

   Portability: Integration of timers and state in user functions for
   streaming and batch execution
   -

   Portability: Combiner lifting (performance optimization)
   -

   Portability: More easily readable operator names
   -

   Support for Beam in Flink 1.5 and 1.6
   -

   Support for scaling up Beam applications on Flink
   -

   Bug fixes and improved testing


TableMacroUDF (by: Gleb Kanterov)

   -

   Support for user defined table functions (UDTF).
   -

   See BEAM-6133 for more details.



     What we’re working on...

------------------------------

Flink Runner (by: Ankur Goenka, Maximilian Michels, Thomas Weise, Ryan
Williams, Robert Bradshaw)

   -

   Portability: Integration of metrics
   -

   Better support for Flink savepoints
   -

   Performance tuning of portable pipelines
   -

   Portability: Integration with Kubernetes
   -

   Improving the testing infrastructure on Jenkins


Load tests of Core Apache Beam Operations (by: Łukasz Gajowy, Katarzyna
Kucharczyk)

   -

   Test operations such as GroupByKey, ParDo, Combine etc in stressful
   conditions.
   -

   See https://s.apache.org/load-test-basic-operations for more details on
    how it works.



     New committers
------------------------------



   -

   Matthias Baetens, London, England



     Talks & meetups
------------------------------

DevFest 2018 @ Warsaw

   -

   “Apache Beam - what do I Gain?” with speaker Łukasz Gajowy
   -

   Link to description here
   <https://warsaw.devfest.pl/schedule/2018-12-02?sessionId=131> and slides
   here
   <https://drive.google.com/file/d/15W4kAcYxZjd7BAYHVBxpLSmEIcUPPfl7/view>


Bay Area Apache Beam Meetup @ San Francisco

   -

   Kicking off the first Bay Area Apache Beam Meetup in SF, December 12th.
   -

   Speakers are Andrew Pilloud and Kenn Knowles
   -

   To be recorded and posted here.


Open Source Summit @ Paris

   -

   “Apache Beam: portable and evolutive data-intensive applications”,
   December 5th.
   -

   Speaker: Alexey Romanenko
   -

   Slides: link
   <https://drive.google.com/open?id=18AzsjTkIiB5bIDTd2V7UQs-sTcVYhxHq>


DevFest 2018 @ London / Bucharest

   -

   “Large scale stream analytics with Apache Beam” with speaker Matthias
   Baetens
   -

   Slides: here
   <https://docs.google.com/presentation/d/1-Q5IGauOqM4zvdKXxHMbeexIeK_2TY4xlx64lnWzkRA/edit?usp=sharing>


Until Next Time!

*This edition was curated by our community of contributors, committers and
PMCs. It contains work done in December 2018 and ongoing efforts. We hope
to provide visibility to what's going on in the community, so if you have
questions, feel free to ask in this thread.*--
Rose Thị Nguyễn