Introducing Drone Fly - Decouples listeners from Hive metastore

[Abhimanyu Gupta <ab...@gmail.com>]

Hello Everyone,

We (Expedia Group's Data platform team) have just open-sourced Drone Fly
<https://github.com/ExpediaGroup/drone-fly>.
[image: DroneFly_logo.png]
*What is Drone Fly?*
Drone Fly is a distributed Hive metastore events forwarder service that
allows users to deploy metastore listeners outside the Hive metastore
service.

*How does it work?*
With Drone Fly, you need to install just one listener in your metastore.
i.e. the open-source Apiary-Kafka-Listener
<https://github.com/ExpediaGroup/apiary-extensions/tree/master/apiary-metastore-events/kafka-metastore-events/kafka-metastore-listener>.
You can then install one or more of your listeners in Drone Fly’s virtual
Hive context by providing them on Drone Fly’s classpath.

Terraform module to get started: Apiary-Drone-Fly
<https://github.com/ExpediaGroup/apiary-drone-fly>

*What are the benefits?*
1. With Drone Fly, your Hive metastore is completely decoupled from your
listeners.
2. Even if one of your listeners goes down, other listeners can continue to
function without interruption.
3. Gives you the flexibility to deploy multiple listeners without having to
worry about metastore performance.
4. It streamlines the onboarding of new listeners to our data lake.

*Further reading:*
If you would like to know more about the background of this project and how
your data lake architecture changes with Drone Fly, please read the post on
our external blog here:
https://medium.com/expedia-group-tech/drone-fly-decoupling-event-listeners-from-the-hive-metastore-43b0107b15e2

<https://medium.com/expedia-group-tech/drone-fly-decoupling-event-listeners-from-the-hive-metastore-43b0107b15e2>
Github: https://github.com/ExpediaGroup/drone-fly

Thanks,
-- 
Abhimanyu Gupta
on behalf of, *Expedia Group Data Platform Team*