You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Daniel Ferguson (Jira)" <ji...@apache.org> on 2020/02/12 20:29:00 UTC

[jira] [Created] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

Daniel Ferguson created AIRFLOW-6786:
----------------------------------------

             Summary: Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor
                 Key: AIRFLOW-6786
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6786
             Project: Apache Airflow
          Issue Type: New Feature
          Components: contrib, hooks
    Affects Versions: 1.10.9
            Reporter: Daniel Ferguson
            Assignee: Daniel Ferguson


Add the KafkaProducerHook.
Add the KafkaConsumerHook.
Add the KafkaSensor which listens to messages with a specific topic.
Related Issue:
#1311 (Pre-dates Jira Migration)

Reminder to contributors:

You must add an Apache License header to all new files
Please squash your commits when possible and follow the 7 rules of good Git commits
I am new to the community, I am not sure the files are at the right place or missing anything.

The sensor could be used as the first node of a dag where the second node can be a TriggerDagRunOperator. The messages are polled in a batch and the dag runs are dynamically generated.

Thanks!

Note, as per denied PR [#1415|https://github.com/apache/airflow/pull/1415], it is important to mention these integrations are not suitable for low-latency/high-throughput/streaming. For reference, [#1415 (comment)|https://github.com/apache/airflow/pull/1415#issuecomment-484429806].

Co-authored-by: Dan Ferguson [dferguson992@gmail.com|mailto:dferguson992@gmail.com]
Co-authored-by: Hanan Shteingart
Co-authored-by: YuanfΞi Zhu



--
This message was sent by Atlassian Jira
(v8.3.4#803005)