[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive

["Nishant Bangarwa (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nishant Bangarwa updated HIVE-18976:
------------------------------------
    Attachment: HIVE-18976.patch

> Add ability to setup Druid Kafka Ingestion from Hive
> ----------------------------------------------------
>
>                 Key: HIVE-18976
>                 URL: https://issues.apache.org/jira/browse/HIVE-18976
>             Project: Hive
>          Issue Type: Bug
>          Components: Druid integration
>            Reporter: Nishant Bangarwa
>            Assignee: Nishant Bangarwa
>            Priority: Major
>         Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and druid can start ingesting events from kafka. 
> {code:java}
>  
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language string, `user` string, added int, deleted int, delta int)
>         STORED BY 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
>         TBLPROPERTIES (
>         "druid.segment.granularity" = "HOUR",
>         "druid.query.granularity" = "MINUTE",
>         "kafka.bootstrap.servers" = "localhost:9092",
>         "kafka.topic" = "test-topic",
>         "druid.kafka.ingest.useEarliestOffset" = "true"
>         );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that extends existing DruidStorageHandler and adds the additional functionality for Streaming. 
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster + Single Node Kafka Broker. The broker can be populated with a test topic that has some predefined data. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)