[jira] [Created] (NIFI-7989) Add Hive "data drift" processor

["Matt Burgess (Jira)" <ji...@apache.org>]

Matt Burgess created NIFI-7989:
----------------------------------

             Summary: Add Hive "data drift" processor
                 Key: NIFI-7989
                 URL: https://issues.apache.org/jira/browse/NIFI-7989
             Project: Apache NiFi
          Issue Type: New Feature
          Components: Extensions
            Reporter: Matt Burgess


It would be nice to have a Hive processor (one for each Hive NAR) that could check an incoming record-based flowfile against a destination table, and either add columns and/or partition values, or even create the table if it does not exist. Such a processor could be used in a flow where the incoming data's schema can change and we want to be able to write it to a Hive table, preferably by using PutHDFS, PutParquet, or PutORC to place it directly where it can be queried.

Such a processor should be able to use a HiveConnectionPool to execute any DDL (ALTER TABLE ADD COLUMN, e.g.) necessary to make the table match the incoming data. For Partition Values, they could be provided via a property that supports Expression Language. In such a case, an ALTER TABLE would be issued to add the partition directory.

Whether the table is created or updated, and whether there are partition values to consider, an attribute should be written to the outgoing flowfile corresponding to the location of the table (and any associated partitions). This supports the idea of having a flow that updates a Hive table based on the incoming data, and then allows the user to put the flowfile directly into the destination location (PutHDFS, e.g.) instead of having to load it using HiveQL or being subject to the restrictions of Hive Streaming tables (ORC-backed, transactional, etc.)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)