You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Anshuman Ghosh <an...@gmail.com> on 2017/03/24 12:52:59 UTC
Need help in knowing a procedure to automatically refresh HIVE
metadata when a new partition is added
Hello Team,
This is Anshu here, nice to meet you!
I am working at Crealytics, Berlin as a Big Data Consultant/ Solution
Architect.
We are building/ designing a data flow pipeline using mainly *Apache NiFi
--> Consume data from Kafka --> Transform, Convert and Merge --> Write to
HDFS*
We have several *HIVE External table*s defined to point to HDFS location
where we are dumping the data.
All these tables are partitioned (mostly by Date); through NiFi we are able
to create and write to a new partition.
However we need to execute a meta refresh (MSCK REPAIR ..) in order for
HIVE to pick the new partition.
Is there any way to automate this process?
We can have a small script to run and do the same but then we wouldn't know
the run frequency of it (in case of a partition apart from Date).
Thanking you in advance!
______________________
*Kind Regards,*
*Anshuman Ghosh*
*Contact - +49 179 9090964*