You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mahender Sarangam <ma...@outlook.com> on 2018/06/03 07:26:21 UTC
Hive External Table on particular set of files.
We are copying files from our upstream system which are in JSON GZ format. They are following a pattern for very daily slice say YYYYMMDDHH (2018053100) they are maintianing two folders DATA and METADATA. Where DATA holds actual data and METADATA holds RowCount of that day's data.We need to create external table on top of copied data, where it only consider *.json.gz extension files only, excluding other file extensions. We dont want to copy files into another location since they are large in size. We also tried INPUT_ FILENAME .... virtual column, it didn't work. Any suggesstion for this scnearios ?
Re: Hive External Table on particular set of files.
Posted by Elliot West <te...@gmail.com>.
On which type of file system are you storing the data? S3? HDFS? Other?
On Sun, 3 Jun 2018 at 08:26, Mahender Sarangam <ma...@outlook.com>
wrote:
> We are copying files from our upstream system which are in JSON GZ format.
> They are following a pattern for very daily slice say YYYYMMDDHH
> (2018053100) they are maintianing two folders DATA and METADATA. Where DATA
> holds actual data and METADATA holds RowCount of that day's data.We need to
> create external table on top of copied data, where it only consider
> *.json.gz extension files only, excluding other file extensions. We dont
> want to copy files into another location since they are large in size. We
> also tried INPUT_ FILENAME .... virtual column, it didn't work. Any
> suggesstion for this scnearios ?
>