Retrieve the file content based on file name

[Arun Patel <ar...@gmail.com>]

Hive Experts,

I am using INPUT__FILE__NAME to store the file name in a column of ORC
table.

Now, Is there a way to retrieve the content of the HDFS file using any Hive
function?

If there is no function available, Is there a UDF I can use?

Thank you very much for help.

- Arun

Re: Retrieve the file content based on file name

[Staņislavs Rogozins <st...@gmail.com>]

I don't think the file name s are stored per se, more like it's metadata
that can be accessed in queries like pseudocolumns in Oracle.

More info:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+VirtualColumns

On Nov 22, 2016 10:41 PM, "Gopal Vijayaraghavan" <go...@apache.org> wrote:

> > I am using INPUT__FILE__NAME to store the file name in a column of ORC
> table.
>
> That's something I don't understand (the use of the virtual column).
>
> It is common practice to normalize stuff like images as S3 file-paths
> instead of actual bytes inside an ORC file for efficiency of storage.
>
> > If there is no function available, Is there a UDF I can use?
>
> No, but it should be pretty easy to write a UDF to do whatever you want it
> to do in Java.
>
> This isn’t really recommended (for performance reasons) to do IO in the
> middle of a Hive query, but it will work.
>
> Cheers,
> Gopal
>
>
>

Re: Retrieve the file content based on file name

[Gopal Vijayaraghavan <go...@apache.org>]

> I am using INPUT__FILE__NAME to store the file name in a column of ORC table.  

That's something I don't understand (the use of the virtual column).

It is common practice to normalize stuff like images as S3 file-paths instead of actual bytes inside an ORC file for efficiency of storage.

> If there is no function available, Is there a UDF I can use?

No, but it should be pretty easy to write a UDF to do whatever you want it to do in Java.

This isn’t really recommended (for performance reasons) to do IO in the middle of a Hive query, but it will work.

Cheers,
Gopal

Re: Retrieve the file content based on file name

[Staņislavs Rogozins <st...@gmail.com>]

Raw file contents? Maybe create an external single-column textfile table
over the data?

On Nov 21, 2016 5:33 PM, "Arun Patel" <ar...@gmail.com> wrote:

> Hive Experts,
>
> I am using INPUT__FILE__NAME to store the file name in a column of ORC
> table.
>
> Now, Is there a way to retrieve the content of the HDFS file using any
> Hive function?
>
> If there is no function available, Is there a UDF I can use?
>
> Thank you very much for help.
>
> - Arun
>