You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Mahender Sarangam <Ma...@outlook.com> on 2017/01/03 21:31:07 UTC
Re: Zero Bytes Files importance
Thanks Gopal. Yeah I'm using CloudBerry. Storage is Azure.
Are you saying this 0000_0,1,2,3 are directories ?.
[cid:part1.C19351C7.1FC7FD04@outlook.com]
On 12/29/2016 11:18 AM, Gopal Vijayaraghavan wrote:
For any insert operation, there will be one Zero bytes file. I would like to know importance of this Zero bytes file.
They are directories.
I'm assuming you're using S3A + screenshots from something like Bucket explorer.
These directory entries will not be shown if you do something like "hadoop fs -ls s3a://…/"
I had a recent talk covering the specifics of S3 + Hive - https://www.slideshare.net/secret/3cfQbeo3cI6GpK/3
Cheers,
Gopal
Re: Zero Bytes Files importance
Posted by Mahender Sarangam <Ma...@outlook.com>.
Thanks Gopal for providing detail explanation.
On 1/3/2017 5:59 PM, Gopal Vijayaraghavan wrote:
>> Thanks Gopal. Yeah I'm using CloudBerry. Storage is Azure.
> Makes sense, only an object store would have this.
>
>> Are you saying this 0000_0,1,2,3 are directories ?.
> No, only the zero size "files".
>
> This is really for compat with regular filesystems.
>
> If you have /tmp/1/foo in an object store that's a single key. That does not imply you'll find "/tmp" or "/tmp/1" in the object store keys.
>
> A FileSystem however assumes parent directories are "real things", so any FileSystem abstraction has to maintain "/tmp", "/tmp/1/" and "/tmp/1/foo" to keep up the basic compatibility requirements of fs.exists("/tmp").
>
> Cheers,
> Gopal
>
>
Re: Zero Bytes Files importance
Posted by Mahender Sarangam <Ma...@outlook.com>.
Thanks Gopal for providing detail explanation.
On 1/3/2017 5:59 PM, Gopal Vijayaraghavan wrote:
>> Thanks Gopal. Yeah I'm using CloudBerry. Storage is Azure.
> Makes sense, only an object store would have this.
>
>> Are you saying this 0000_0,1,2,3 are directories ?.
> No, only the zero size "files".
>
> This is really for compat with regular filesystems.
>
> If you have /tmp/1/foo in an object store that's a single key. That does not imply you'll find "/tmp" or "/tmp/1" in the object store keys.
>
> A FileSystem however assumes parent directories are "real things", so any FileSystem abstraction has to maintain "/tmp", "/tmp/1/" and "/tmp/1/foo" to keep up the basic compatibility requirements of fs.exists("/tmp").
>
> Cheers,
> Gopal
>
>
Re: Zero Bytes Files importance
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> Thanks Gopal. Yeah I'm using CloudBerry. Storage is Azure.
Makes sense, only an object store would have this.
> Are you saying this 0000_0,1,2,3 are directories ?.
No, only the zero size "files".
This is really for compat with regular filesystems.
If you have /tmp/1/foo in an object store that's a single key. That does not imply you'll find "/tmp" or "/tmp/1" in the object store keys.
A FileSystem however assumes parent directories are "real things", so any FileSystem abstraction has to maintain "/tmp", "/tmp/1/" and "/tmp/1/foo" to keep up the basic compatibility requirements of fs.exists("/tmp").
Cheers,
Gopal