You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Deepak Goel <de...@gmail.com> on 2016/06/03 08:30:13 UTC
Hadoop/HDFS/POSIX
Hey
Namaskara~Nalama~Guten Tag~Bonjour
The Hadoop document gives a pointer that HDFS relaxes some POSIX
requirements to maintain high throughput rate.
https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
But it does not give exactly what are the things which it relaxes? Any idea
which programs should I look into to find the things which HDFS relaxes?
Thanks
Deepak
--
Keigu
Deepak
73500 12833
www.simtree.net, deepak@simtree.net
deicool@gmail.com
LinkedIn: www.linkedin.com/in/deicool
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
Re: Hadoop/HDFS/POSIX
Posted by Deepak Goel <de...@gmail.com>.
Thank You. Would you suggest profiling the application (if it is ethical)
and finding out what are the differences? Or does it seem a wasteful
exercise?
Hey
Namaskara~Nalama~Guten Tag~Bonjour
--
Keigu
Deepak
73500 12833
www.simtree.net, deepak@simtree.net
deicool@gmail.com
LinkedIn: www.linkedin.com/in/deicool
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
On Sat, Jun 4, 2016 at 1:32 AM, Chris Nauroth <cn...@hortonworks.com>
wrote:
> Unfortunately, I'm not aware of any comprehensive documentation on how
> HDFS adheres to POSIX and how it differs. To some degree, it's possible to
> look at the FileSystem specification and corresponding contract tests as a
> source of information, but it's not structured to answer that question
> directly. I filed HADOOP-13104 to track improving that, but I haven't been
> able to work on it yet.
>
>
> http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/filesystem/index.html
>
> https://issues.apache.org/jira/browse/HADOOP-13104
>
> There is also some discussion of POSIX specifically related to permission
> handling in the HDFS permissions guide.
>
>
> http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
>
> --Chris Nauroth
>
> From: Gagan Brahmi <ga...@gmail.com>
> Date: Friday, June 3, 2016 at 9:18 AM
> To: Deepak Goel <de...@gmail.com>
> Cc: user <us...@hadoop.apache.org>
> Subject: Re: Hadoop/HDFS/POSIX
>
> AFAIK the major difference include the simplified Coherency model
> <https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Simple_Coherency_Model>
> in HDFS.
>
> The other one is the file permissions. The read (r), write (w) & execute
> (x) works the same way for directories in HDFS like a POSIX file system.
> But for the files, execute permission is ignored in HDFS since you can't
> really execute a file in HDFS.
>
> HTH
>
>
> Regards,
> Gagan Brahmi
>
> On Fri, Jun 3, 2016 at 1:30 AM, Deepak Goel <de...@gmail.com> wrote:
>
>>
>> Hey
>>
>> Namaskara~Nalama~Guten Tag~Bonjour
>>
>> The Hadoop document gives a pointer that HDFS relaxes some POSIX
>> requirements to maintain high throughput rate.
>>
>> https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
>>
>> But it does not give exactly what are the things which it relaxes? Any
>> idea which programs should I look into to find the things which HDFS
>> relaxes?
>>
>> Thanks
>> Deepak
>> --
>> Keigu
>>
>> Deepak
>> 73500 12833
>> www.simtree.net, deepak@simtree.net
>> deicool@gmail.com
>>
>> LinkedIn: www.linkedin.com/in/deicool
>> Skype: thumsupdeicool
>> Google talk: deicool
>> Blog: http://loveandfearless.wordpress.com
>> Facebook: http://www.facebook.com/deicool
>>
>> "Contribute to the world, environment and more :
>> http://www.gridrepublic.org
>> "
>>
>
>
Re: Hadoop/HDFS/POSIX
Posted by Chris Nauroth <cn...@hortonworks.com>.
Unfortunately, I'm not aware of any comprehensive documentation on how HDFS adheres to POSIX and how it differs. To some degree, it's possible to look at the FileSystem specification and corresponding contract tests as a source of information, but it's not structured to answer that question directly. I filed HADOOP-13104 to track improving that, but I haven't been able to work on it yet.
http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/filesystem/index.html
https://issues.apache.org/jira/browse/HADOOP-13104
There is also some discussion of POSIX specifically related to permission handling in the HDFS permissions guide.
http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html
--Chris Nauroth
From: Gagan Brahmi <ga...@gmail.com>>
Date: Friday, June 3, 2016 at 9:18 AM
To: Deepak Goel <de...@gmail.com>>
Cc: user <us...@hadoop.apache.org>>
Subject: Re: Hadoop/HDFS/POSIX
AFAIK the major difference include the simplified Coherency model<https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Simple_Coherency_Model> in HDFS.
The other one is the file permissions. The read (r), write (w) & execute (x) works the same way for directories in HDFS like a POSIX file system. But for the files, execute permission is ignored in HDFS since you can't really execute a file in HDFS.
HTH
Regards,
Gagan Brahmi
On Fri, Jun 3, 2016 at 1:30 AM, Deepak Goel <de...@gmail.com>> wrote:
Hey
Namaskara~Nalama~Guten Tag~Bonjour
The Hadoop document gives a pointer that HDFS relaxes some POSIX requirements to maintain high throughput rate.
https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
But it does not give exactly what are the things which it relaxes? Any idea which programs should I look into to find the things which HDFS relaxes?
Thanks
Deepak
--
Keigu
Deepak
73500 12833
www.simtree.net<http://www.simtree.net>, deepak@simtree.net<ma...@simtree.net>
deicool@gmail.com<ma...@gmail.com>
LinkedIn: www.linkedin.com/in/deicool<http://www.linkedin.com/in/deicool>
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
Re: Hadoop/HDFS/POSIX
Posted by Gagan Brahmi <ga...@gmail.com>.
AFAIK the major difference include the simplified Coherency model
<https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Simple_Coherency_Model>
in HDFS.
The other one is the file permissions. The read (r), write (w) & execute
(x) works the same way for directories in HDFS like a POSIX file system.
But for the files, execute permission is ignored in HDFS since you can't
really execute a file in HDFS.
HTH
Regards,
Gagan Brahmi
On Fri, Jun 3, 2016 at 1:30 AM, Deepak Goel <de...@gmail.com> wrote:
>
> Hey
>
> Namaskara~Nalama~Guten Tag~Bonjour
>
> The Hadoop document gives a pointer that HDFS relaxes some POSIX
> requirements to maintain high throughput rate.
>
> https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html
>
> But it does not give exactly what are the things which it relaxes? Any
> idea which programs should I look into to find the things which HDFS
> relaxes?
>
> Thanks
> Deepak
> --
> Keigu
>
> Deepak
> 73500 12833
> www.simtree.net, deepak@simtree.net
> deicool@gmail.com
>
> LinkedIn: www.linkedin.com/in/deicool
> Skype: thumsupdeicool
> Google talk: deicool
> Blog: http://loveandfearless.wordpress.com
> Facebook: http://www.facebook.com/deicool
>
> "Contribute to the world, environment and more :
> http://www.gridrepublic.org
> "
>