You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Raghu Angadi <ra...@yahoo-inc.com> on 2007/11/06 08:29:45 UTC
hard link count for a file
Is there a way to find hard link count for a file in Java? (id
'st_nlink' in struct stat).
Thanks,
Raghu.
Re: hard link count for a file
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Raghu Angadi wrote:
>
> Thanks Nigel.
>
> Another hack I can think of is, if DataNode is not finalized we can try
> to lock current/subdir.../blk_id and previous/subdir../blk_id. If second
> lock fails, we can assume they are the same.
This works on both Linux and Windows. I am thinking of using it for new
method FSDataset.detachBackup(Block).
Raghu.
Re: hard link count for a file
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
dhruba Borthakur wrote:
> Hi Raghu,
>
> Can you pl explain what curFile and backupFile refer to in this context?
> Are you referring to the block data file and the metafile?
backupFile refers to the hard link under "previous" directory (previous
directory is created during upgrade).
So if dir1/current/subdir41/blk_### is curFile, backupFile would be
dir1/previous/subdir41/blk_###.
Raghu.
> Thanks,
> dhruba
>
> -----Original Message-----
> From: Raghu Angadi [mailto:rangadi@yahoo-inc.com]
> Sent: Friday, November 30, 2007 4:05 PM
> To: hadoop-dev@lucene.apache.org
> Subject: Re: hard link count for a file
>
>
> I am back to writing this code as part of periodic block verification.
>
> Another approach I am thinking of is to check curFile.lastModified() and
>
> backupFile.lastModified(). As long as these two are different, we could
> assume they are different files. This is simpler than trying to lock the
>
> files.
>
> Raghu.
>
> Raghu Angadi wrote:
>> Thanks Nigel.
>>
>> Another hack I can think of is, if DataNode is not finalized we can
> try
>> to lock current/subdir.../blk_id and previous/subdir../blk_id. If
> second
>> lock fails, we can assume they are the same.
>>
>> Raghu.
>>
>> Nigel Daley wrote:
>>> Raghu,
>>>
>>> LGPL not allowed in distro:
>>> http://people.apache.org/~cliffs/3party.html
>>>
>>> Nige
>>>
>>> On Nov 6, 2007, at 11:32 PM, Raghu Angadi wrote:
>>>
>
RE: hard link count for a file
Posted by dhruba Borthakur <dh...@yahoo-inc.com>.
Hi Raghu,
Can you pl explain what curFile and backupFile refer to in this context?
Are you referring to the block data file and the metafile?
Thanks,
dhruba
-----Original Message-----
From: Raghu Angadi [mailto:rangadi@yahoo-inc.com]
Sent: Friday, November 30, 2007 4:05 PM
To: hadoop-dev@lucene.apache.org
Subject: Re: hard link count for a file
I am back to writing this code as part of periodic block verification.
Another approach I am thinking of is to check curFile.lastModified() and
backupFile.lastModified(). As long as these two are different, we could
assume they are different files. This is simpler than trying to lock the
files.
Raghu.
Raghu Angadi wrote:
>
> Thanks Nigel.
>
> Another hack I can think of is, if DataNode is not finalized we can
try
> to lock current/subdir.../blk_id and previous/subdir../blk_id. If
second
> lock fails, we can assume they are the same.
>
> Raghu.
>
> Nigel Daley wrote:
>> Raghu,
>>
>> LGPL not allowed in distro:
>> http://people.apache.org/~cliffs/3party.html
>>
>> Nige
>>
>> On Nov 6, 2007, at 11:32 PM, Raghu Angadi wrote:
>>
>
Re: hard link count for a file
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
I am back to writing this code as part of periodic block verification.
Another approach I am thinking of is to check curFile.lastModified() and
backupFile.lastModified(). As long as these two are different, we could
assume they are different files. This is simpler than trying to lock the
files.
Raghu.
Raghu Angadi wrote:
>
> Thanks Nigel.
>
> Another hack I can think of is, if DataNode is not finalized we can try
> to lock current/subdir.../blk_id and previous/subdir../blk_id. If second
> lock fails, we can assume they are the same.
>
> Raghu.
>
> Nigel Daley wrote:
>> Raghu,
>>
>> LGPL not allowed in distro:
>> http://people.apache.org/~cliffs/3party.html
>>
>> Nige
>>
>> On Nov 6, 2007, at 11:32 PM, Raghu Angadi wrote:
>>
>
Re: hard link count for a file
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
Thanks Nigel.
Another hack I can think of is, if DataNode is not finalized we can try
to lock current/subdir.../blk_id and previous/subdir../blk_id. If second
lock fails, we can assume they are the same.
Raghu.
Nigel Daley wrote:
> Raghu,
>
> LGPL not allowed in distro:
> http://people.apache.org/~cliffs/3party.html
>
> Nige
>
> On Nov 6, 2007, at 11:32 PM, Raghu Angadi wrote:
>
Re: hard link count for a file
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Raghu,
LGPL not allowed in distro:
http://people.apache.org/~cliffs/3party.html
Nige
On Nov 6, 2007, at 11:32 PM, Raghu Angadi wrote:
>
> A Java incubator project 'Java Native Access' looks promising :
> https://jna.dev.java.net/
>
> Can we include LGPL jar in Hadoop?
>
> From examples, code for accessing atol() from libc looks very simple :
>
> https://jna.dev.java.net/source/browse/jna/trunk/jnalib/src/com/sun/
> jna/examples/CLibrary.java?rev=7&view=markup
>
> Raghu.
>
> Raghu Angadi wrote:
>> > I think you'd need to exec a shell command and parse the output.
>> We might need to invoke this for a lot of blocks at run time.
>> The use case is that datanode needs to know if a block is also
>> linked by a back up (after start-dfs.sh -upgrade).
>> The current use case is when we want to update block metadata when
>> datanode verifies a block (HADOOP-2012). Shell command might fine
>> for this. Later this might be used for appends.
>> Raghu.
>> Doug Cutting wrote:
>>> Raghu Angadi wrote:
>>>> Is there a way to find hard link count for a file in Java? (id
>>>> 'st_nlink' in struct stat).
>>>
>>> I think you'd need to exec a shell command and parse the output.
>>>
>>> You could add this to FileUtil.java...
>>>
>>> Doug
>
Re: hard link count for a file
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
A Java incubator project 'Java Native Access' looks promising :
https://jna.dev.java.net/
Can we include LGPL jar in Hadoop?
From examples, code for accessing atol() from libc looks very simple :
https://jna.dev.java.net/source/browse/jna/trunk/jnalib/src/com/sun/jna/examples/CLibrary.java?rev=7&view=markup
Raghu.
Raghu Angadi wrote:
>
> > I think you'd need to exec a shell command and parse the output.
>
> We might need to invoke this for a lot of blocks at run time.
>
> The use case is that datanode needs to know if a block is also linked by
> a back up (after start-dfs.sh -upgrade).
>
> The current use case is when we want to update block metadata when
> datanode verifies a block (HADOOP-2012). Shell command might fine for
> this. Later this might be used for appends.
>
> Raghu.
>
> Doug Cutting wrote:
>> Raghu Angadi wrote:
>>> Is there a way to find hard link count for a file in Java? (id
>>> 'st_nlink' in struct stat).
>>
>> I think you'd need to exec a shell command and parse the output.
>>
>> You could add this to FileUtil.java...
>>
>> Doug
>
Re: hard link count for a file
Posted by Raghu Angadi <ra...@yahoo-inc.com>.
> I think you'd need to exec a shell command and parse the output.
We might need to invoke this for a lot of blocks at run time.
The use case is that datanode needs to know if a block is also linked by
a back up (after start-dfs.sh -upgrade).
The current use case is when we want to update block metadata when
datanode verifies a block (HADOOP-2012). Shell command might fine for
this. Later this might be used for appends.
Raghu.
Doug Cutting wrote:
> Raghu Angadi wrote:
>> Is there a way to find hard link count for a file in Java? (id
>> 'st_nlink' in struct stat).
>
> I think you'd need to exec a shell command and parse the output.
>
> You could add this to FileUtil.java...
>
> Doug
Re: hard link count for a file
Posted by Doug Cutting <cu...@apache.org>.
Raghu Angadi wrote:
> Is there a way to find hard link count for a file in Java? (id
> 'st_nlink' in struct stat).
I think you'd need to exec a shell command and parse the output.
You could add this to FileUtil.java...
Doug