You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by jiang licht <li...@yahoo.com> on 2010/03/10 01:10:13 UTC
Symbolic link in Hadoop HDFS?
Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks!
Michael
Re: Symbolic link in Hadoop HDFS?
Posted by Eli Collins <el...@cloudera.com>.
Yup, it needs to be ported to FileContext first (HADOOP-6424).
Thanks,
Eli
On Wed, Mar 10, 2010 at 9:47 AM, jiang licht <li...@yahoo.com> wrote:
> Thanks, Eli. I c. Any plan to support symlink in hadoop fs shell? That'll make life easier instead of everyone writing similar codes. With that support in shell commands, with one physical copy of a big data set, different ppl can easily create different subset of it to work on by using symlinks.
>
> Thanks,
>
> Michael
>
> --- On Wed, 3/10/10, Eli Collins <el...@cloudera.com> wrote:
>
> From: Eli Collins <el...@cloudera.com>
> Subject: Re: Symbolic link in Hadoop HDFS?
> To: common-user@hadoop.apache.org
> Date: Wednesday, March 10, 2010, 11:08 AM
>
> Those patches won't apply to an existing release so you'd need to use trunk
> to get symlinks. They will be included in the next release, however only via
> the FileContext API.
>
> Thanks,
> Eli
>
>
> On Tue, Mar 9, 2010 at 6:16 PM, jiang licht <li...@yahoo.com> wrote:
>
>> Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as
>> I understand, symlink is available by using some relevant patches, right?
>> From what you mentioned, it seems to me that symlink support is quite stable
>> as of now. Just curious, why not included in a recent hadoop release yet or
>> will it be considered in the next release?
>>
>> Thanks,
>>
>> Michael
>>
>> --- On Tue, 3/9/10, Eli Collins <el...@cloudera.com> wrote:
>>
>> From: Eli Collins <el...@cloudera.com>
>> Subject: Re: Symbolic link in Hadoop HDFS?
>> To: common-user@hadoop.apache.org
>> Date: Tuesday, March 9, 2010, 8:01 PM
>>
>> Hey Michael,
>>
>> Symbolic links has been implemented [1] but are not yet available in a
>> Hadoop release. The implementation is only available to clients that
>> use the new FileContext API so clients like Hive need to be migrated
>> from using FileSystem to FileContext. This is currently being done in
>> Hadoop itself [2].
>>
>> Thanks,
>> Eli
>>
>> [1] http://issues.apache.org/jira/browse/HDFS-245
>> [2] http://issues.apache.org/jira/browse/HADOOP-6446
>>
>>
>> On Tue, Mar 9, 2010 at 4:10 PM, jiang licht <li...@yahoo.com> wrote:
>> > Is there a way to create symlink in hdfs? And does LOAD function in Pig
>> follows such a link? Thanks!
>> >
>> >
>> > Michael
>> >
>> >
>> >
>>
>>
>>
>>
>>
>
>
>
>
Re: Symbolic link in Hadoop HDFS?
Posted by jiang licht <li...@yahoo.com>.
Thanks, Eli. I c. Any plan to support symlink in hadoop fs shell? That'll make life easier instead of everyone writing similar codes. With that support in shell commands, with one physical copy of a big data set, different ppl can easily create different subset of it to work on by using symlinks.
Thanks,
Michael
--- On Wed, 3/10/10, Eli Collins <el...@cloudera.com> wrote:
From: Eli Collins <el...@cloudera.com>
Subject: Re: Symbolic link in Hadoop HDFS?
To: common-user@hadoop.apache.org
Date: Wednesday, March 10, 2010, 11:08 AM
Those patches won't apply to an existing release so you'd need to use trunk
to get symlinks. They will be included in the next release, however only via
the FileContext API.
Thanks,
Eli
On Tue, Mar 9, 2010 at 6:16 PM, jiang licht <li...@yahoo.com> wrote:
> Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as
> I understand, symlink is available by using some relevant patches, right?
> From what you mentioned, it seems to me that symlink support is quite stable
> as of now. Just curious, why not included in a recent hadoop release yet or
> will it be considered in the next release?
>
> Thanks,
>
> Michael
>
> --- On Tue, 3/9/10, Eli Collins <el...@cloudera.com> wrote:
>
> From: Eli Collins <el...@cloudera.com>
> Subject: Re: Symbolic link in Hadoop HDFS?
> To: common-user@hadoop.apache.org
> Date: Tuesday, March 9, 2010, 8:01 PM
>
> Hey Michael,
>
> Symbolic links has been implemented [1] but are not yet available in a
> Hadoop release. The implementation is only available to clients that
> use the new FileContext API so clients like Hive need to be migrated
> from using FileSystem to FileContext. This is currently being done in
> Hadoop itself [2].
>
> Thanks,
> Eli
>
> [1] http://issues.apache.org/jira/browse/HDFS-245
> [2] http://issues.apache.org/jira/browse/HADOOP-6446
>
>
> On Tue, Mar 9, 2010 at 4:10 PM, jiang licht <li...@yahoo.com> wrote:
> > Is there a way to create symlink in hdfs? And does LOAD function in Pig
> follows such a link? Thanks!
> >
> >
> > Michael
> >
> >
> >
>
>
>
>
>
Re: Symbolic link in Hadoop HDFS?
Posted by Eli Collins <el...@cloudera.com>.
Those patches won't apply to an existing release so you'd need to use trunk
to get symlinks. They will be included in the next release, however only via
the FileContext API.
Thanks,
Eli
On Tue, Mar 9, 2010 at 6:16 PM, jiang licht <li...@yahoo.com> wrote:
> Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as
> I understand, symlink is available by using some relevant patches, right?
> From what you mentioned, it seems to me that symlink support is quite stable
> as of now. Just curious, why not included in a recent hadoop release yet or
> will it be considered in the next release?
>
> Thanks,
>
> Michael
>
> --- On Tue, 3/9/10, Eli Collins <el...@cloudera.com> wrote:
>
> From: Eli Collins <el...@cloudera.com>
> Subject: Re: Symbolic link in Hadoop HDFS?
> To: common-user@hadoop.apache.org
> Date: Tuesday, March 9, 2010, 8:01 PM
>
> Hey Michael,
>
> Symbolic links has been implemented [1] but are not yet available in a
> Hadoop release. The implementation is only available to clients that
> use the new FileContext API so clients like Hive need to be migrated
> from using FileSystem to FileContext. This is currently being done in
> Hadoop itself [2].
>
> Thanks,
> Eli
>
> [1] http://issues.apache.org/jira/browse/HDFS-245
> [2] http://issues.apache.org/jira/browse/HADOOP-6446
>
>
> On Tue, Mar 9, 2010 at 4:10 PM, jiang licht <li...@yahoo.com> wrote:
> > Is there a way to create symlink in hdfs? And does LOAD function in Pig
> follows such a link? Thanks!
> >
> >
> > Michael
> >
> >
> >
>
>
>
>
>
Re: Symbolic link in Hadoop HDFS?
Posted by jiang licht <li...@yahoo.com>.
Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as I understand, symlink is available by using some relevant patches, right? From what you mentioned, it seems to me that symlink support is quite stable as of now. Just curious, why not included in a recent hadoop release yet or will it be considered in the next release?
Thanks,
Michael
--- On Tue, 3/9/10, Eli Collins <el...@cloudera.com> wrote:
From: Eli Collins <el...@cloudera.com>
Subject: Re: Symbolic link in Hadoop HDFS?
To: common-user@hadoop.apache.org
Date: Tuesday, March 9, 2010, 8:01 PM
Hey Michael,
Symbolic links has been implemented [1] but are not yet available in a
Hadoop release. The implementation is only available to clients that
use the new FileContext API so clients like Hive need to be migrated
from using FileSystem to FileContext. This is currently being done in
Hadoop itself [2].
Thanks,
Eli
[1] http://issues.apache.org/jira/browse/HDFS-245
[2] http://issues.apache.org/jira/browse/HADOOP-6446
On Tue, Mar 9, 2010 at 4:10 PM, jiang licht <li...@yahoo.com> wrote:
> Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks!
>
>
> Michael
>
>
>
Re: Symbolic link in Hadoop HDFS?
Posted by Eli Collins <el...@cloudera.com>.
Hey Michael,
Symbolic links has been implemented [1] but are not yet available in a
Hadoop release. The implementation is only available to clients that
use the new FileContext API so clients like Hive need to be migrated
from using FileSystem to FileContext. This is currently being done in
Hadoop itself [2].
Thanks,
Eli
[1] http://issues.apache.org/jira/browse/HDFS-245
[2] http://issues.apache.org/jira/browse/HADOOP-6446
On Tue, Mar 9, 2010 at 4:10 PM, jiang licht <li...@yahoo.com> wrote:
> Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks!
>
>
> Michael
>
>
>