You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bhupesh Bansal <bb...@linkedin.com> on 2009/10/15 22:36:48 UTC
Error in FileSystem.get()
Hey Folks,
I am seeing a very weird problem in FileSystem.get(Configuration).
I want to get a FileSystem given the configuration, so I am using
Configuration conf = new Configuration();
_fs = FileSystem.get(conf);
The problem is I am getting LocalFileSystem on some machines and Distributed
on others. I am printing conf.get("fs.default.name") at all places and
It returns the right HDFS value 'hdfs://dummy:9000'
My expectation is looking at fs.default.name if it is hdfs:// it should give
me a DistributedFileSystem always.
Best
Bhupesh
Re: Error in FileSystem.get()
Posted by Aaron Kimball <aa...@cloudera.com>.
Bhupesh: If you use FileSystem.newInstance(), does that return the correct
object type? This sidesteps CACHE.
- A
On Thu, Oct 15, 2009 at 3:07 PM, Bhupesh Bansal <bb...@linkedin.com>wrote:
> This code is not map/reduce code and run only on single machine and
> Also each node prints the right value for "fs.default.name" so it is
> Reading the right configuration file too .. The issue looks like use of
> CACHE in filesystem and someplace my code is setting up a wrong value
> If that is possible.
>
> Best
> Bhupesh
>
>
>
>
> On 10/15/09 1:46 PM, "Ashutosh Chauhan" <as...@gmail.com>
> wrote:
>
> > Each node reads its own conf files (mapred-site.xml, hdfs-site.xml etc.)
> > Make sure your configs are consistent on all nodes across entire cluster
> and
> > are pointing to correct fs.
> >
> > Hope it helps,
> > Ashutosh
> >
> > On Thu, Oct 15, 2009 at 16:36, Bhupesh Bansal <bb...@linkedin.com>
> wrote:
> >
> >> Hey Folks,
> >>
> >> I am seeing a very weird problem in FileSystem.get(Configuration).
> >>
> >> I want to get a FileSystem given the configuration, so I am using
> >>
> >> Configuration conf = new Configuration();
> >>
> >> _fs = FileSystem.get(conf);
> >>
> >>
> >> The problem is I am getting LocalFileSystem on some machines and
> >> Distributed
> >> on others. I am printing conf.get("fs.default.name") at all places and
> >> It returns the right HDFS value 'hdfs://dummy:9000'
> >>
> >> My expectation is looking at fs.default.name if it is hdfs:// it should
> >> give
> >> me a DistributedFileSystem always.
> >>
> >> Best
> >> Bhupesh
> >>
> >>
> >>
> >>
> >>
> >>
>
>
Re: Error in FileSystem.get()
Posted by Bhupesh Bansal <bb...@linkedin.com>.
This code is not map/reduce code and run only on single machine and
Also each node prints the right value for "fs.default.name" so it is
Reading the right configuration file too .. The issue looks like use of
CACHE in filesystem and someplace my code is setting up a wrong value
If that is possible.
Best
Bhupesh
On 10/15/09 1:46 PM, "Ashutosh Chauhan" <as...@gmail.com> wrote:
> Each node reads its own conf files (mapred-site.xml, hdfs-site.xml etc.)
> Make sure your configs are consistent on all nodes across entire cluster and
> are pointing to correct fs.
>
> Hope it helps,
> Ashutosh
>
> On Thu, Oct 15, 2009 at 16:36, Bhupesh Bansal <bb...@linkedin.com> wrote:
>
>> Hey Folks,
>>
>> I am seeing a very weird problem in FileSystem.get(Configuration).
>>
>> I want to get a FileSystem given the configuration, so I am using
>>
>> Configuration conf = new Configuration();
>>
>> _fs = FileSystem.get(conf);
>>
>>
>> The problem is I am getting LocalFileSystem on some machines and
>> Distributed
>> on others. I am printing conf.get("fs.default.name") at all places and
>> It returns the right HDFS value 'hdfs://dummy:9000'
>>
>> My expectation is looking at fs.default.name if it is hdfs:// it should
>> give
>> me a DistributedFileSystem always.
>>
>> Best
>> Bhupesh
>>
>>
>>
>>
>>
>>
Re: Error in FileSystem.get()
Posted by Ashutosh Chauhan <as...@gmail.com>.
Each node reads its own conf files (mapred-site.xml, hdfs-site.xml etc.)
Make sure your configs are consistent on all nodes across entire cluster and
are pointing to correct fs.
Hope it helps,
Ashutosh
On Thu, Oct 15, 2009 at 16:36, Bhupesh Bansal <bb...@linkedin.com> wrote:
> Hey Folks,
>
> I am seeing a very weird problem in FileSystem.get(Configuration).
>
> I want to get a FileSystem given the configuration, so I am using
>
> Configuration conf = new Configuration();
>
> _fs = FileSystem.get(conf);
>
>
> The problem is I am getting LocalFileSystem on some machines and
> Distributed
> on others. I am printing conf.get("fs.default.name") at all places and
> It returns the right HDFS value 'hdfs://dummy:9000'
>
> My expectation is looking at fs.default.name if it is hdfs:// it should
> give
> me a DistributedFileSystem always.
>
> Best
> Bhupesh
>
>
>
>
>
>