You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/01/17 01:33:15 UTC
core-site.xml file is being ignored by new Configuration()
Hi guys:
I've finally extracted my problem of loading a special filesystem
into a unit test.
Below, clearly, Im creating a raw configuration and adding a single
resource to it (core-site.xml).
Afterwards, i print to confirm that the file exists.
Finally, I try to create a FileSystem from that file. However, no file
system is created :(.
Any thoughts on why this core-site.xml file is being ignoreed?
To note: I've also tried putting core-site.xml in the working directory,
and in a local conf/ directory. This absolute path was a last effort to
really confirm that the local path wasn't somehow being mangled by the
junit runtime.
@Test
public void test() throws Exception{
Configuration conf = new Configuration(false);
System.out.println("b4:"+conf);
String
theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
conf.addResource(theXMLFile);
System.out.println("File exists : " + theXMLFile.length());
System.out.println("after:"+conf);
System.out.println("file sytem " + conf.get("fs.default.name"));
conf.writeXml(System.out);
System.out.println("Working directory = " +
System.getProperty("user.dir"));
org.apache.hadoop.fs.FileSystem fs =
org.apache.hadoop.fs.FileSystem.get(conf);
System.out.println("before test filesystem = " + fs);
//WordCount.main(new
String[]{"resources/wordcount.txt","wordcountout.txt"});
}
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Can you try prepending the path with file:// or pass in a URL and use the method which takes in a URL?
I remember class loading issues with paths without the file: scheme info from a different scenario (while playing with logj4 config files).
HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Jan 16, 2013, at 4:33 PM, Jay Vyas wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and in a local conf/ directory. This absolute path was a last effort to really confirm that the local path wasn't somehow being mangled by the junit runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " + System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
The configuration XML files are loaded from your CLASSPATH.
If you want to know more about what configuration you're picking up, you
can use the "./bin/hdfs getconf" command.
cheers,
Colin
On Thu, Jan 17, 2013 at 7:49 AM, Jay Vyas <ja...@gmail.com> wrote:
> I agree, but the fact that the configuration is doing the loading, means
> (i think) that it should (at least) do some error handling for that
> loading, correct?
>
>
> On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> imo, as long as the javadoc is clear enough, any behavior is ok.
>> What's written in current version is:
>> name - resource to be added, the classpath is examined for a file with
>> that name.
>>
>> So nothing in this javadoc lets you believe that the file exists or will
>> be loaded. Also you could make sure your file is in the classpath (which is
>> the case for me).
>>
>> Julien
>>
>> 2013/1/17 Jay Vyas <ja...@gmail.com>
>>
>>> Good catch with that string.length() - you're right, that was a silly
>>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>>> :)
>>>
>>> In any case, the same code with file.exists() fails... i've validated
>>> that path many ways.
>>>
>>> On a broader note: Shouldn't the Configuration class just throw a
>>> RuntimeException if the file isn't available? Its very strange to add
>>> all these default resources, which may or may not exist, to a class - then
>>> having to introspect on the class at runtime to see if they actually got
>>> loaded.
>>>
>>>
>>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>>
>>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Jay Vyas
>>> http://jayunit100.blogspot.com
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
The configuration XML files are loaded from your CLASSPATH.
If you want to know more about what configuration you're picking up, you
can use the "./bin/hdfs getconf" command.
cheers,
Colin
On Thu, Jan 17, 2013 at 7:49 AM, Jay Vyas <ja...@gmail.com> wrote:
> I agree, but the fact that the configuration is doing the loading, means
> (i think) that it should (at least) do some error handling for that
> loading, correct?
>
>
> On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> imo, as long as the javadoc is clear enough, any behavior is ok.
>> What's written in current version is:
>> name - resource to be added, the classpath is examined for a file with
>> that name.
>>
>> So nothing in this javadoc lets you believe that the file exists or will
>> be loaded. Also you could make sure your file is in the classpath (which is
>> the case for me).
>>
>> Julien
>>
>> 2013/1/17 Jay Vyas <ja...@gmail.com>
>>
>>> Good catch with that string.length() - you're right, that was a silly
>>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>>> :)
>>>
>>> In any case, the same code with file.exists() fails... i've validated
>>> that path many ways.
>>>
>>> On a broader note: Shouldn't the Configuration class just throw a
>>> RuntimeException if the file isn't available? Its very strange to add
>>> all these default resources, which may or may not exist, to a class - then
>>> having to introspect on the class at runtime to see if they actually got
>>> loaded.
>>>
>>>
>>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>>
>>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Jay Vyas
>>> http://jayunit100.blogspot.com
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
The configuration XML files are loaded from your CLASSPATH.
If you want to know more about what configuration you're picking up, you
can use the "./bin/hdfs getconf" command.
cheers,
Colin
On Thu, Jan 17, 2013 at 7:49 AM, Jay Vyas <ja...@gmail.com> wrote:
> I agree, but the fact that the configuration is doing the loading, means
> (i think) that it should (at least) do some error handling for that
> loading, correct?
>
>
> On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> imo, as long as the javadoc is clear enough, any behavior is ok.
>> What's written in current version is:
>> name - resource to be added, the classpath is examined for a file with
>> that name.
>>
>> So nothing in this javadoc lets you believe that the file exists or will
>> be loaded. Also you could make sure your file is in the classpath (which is
>> the case for me).
>>
>> Julien
>>
>> 2013/1/17 Jay Vyas <ja...@gmail.com>
>>
>>> Good catch with that string.length() - you're right, that was a silly
>>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>>> :)
>>>
>>> In any case, the same code with file.exists() fails... i've validated
>>> that path many ways.
>>>
>>> On a broader note: Shouldn't the Configuration class just throw a
>>> RuntimeException if the file isn't available? Its very strange to add
>>> all these default resources, which may or may not exist, to a class - then
>>> having to introspect on the class at runtime to see if they actually got
>>> loaded.
>>>
>>>
>>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>>
>>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Jay Vyas
>>> http://jayunit100.blogspot.com
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
The configuration XML files are loaded from your CLASSPATH.
If you want to know more about what configuration you're picking up, you
can use the "./bin/hdfs getconf" command.
cheers,
Colin
On Thu, Jan 17, 2013 at 7:49 AM, Jay Vyas <ja...@gmail.com> wrote:
> I agree, but the fact that the configuration is doing the loading, means
> (i think) that it should (at least) do some error handling for that
> loading, correct?
>
>
> On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> imo, as long as the javadoc is clear enough, any behavior is ok.
>> What's written in current version is:
>> name - resource to be added, the classpath is examined for a file with
>> that name.
>>
>> So nothing in this javadoc lets you believe that the file exists or will
>> be loaded. Also you could make sure your file is in the classpath (which is
>> the case for me).
>>
>> Julien
>>
>> 2013/1/17 Jay Vyas <ja...@gmail.com>
>>
>>> Good catch with that string.length() - you're right, that was a silly
>>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>>> :)
>>>
>>> In any case, the same code with file.exists() fails... i've validated
>>> that path many ways.
>>>
>>> On a broader note: Shouldn't the Configuration class just throw a
>>> RuntimeException if the file isn't available? Its very strange to add
>>> all these default resources, which may or may not exist, to a class - then
>>> having to introspect on the class at runtime to see if they actually got
>>> loaded.
>>>
>>>
>>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>>
>>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Jay Vyas
>>> http://jayunit100.blogspot.com
>>>
>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
I agree, but the fact that the configuration is doing the loading, means (i
think) that it should (at least) do some error handling for that loading,
correct?
On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
> imo, as long as the javadoc is clear enough, any behavior is ok.
> What's written in current version is:
> name - resource to be added, the classpath is examined for a file with
> that name.
>
> So nothing in this javadoc lets you believe that the file exists or will
> be loaded. Also you could make sure your file is in the classpath (which is
> the case for me).
>
> Julien
>
> 2013/1/17 Jay Vyas <ja...@gmail.com>
>
>> Good catch with that string.length() - you're right, that was a silly
>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>> :)
>>
>> In any case, the same code with file.exists() fails... i've validated
>> that path many ways.
>>
>> On a broader note: Shouldn't the Configuration class just throw a
>> RuntimeException if the file isn't available? Its very strange to add
>> all these default resources, which may or may not exist, to a class - then
>> having to introspect on the class at runtime to see if they actually got
>> loaded.
>>
>>
>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>
>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
I agree, but the fact that the configuration is doing the loading, means (i
think) that it should (at least) do some error handling for that loading,
correct?
On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
> imo, as long as the javadoc is clear enough, any behavior is ok.
> What's written in current version is:
> name - resource to be added, the classpath is examined for a file with
> that name.
>
> So nothing in this javadoc lets you believe that the file exists or will
> be loaded. Also you could make sure your file is in the classpath (which is
> the case for me).
>
> Julien
>
> 2013/1/17 Jay Vyas <ja...@gmail.com>
>
>> Good catch with that string.length() - you're right, that was a silly
>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>> :)
>>
>> In any case, the same code with file.exists() fails... i've validated
>> that path many ways.
>>
>> On a broader note: Shouldn't the Configuration class just throw a
>> RuntimeException if the file isn't available? Its very strange to add
>> all these default resources, which may or may not exist, to a class - then
>> having to introspect on the class at runtime to see if they actually got
>> loaded.
>>
>>
>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>
>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
I agree, but the fact that the configuration is doing the loading, means (i
think) that it should (at least) do some error handling for that loading,
correct?
On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
> imo, as long as the javadoc is clear enough, any behavior is ok.
> What's written in current version is:
> name - resource to be added, the classpath is examined for a file with
> that name.
>
> So nothing in this javadoc lets you believe that the file exists or will
> be loaded. Also you could make sure your file is in the classpath (which is
> the case for me).
>
> Julien
>
> 2013/1/17 Jay Vyas <ja...@gmail.com>
>
>> Good catch with that string.length() - you're right, that was a silly
>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>> :)
>>
>> In any case, the same code with file.exists() fails... i've validated
>> that path many ways.
>>
>> On a broader note: Shouldn't the Configuration class just throw a
>> RuntimeException if the file isn't available? Its very strange to add
>> all these default resources, which may or may not exist, to a class - then
>> having to introspect on the class at runtime to see if they actually got
>> loaded.
>>
>>
>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>
>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
I agree, but the fact that the configuration is doing the loading, means (i
think) that it should (at least) do some error handling for that loading,
correct?
On Thu, Jan 17, 2013 at 10:36 AM, Julien Muller <ju...@ezako.com>wrote:
> imo, as long as the javadoc is clear enough, any behavior is ok.
> What's written in current version is:
> name - resource to be added, the classpath is examined for a file with
> that name.
>
> So nothing in this javadoc lets you believe that the file exists or will
> be loaded. Also you could make sure your file is in the classpath (which is
> the case for me).
>
> Julien
>
> 2013/1/17 Jay Vyas <ja...@gmail.com>
>
>> Good catch with that string.length() - you're right, that was a silly
>> mistake. --- sorry - im not sure what i was thinking. it was a late night
>> :)
>>
>> In any case, the same code with file.exists() fails... i've validated
>> that path many ways.
>>
>> On a broader note: Shouldn't the Configuration class just throw a
>> RuntimeException if the file isn't available? Its very strange to add
>> all these default resources, which may or may not exist, to a class - then
>> having to introspect on the class at runtime to see if they actually got
>> loaded.
>>
>>
>> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>>
>>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>>
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>>
>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
imo, as long as the javadoc is clear enough, any behavior is ok.
What's written in current version is:
name - resource to be added, the classpath is examined for a file with that
name.
So nothing in this javadoc lets you believe that the file exists or will be
loaded. Also you could make sure your file is in the classpath (which is
the case for me).
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> Good catch with that string.length() - you're right, that was a silly
> mistake. --- sorry - im not sure what i was thinking. it was a late night
> :)
>
> In any case, the same code with file.exists() fails... i've validated
> that path many ways.
>
> On a broader note: Shouldn't the Configuration class just throw a
> RuntimeException if the file isn't available? Its very strange to add
> all these default resources, which may or may not exist, to a class - then
> having to introspect on the class at runtime to see if they actually got
> loaded.
>
>
> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
imo, as long as the javadoc is clear enough, any behavior is ok.
What's written in current version is:
name - resource to be added, the classpath is examined for a file with that
name.
So nothing in this javadoc lets you believe that the file exists or will be
loaded. Also you could make sure your file is in the classpath (which is
the case for me).
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> Good catch with that string.length() - you're right, that was a silly
> mistake. --- sorry - im not sure what i was thinking. it was a late night
> :)
>
> In any case, the same code with file.exists() fails... i've validated
> that path many ways.
>
> On a broader note: Shouldn't the Configuration class just throw a
> RuntimeException if the file isn't available? Its very strange to add
> all these default resources, which may or may not exist, to a class - then
> having to introspect on the class at runtime to see if they actually got
> loaded.
>
>
> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
imo, as long as the javadoc is clear enough, any behavior is ok.
What's written in current version is:
name - resource to be added, the classpath is examined for a file with that
name.
So nothing in this javadoc lets you believe that the file exists or will be
loaded. Also you could make sure your file is in the classpath (which is
the case for me).
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> Good catch with that string.length() - you're right, that was a silly
> mistake. --- sorry - im not sure what i was thinking. it was a late night
> :)
>
> In any case, the same code with file.exists() fails... i've validated
> that path many ways.
>
> On a broader note: Shouldn't the Configuration class just throw a
> RuntimeException if the file isn't available? Its very strange to add
> all these default resources, which may or may not exist, to a class - then
> having to introspect on the class at runtime to see if they actually got
> loaded.
>
>
> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
imo, as long as the javadoc is clear enough, any behavior is ok.
What's written in current version is:
name - resource to be added, the classpath is examined for a file with that
name.
So nothing in this javadoc lets you believe that the file exists or will be
loaded. Also you could make sure your file is in the classpath (which is
the case for me).
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> Good catch with that string.length() - you're right, that was a silly
> mistake. --- sorry - im not sure what i was thinking. it was a late night
> :)
>
> In any case, the same code with file.exists() fails... i've validated
> that path many ways.
>
> On a broader note: Shouldn't the Configuration class just throw a
> RuntimeException if the file isn't available? Its very strange to add
> all these default resources, which may or may not exist, to a class - then
> having to introspect on the class at runtime to see if they actually got
> loaded.
>
>
> On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
>
>> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>>
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
Good catch with that string.length() - you're right, that was a silly
mistake. --- sorry - im not sure what i was thinking. it was a late night
:)
In any case, the same code with file.exists() fails... i've validated that
path many ways.
On a broader note: Shouldn't the Configuration class just throw a
RuntimeException if the file isn't available? Its very strange to add
all these default resources, which may or may not exist, to a class - then
having to introspect on the class at runtime to see if they actually got
loaded.
On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
Good catch with that string.length() - you're right, that was a silly
mistake. --- sorry - im not sure what i was thinking. it was a late night
:)
In any case, the same code with file.exists() fails... i've validated that
path many ways.
On a broader note: Shouldn't the Configuration class just throw a
RuntimeException if the file isn't available? Its very strange to add
all these default resources, which may or may not exist, to a class - then
having to introspect on the class at runtime to see if they actually got
loaded.
On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
Good catch with that string.length() - you're right, that was a silly
mistake. --- sorry - im not sure what i was thinking. it was a late night
:)
In any case, the same code with file.exists() fails... i've validated that
path many ways.
On a broader note: Shouldn't the Configuration class just throw a
RuntimeException if the file isn't available? Its very strange to add
all these default resources, which may or may not exist, to a class - then
having to introspect on the class at runtime to see if they actually got
loaded.
On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
Good catch with that string.length() - you're right, that was a silly
mistake. --- sorry - im not sure what i was thinking. it was a late night
:)
In any case, the same code with file.exists() fails... i've validated that
path many ways.
On a broader note: Shouldn't the Configuration class just throw a
RuntimeException if the file isn't available? Its very strange to add
all these default resources, which may or may not exist, to a class - then
having to introspect on the class at runtime to see if they actually got
loaded.
On Thu, Jan 17, 2013 at 5:06 AM, Julien Muller <ju...@ezako.com>wrote:
> conf.addResource(file.getAbsoluteFile().toURI().toURL());
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
I am not sure you actually test if the file exists in your code:
System.out.println("File exists : " + theXMLFile.length());
This is really only the file path.
You could try to load the file to make sure:
File file = new File(theXMLFile);
// Test if exists
// ...
// Load in ressource, I had issues loading confs and found this way is
working in all my cases, including custom FS in fs.default.name
conf.addResource(file.getAbsoluteFile().toURI().toURL());
(This is not really optimized but I believe it is ok for a JUnit setup
phase)
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> no difference.
>
> This is really odd.
>
> The only way I can get it to work is by the following:
>
>
> /**
> * A simple test harness for glusterfs
> * @author jvyas
> */
> public class TestMapReducerGlusterFS {
>
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("before reload:"+conf);
> //necessary?
> conf.set("fs.default.name", ".......");
> conf.set("fs.blah.mount", "/mnt/......");
> conf.set("fs.blahserver", "192.168.1.36");
> conf.set("iomod","Off");
> ...
> }
> }
>
> Thinking that this is a little known issue in the current hadoop
> parser/configurator? because very rarely do people write unit tests which
> utilize abnormal ( i.e. non-local or non-hdfs) file systems.
>
>
> On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
>
>> Result any different if you call reloadConfiguration()?
>>
>> Thanks,
>>
>> Tom
>>
>> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
>> > Hi guys:
>> >
>> > I've finally extracted my problem of loading a special filesystem
>> > into a unit test.
>> >
>> > Below, clearly, Im creating a raw configuration and adding a single
>> resource
>> > to it (core-site.xml).
>> >
>> > Afterwards, i print to confirm that the file exists.
>> >
>> > Finally, I try to create a FileSystem from that file. However, no file
>> > system is created :(.
>> >
>> > Any thoughts on why this core-site.xml file is being ignoreed?
>> > To note: I've also tried putting core-site.xml in the working
>> directory, and
>> > in a local conf/ directory. This absolute path was a last effort to
>> really
>> > confirm that the local path wasn't somehow being mangled by the junit
>> > runtime.
>> >
>> > @Test
>> > public void test() throws Exception{
>> > Configuration conf = new Configuration(false);
>> > System.out.println("b4:"+conf);
>> > String
>> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>> > conf.addResource(theXMLFile);
>> > System.out.println("File exists : " + theXMLFile.length());
>> > System.out.println("after:"+conf);
>> > System.out.println("file sytem " + conf.get("fs.default.name
>> "));
>> > conf.writeXml(System.out);
>> >
>> >
>> > System.out.println("Working directory = " +
>> > System.getProperty("user.dir"));
>> > org.apache.hadoop.fs.FileSystem fs =
>> > org.apache.hadoop.fs.FileSystem.get(conf);
>> > System.out.println("before test filesystem = " + fs);
>> > //WordCount.main(new
>> > String[]{"resources/wordcount.txt","wordcountout.txt"});
>> > }
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
I am not sure you actually test if the file exists in your code:
System.out.println("File exists : " + theXMLFile.length());
This is really only the file path.
You could try to load the file to make sure:
File file = new File(theXMLFile);
// Test if exists
// ...
// Load in ressource, I had issues loading confs and found this way is
working in all my cases, including custom FS in fs.default.name
conf.addResource(file.getAbsoluteFile().toURI().toURL());
(This is not really optimized but I believe it is ok for a JUnit setup
phase)
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> no difference.
>
> This is really odd.
>
> The only way I can get it to work is by the following:
>
>
> /**
> * A simple test harness for glusterfs
> * @author jvyas
> */
> public class TestMapReducerGlusterFS {
>
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("before reload:"+conf);
> //necessary?
> conf.set("fs.default.name", ".......");
> conf.set("fs.blah.mount", "/mnt/......");
> conf.set("fs.blahserver", "192.168.1.36");
> conf.set("iomod","Off");
> ...
> }
> }
>
> Thinking that this is a little known issue in the current hadoop
> parser/configurator? because very rarely do people write unit tests which
> utilize abnormal ( i.e. non-local or non-hdfs) file systems.
>
>
> On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
>
>> Result any different if you call reloadConfiguration()?
>>
>> Thanks,
>>
>> Tom
>>
>> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
>> > Hi guys:
>> >
>> > I've finally extracted my problem of loading a special filesystem
>> > into a unit test.
>> >
>> > Below, clearly, Im creating a raw configuration and adding a single
>> resource
>> > to it (core-site.xml).
>> >
>> > Afterwards, i print to confirm that the file exists.
>> >
>> > Finally, I try to create a FileSystem from that file. However, no file
>> > system is created :(.
>> >
>> > Any thoughts on why this core-site.xml file is being ignoreed?
>> > To note: I've also tried putting core-site.xml in the working
>> directory, and
>> > in a local conf/ directory. This absolute path was a last effort to
>> really
>> > confirm that the local path wasn't somehow being mangled by the junit
>> > runtime.
>> >
>> > @Test
>> > public void test() throws Exception{
>> > Configuration conf = new Configuration(false);
>> > System.out.println("b4:"+conf);
>> > String
>> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>> > conf.addResource(theXMLFile);
>> > System.out.println("File exists : " + theXMLFile.length());
>> > System.out.println("after:"+conf);
>> > System.out.println("file sytem " + conf.get("fs.default.name
>> "));
>> > conf.writeXml(System.out);
>> >
>> >
>> > System.out.println("Working directory = " +
>> > System.getProperty("user.dir"));
>> > org.apache.hadoop.fs.FileSystem fs =
>> > org.apache.hadoop.fs.FileSystem.get(conf);
>> > System.out.println("before test filesystem = " + fs);
>> > //WordCount.main(new
>> > String[]{"resources/wordcount.txt","wordcountout.txt"});
>> > }
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
I am not sure you actually test if the file exists in your code:
System.out.println("File exists : " + theXMLFile.length());
This is really only the file path.
You could try to load the file to make sure:
File file = new File(theXMLFile);
// Test if exists
// ...
// Load in ressource, I had issues loading confs and found this way is
working in all my cases, including custom FS in fs.default.name
conf.addResource(file.getAbsoluteFile().toURI().toURL());
(This is not really optimized but I believe it is ok for a JUnit setup
phase)
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> no difference.
>
> This is really odd.
>
> The only way I can get it to work is by the following:
>
>
> /**
> * A simple test harness for glusterfs
> * @author jvyas
> */
> public class TestMapReducerGlusterFS {
>
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("before reload:"+conf);
> //necessary?
> conf.set("fs.default.name", ".......");
> conf.set("fs.blah.mount", "/mnt/......");
> conf.set("fs.blahserver", "192.168.1.36");
> conf.set("iomod","Off");
> ...
> }
> }
>
> Thinking that this is a little known issue in the current hadoop
> parser/configurator? because very rarely do people write unit tests which
> utilize abnormal ( i.e. non-local or non-hdfs) file systems.
>
>
> On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
>
>> Result any different if you call reloadConfiguration()?
>>
>> Thanks,
>>
>> Tom
>>
>> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
>> > Hi guys:
>> >
>> > I've finally extracted my problem of loading a special filesystem
>> > into a unit test.
>> >
>> > Below, clearly, Im creating a raw configuration and adding a single
>> resource
>> > to it (core-site.xml).
>> >
>> > Afterwards, i print to confirm that the file exists.
>> >
>> > Finally, I try to create a FileSystem from that file. However, no file
>> > system is created :(.
>> >
>> > Any thoughts on why this core-site.xml file is being ignoreed?
>> > To note: I've also tried putting core-site.xml in the working
>> directory, and
>> > in a local conf/ directory. This absolute path was a last effort to
>> really
>> > confirm that the local path wasn't somehow being mangled by the junit
>> > runtime.
>> >
>> > @Test
>> > public void test() throws Exception{
>> > Configuration conf = new Configuration(false);
>> > System.out.println("b4:"+conf);
>> > String
>> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>> > conf.addResource(theXMLFile);
>> > System.out.println("File exists : " + theXMLFile.length());
>> > System.out.println("after:"+conf);
>> > System.out.println("file sytem " + conf.get("fs.default.name
>> "));
>> > conf.writeXml(System.out);
>> >
>> >
>> > System.out.println("Working directory = " +
>> > System.getProperty("user.dir"));
>> > org.apache.hadoop.fs.FileSystem fs =
>> > org.apache.hadoop.fs.FileSystem.get(conf);
>> > System.out.println("before test filesystem = " + fs);
>> > //WordCount.main(new
>> > String[]{"resources/wordcount.txt","wordcountout.txt"});
>> > }
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Julien Muller <ju...@ezako.com>.
I am not sure you actually test if the file exists in your code:
System.out.println("File exists : " + theXMLFile.length());
This is really only the file path.
You could try to load the file to make sure:
File file = new File(theXMLFile);
// Test if exists
// ...
// Load in ressource, I had issues loading confs and found this way is
working in all my cases, including custom FS in fs.default.name
conf.addResource(file.getAbsoluteFile().toURI().toURL());
(This is not really optimized but I believe it is ok for a JUnit setup
phase)
Julien
2013/1/17 Jay Vyas <ja...@gmail.com>
> no difference.
>
> This is really odd.
>
> The only way I can get it to work is by the following:
>
>
> /**
> * A simple test harness for glusterfs
> * @author jvyas
> */
> public class TestMapReducerGlusterFS {
>
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("before reload:"+conf);
> //necessary?
> conf.set("fs.default.name", ".......");
> conf.set("fs.blah.mount", "/mnt/......");
> conf.set("fs.blahserver", "192.168.1.36");
> conf.set("iomod","Off");
> ...
> }
> }
>
> Thinking that this is a little known issue in the current hadoop
> parser/configurator? because very rarely do people write unit tests which
> utilize abnormal ( i.e. non-local or non-hdfs) file systems.
>
>
> On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
>
>> Result any different if you call reloadConfiguration()?
>>
>> Thanks,
>>
>> Tom
>>
>> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
>> > Hi guys:
>> >
>> > I've finally extracted my problem of loading a special filesystem
>> > into a unit test.
>> >
>> > Below, clearly, Im creating a raw configuration and adding a single
>> resource
>> > to it (core-site.xml).
>> >
>> > Afterwards, i print to confirm that the file exists.
>> >
>> > Finally, I try to create a FileSystem from that file. However, no file
>> > system is created :(.
>> >
>> > Any thoughts on why this core-site.xml file is being ignoreed?
>> > To note: I've also tried putting core-site.xml in the working
>> directory, and
>> > in a local conf/ directory. This absolute path was a last effort to
>> really
>> > confirm that the local path wasn't somehow being mangled by the junit
>> > runtime.
>> >
>> > @Test
>> > public void test() throws Exception{
>> > Configuration conf = new Configuration(false);
>> > System.out.println("b4:"+conf);
>> > String
>> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
>> > conf.addResource(theXMLFile);
>> > System.out.println("File exists : " + theXMLFile.length());
>> > System.out.println("after:"+conf);
>> > System.out.println("file sytem " + conf.get("fs.default.name
>> "));
>> > conf.writeXml(System.out);
>> >
>> >
>> > System.out.println("Working directory = " +
>> > System.getProperty("user.dir"));
>> > org.apache.hadoop.fs.FileSystem fs =
>> > org.apache.hadoop.fs.FileSystem.get(conf);
>> > System.out.println("before test filesystem = " + fs);
>> > //WordCount.main(new
>> > String[]{"resources/wordcount.txt","wordcountout.txt"});
>> > }
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
>>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
no difference.
This is really odd.
The only way I can get it to work is by the following:
/**
* A simple test harness for glusterfs
* @author jvyas
*/
public class TestMapReducerGlusterFS {
@Test
public void test() throws Exception{
Configuration conf = new Configuration(false);
System.out.println("before reload:"+conf);
//necessary?
conf.set("fs.default.name", ".......");
conf.set("fs.blah.mount", "/mnt/......");
conf.set("fs.blahserver", "192.168.1.36");
conf.set("iomod","Off");
...
}
}
Thinking that this is a little known issue in the current hadoop
parser/configurator? because very rarely do people write unit tests which
utilize abnormal ( i.e. non-local or non-hdfs) file systems.
On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
> Result any different if you call reloadConfiguration()?
>
> Thanks,
>
> Tom
>
> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Hi guys:
> >
> > I've finally extracted my problem of loading a special filesystem
> > into a unit test.
> >
> > Below, clearly, Im creating a raw configuration and adding a single
> resource
> > to it (core-site.xml).
> >
> > Afterwards, i print to confirm that the file exists.
> >
> > Finally, I try to create a FileSystem from that file. However, no file
> > system is created :(.
> >
> > Any thoughts on why this core-site.xml file is being ignoreed?
> > To note: I've also tried putting core-site.xml in the working directory,
> and
> > in a local conf/ directory. This absolute path was a last effort to
> really
> > confirm that the local path wasn't somehow being mangled by the junit
> > runtime.
> >
> > @Test
> > public void test() throws Exception{
> > Configuration conf = new Configuration(false);
> > System.out.println("b4:"+conf);
> > String
> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> > conf.addResource(theXMLFile);
> > System.out.println("File exists : " + theXMLFile.length());
> > System.out.println("after:"+conf);
> > System.out.println("file sytem " + conf.get("fs.default.name"));
> > conf.writeXml(System.out);
> >
> >
> > System.out.println("Working directory = " +
> > System.getProperty("user.dir"));
> > org.apache.hadoop.fs.FileSystem fs =
> > org.apache.hadoop.fs.FileSystem.get(conf);
> > System.out.println("before test filesystem = " + fs);
> > //WordCount.main(new
> > String[]{"resources/wordcount.txt","wordcountout.txt"});
> > }
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
no difference.
This is really odd.
The only way I can get it to work is by the following:
/**
* A simple test harness for glusterfs
* @author jvyas
*/
public class TestMapReducerGlusterFS {
@Test
public void test() throws Exception{
Configuration conf = new Configuration(false);
System.out.println("before reload:"+conf);
//necessary?
conf.set("fs.default.name", ".......");
conf.set("fs.blah.mount", "/mnt/......");
conf.set("fs.blahserver", "192.168.1.36");
conf.set("iomod","Off");
...
}
}
Thinking that this is a little known issue in the current hadoop
parser/configurator? because very rarely do people write unit tests which
utilize abnormal ( i.e. non-local or non-hdfs) file systems.
On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
> Result any different if you call reloadConfiguration()?
>
> Thanks,
>
> Tom
>
> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Hi guys:
> >
> > I've finally extracted my problem of loading a special filesystem
> > into a unit test.
> >
> > Below, clearly, Im creating a raw configuration and adding a single
> resource
> > to it (core-site.xml).
> >
> > Afterwards, i print to confirm that the file exists.
> >
> > Finally, I try to create a FileSystem from that file. However, no file
> > system is created :(.
> >
> > Any thoughts on why this core-site.xml file is being ignoreed?
> > To note: I've also tried putting core-site.xml in the working directory,
> and
> > in a local conf/ directory. This absolute path was a last effort to
> really
> > confirm that the local path wasn't somehow being mangled by the junit
> > runtime.
> >
> > @Test
> > public void test() throws Exception{
> > Configuration conf = new Configuration(false);
> > System.out.println("b4:"+conf);
> > String
> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> > conf.addResource(theXMLFile);
> > System.out.println("File exists : " + theXMLFile.length());
> > System.out.println("after:"+conf);
> > System.out.println("file sytem " + conf.get("fs.default.name"));
> > conf.writeXml(System.out);
> >
> >
> > System.out.println("Working directory = " +
> > System.getProperty("user.dir"));
> > org.apache.hadoop.fs.FileSystem fs =
> > org.apache.hadoop.fs.FileSystem.get(conf);
> > System.out.println("before test filesystem = " + fs);
> > //WordCount.main(new
> > String[]{"resources/wordcount.txt","wordcountout.txt"});
> > }
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
no difference.
This is really odd.
The only way I can get it to work is by the following:
/**
* A simple test harness for glusterfs
* @author jvyas
*/
public class TestMapReducerGlusterFS {
@Test
public void test() throws Exception{
Configuration conf = new Configuration(false);
System.out.println("before reload:"+conf);
//necessary?
conf.set("fs.default.name", ".......");
conf.set("fs.blah.mount", "/mnt/......");
conf.set("fs.blahserver", "192.168.1.36");
conf.set("iomod","Off");
...
}
}
Thinking that this is a little known issue in the current hadoop
parser/configurator? because very rarely do people write unit tests which
utilize abnormal ( i.e. non-local or non-hdfs) file systems.
On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
> Result any different if you call reloadConfiguration()?
>
> Thanks,
>
> Tom
>
> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Hi guys:
> >
> > I've finally extracted my problem of loading a special filesystem
> > into a unit test.
> >
> > Below, clearly, Im creating a raw configuration and adding a single
> resource
> > to it (core-site.xml).
> >
> > Afterwards, i print to confirm that the file exists.
> >
> > Finally, I try to create a FileSystem from that file. However, no file
> > system is created :(.
> >
> > Any thoughts on why this core-site.xml file is being ignoreed?
> > To note: I've also tried putting core-site.xml in the working directory,
> and
> > in a local conf/ directory. This absolute path was a last effort to
> really
> > confirm that the local path wasn't somehow being mangled by the junit
> > runtime.
> >
> > @Test
> > public void test() throws Exception{
> > Configuration conf = new Configuration(false);
> > System.out.println("b4:"+conf);
> > String
> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> > conf.addResource(theXMLFile);
> > System.out.println("File exists : " + theXMLFile.length());
> > System.out.println("after:"+conf);
> > System.out.println("file sytem " + conf.get("fs.default.name"));
> > conf.writeXml(System.out);
> >
> >
> > System.out.println("Working directory = " +
> > System.getProperty("user.dir"));
> > org.apache.hadoop.fs.FileSystem fs =
> > org.apache.hadoop.fs.FileSystem.get(conf);
> > System.out.println("before test filesystem = " + fs);
> > //WordCount.main(new
> > String[]{"resources/wordcount.txt","wordcountout.txt"});
> > }
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Jay Vyas <ja...@gmail.com>.
no difference.
This is really odd.
The only way I can get it to work is by the following:
/**
* A simple test harness for glusterfs
* @author jvyas
*/
public class TestMapReducerGlusterFS {
@Test
public void test() throws Exception{
Configuration conf = new Configuration(false);
System.out.println("before reload:"+conf);
//necessary?
conf.set("fs.default.name", ".......");
conf.set("fs.blah.mount", "/mnt/......");
conf.set("fs.blahserver", "192.168.1.36");
conf.set("iomod","Off");
...
}
}
Thinking that this is a little known issue in the current hadoop
parser/configurator? because very rarely do people write unit tests which
utilize abnormal ( i.e. non-local or non-hdfs) file systems.
On Wed, Jan 16, 2013 at 7:45 PM, Tom Melendez <to...@supertom.com> wrote:
> Result any different if you call reloadConfiguration()?
>
> Thanks,
>
> Tom
>
> On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> > Hi guys:
> >
> > I've finally extracted my problem of loading a special filesystem
> > into a unit test.
> >
> > Below, clearly, Im creating a raw configuration and adding a single
> resource
> > to it (core-site.xml).
> >
> > Afterwards, i print to confirm that the file exists.
> >
> > Finally, I try to create a FileSystem from that file. However, no file
> > system is created :(.
> >
> > Any thoughts on why this core-site.xml file is being ignoreed?
> > To note: I've also tried putting core-site.xml in the working directory,
> and
> > in a local conf/ directory. This absolute path was a last effort to
> really
> > confirm that the local path wasn't somehow being mangled by the junit
> > runtime.
> >
> > @Test
> > public void test() throws Exception{
> > Configuration conf = new Configuration(false);
> > System.out.println("b4:"+conf);
> > String
> > theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> > conf.addResource(theXMLFile);
> > System.out.println("File exists : " + theXMLFile.length());
> > System.out.println("after:"+conf);
> > System.out.println("file sytem " + conf.get("fs.default.name"));
> > conf.writeXml(System.out);
> >
> >
> > System.out.println("Working directory = " +
> > System.getProperty("user.dir"));
> > org.apache.hadoop.fs.FileSystem fs =
> > org.apache.hadoop.fs.FileSystem.get(conf);
> > System.out.println("before test filesystem = " + fs);
> > //WordCount.main(new
> > String[]{"resources/wordcount.txt","wordcountout.txt"});
> > }
> >
> > --
> > Jay Vyas
> > http://jayunit100.blogspot.com
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Tom Melendez <to...@supertom.com>.
Result any different if you call reloadConfiguration()?
Thanks,
Tom
On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource
> to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file
> system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and
> in a local conf/ directory. This absolute path was a last effort to really
> confirm that the local path wasn't somehow being mangled by the junit
> runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String
> theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " +
> System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs =
> org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new
> String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Tom Melendez <to...@supertom.com>.
Result any different if you call reloadConfiguration()?
Thanks,
Tom
On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource
> to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file
> system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and
> in a local conf/ directory. This absolute path was a last effort to really
> confirm that the local path wasn't somehow being mangled by the junit
> runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String
> theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " +
> System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs =
> org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new
> String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Can you try prepending the path with file:// or pass in a URL and use the method which takes in a URL?
I remember class loading issues with paths without the file: scheme info from a different scenario (while playing with logj4 config files).
HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Jan 16, 2013, at 4:33 PM, Jay Vyas wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and in a local conf/ directory. This absolute path was a last effort to really confirm that the local path wasn't somehow being mangled by the junit runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " + System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Tom Melendez <to...@supertom.com>.
Result any different if you call reloadConfiguration()?
Thanks,
Tom
On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource
> to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file
> system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and
> in a local conf/ directory. This absolute path was a last effort to really
> confirm that the local path wasn't somehow being mangled by the junit
> runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String
> theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " +
> System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs =
> org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new
> String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Can you try prepending the path with file:// or pass in a URL and use the method which takes in a URL?
I remember class loading issues with paths without the file: scheme info from a different scenario (while playing with logj4 config files).
HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Jan 16, 2013, at 4:33 PM, Jay Vyas wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and in a local conf/ directory. This absolute path was a last effort to really confirm that the local path wasn't somehow being mangled by the junit runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " + System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Can you try prepending the path with file:// or pass in a URL and use the method which takes in a URL?
I remember class loading issues with paths without the file: scheme info from a different scenario (while playing with logj4 config files).
HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Jan 16, 2013, at 4:33 PM, Jay Vyas wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and in a local conf/ directory. This absolute path was a last effort to really confirm that the local path wasn't somehow being mangled by the junit runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " + System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: core-site.xml file is being ignored by new Configuration()
Posted by Tom Melendez <to...@supertom.com>.
Result any different if you call reloadConfiguration()?
Thanks,
Tom
On Wed, Jan 16, 2013 at 4:33 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I've finally extracted my problem of loading a special filesystem
> into a unit test.
>
> Below, clearly, Im creating a raw configuration and adding a single resource
> to it (core-site.xml).
>
> Afterwards, i print to confirm that the file exists.
>
> Finally, I try to create a FileSystem from that file. However, no file
> system is created :(.
>
> Any thoughts on why this core-site.xml file is being ignoreed?
> To note: I've also tried putting core-site.xml in the working directory, and
> in a local conf/ directory. This absolute path was a last effort to really
> confirm that the local path wasn't somehow being mangled by the junit
> runtime.
>
> @Test
> public void test() throws Exception{
> Configuration conf = new Configuration(false);
> System.out.println("b4:"+conf);
> String
> theXMLFile="/Users/jayunit100/Development/blah/conf/core-site.xml";
> conf.addResource(theXMLFile);
> System.out.println("File exists : " + theXMLFile.length());
> System.out.println("after:"+conf);
> System.out.println("file sytem " + conf.get("fs.default.name"));
> conf.writeXml(System.out);
>
>
> System.out.println("Working directory = " +
> System.getProperty("user.dir"));
> org.apache.hadoop.fs.FileSystem fs =
> org.apache.hadoop.fs.FileSystem.get(conf);
> System.out.println("before test filesystem = " + fs);
> //WordCount.main(new
> String[]{"resources/wordcount.txt","wordcountout.txt"});
> }
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com