You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Pierre ANCELOT <pi...@gmail.com> on 2010/06/23 10:23:14 UTC
Feed hdfs with external data.
Hi everyone,
I have a program that generates the data that's supposed to be treated by
hadoop.
It's a java program that should write right on hdfs.
So as a test, I do this:
Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
dfs.mkdirs(new Path("/test"));
Which of course failed, certainly because it's unaware of the position of
the configuration files.
How should I do this?
Thanks.
P.S. Just in case, here's my error:
10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
setXIncludeAware(true) for parser
gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
This parser does not support specification "null" version "null"
java.lang.UnsupportedOperationException: This parser does not support
specification "null" version "null"
at
javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1054)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1030)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:436)
at
org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at fr.tst.testhdfsfromlocal.HDFSWriter.<init>(HDFSWriter.java:29)
at fr.tst.testhdfsfromlocal.HdfsTest.main(HdfsTest.java:8)
10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
setXIncludeAware(true) for parser
gnu.xml.dom.JAXPFactory@7b7072:java.lang.UnsupportedOperationException:
This parser does not support specification "null" version "null"
java.lang.UnsupportedOperationException: This parser does not support
specification "null" version "null"
at
javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1054)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1030)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:436)
at
org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:103)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at fr.tst.testhdfsfromlocal.HDFSWriter.<init>(HDFSWriter.java:29)
at fr.tst.testhdfsfromlocal.HdfsTest.main(HdfsTest.java:8)
Re: Feed hdfs with external data.
Posted by Hemanth Yamijala <yh...@gmail.com>.
Sorry, I might have misunderstood... So, you have a Java application
that needs to use this configuration. In that case, I think setting
the directory containing the config files into the classpath of your
application should work fine.
Thanks
Hemanth
On Wed, Jun 23, 2010 at 2:49 PM, Hemanth Yamijala <yh...@gmail.com> wrote:
> Pierre,
>
>> Ok, thank a lot about this, it worked out by specifying manually the conf...
>> as in:
>>
>> Configuration config = new Configuration();
>> config.set("dfs.name.dir", "/hd/name/");
>> config.set("dfs.data.dir", "/hd/data/");
>> config.set("fs.default.name", "hdfs://hadoop1/");
>>
>> Now, I would like to specify the conf dir location and have all this loaded
>> automatically.
>> Anyone?
>
> You can use --config <path to conf dir> to your bin/hadoop commands. I
> think it would also work if you set the HADOOP_CONF_DIR environment
> variable to point to this path.
>
>>
>>
>> On Wed, Jun 23, 2010 at 10:52 AM, Hemanth Yamijala <yh...@gmail.com>wrote:
>>
>>> Pierre,
>>>
>>> > I have a program that generates the data that's supposed to be treated by
>>> > hadoop.
>>> > It's a java program that should write right on hdfs.
>>> > So as a test, I do this:
>>> >
>>> >
>>> >
>>> > Configuration config = new Configuration();
>>> > FileSystem dfs = FileSystem.get(config);
>>> > dfs.mkdirs(new Path("/test"));
>>> >
>>> > Which of course failed, certainly because it's unaware of the position of
>>> > the configuration files.
>>> > How should I do this?
>>> > Thanks.
>>> >
>>> >
>>> > P.S. Just in case, here's my error:
>>> >
>>> > 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
>>> > setXIncludeAware(true) for parser
>>> > gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
>>> > This parser does not support specification "null" version "null"
>>> > java.lang.UnsupportedOperationException: This parser does not support
>>> > specification "null" version "null"
>>> > at
>>> >
>>> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
>>> > at
>>>
>>> This exception seems very similar to the one we faced and fixed using
>>> the information here: http://bit.ly/a5zX1e. Could you give it a try ?
>>>
>>> Thanks
>>> Hemanth
>>>
>>
>>
>>
>> --
>> http://www.neko-consulting.com
>> Ego sum quis ego servo
>> "Je suis ce que je protège"
>> "I am what I protect"
>>
>
Re: Feed hdfs with external data.
Posted by Hemanth Yamijala <yh...@gmail.com>.
Pierre,
> Ok, thank a lot about this, it worked out by specifying manually the conf...
> as in:
>
> Configuration config = new Configuration();
> config.set("dfs.name.dir", "/hd/name/");
> config.set("dfs.data.dir", "/hd/data/");
> config.set("fs.default.name", "hdfs://hadoop1/");
>
> Now, I would like to specify the conf dir location and have all this loaded
> automatically.
> Anyone?
You can use --config <path to conf dir> to your bin/hadoop commands. I
think it would also work if you set the HADOOP_CONF_DIR environment
variable to point to this path.
>
>
> On Wed, Jun 23, 2010 at 10:52 AM, Hemanth Yamijala <yh...@gmail.com>wrote:
>
>> Pierre,
>>
>> > I have a program that generates the data that's supposed to be treated by
>> > hadoop.
>> > It's a java program that should write right on hdfs.
>> > So as a test, I do this:
>> >
>> >
>> >
>> > Configuration config = new Configuration();
>> > FileSystem dfs = FileSystem.get(config);
>> > dfs.mkdirs(new Path("/test"));
>> >
>> > Which of course failed, certainly because it's unaware of the position of
>> > the configuration files.
>> > How should I do this?
>> > Thanks.
>> >
>> >
>> > P.S. Just in case, here's my error:
>> >
>> > 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
>> > setXIncludeAware(true) for parser
>> > gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
>> > This parser does not support specification "null" version "null"
>> > java.lang.UnsupportedOperationException: This parser does not support
>> > specification "null" version "null"
>> > at
>> >
>> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
>> > at
>>
>> This exception seems very similar to the one we faced and fixed using
>> the information here: http://bit.ly/a5zX1e. Could you give it a try ?
>>
>> Thanks
>> Hemanth
>>
>
>
>
> --
> http://www.neko-consulting.com
> Ego sum quis ego servo
> "Je suis ce que je protège"
> "I am what I protect"
>
Re: Feed hdfs with external data.
Posted by Pierre ANCELOT <pi...@gmail.com>.
Ok, thank a lot about this, it worked out by specifying manually the conf...
as in:
Configuration config = new Configuration();
config.set("dfs.name.dir", "/hd/name/");
config.set("dfs.data.dir", "/hd/data/");
config.set("fs.default.name", "hdfs://hadoop1/");
Now, I would like to specify the conf dir location and have all this loaded
automatically.
Anyone?
On Wed, Jun 23, 2010 at 10:52 AM, Hemanth Yamijala <yh...@gmail.com>wrote:
> Pierre,
>
> > I have a program that generates the data that's supposed to be treated by
> > hadoop.
> > It's a java program that should write right on hdfs.
> > So as a test, I do this:
> >
> >
> >
> > Configuration config = new Configuration();
> > FileSystem dfs = FileSystem.get(config);
> > dfs.mkdirs(new Path("/test"));
> >
> > Which of course failed, certainly because it's unaware of the position of
> > the configuration files.
> > How should I do this?
> > Thanks.
> >
> >
> > P.S. Just in case, here's my error:
> >
> > 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
> > setXIncludeAware(true) for parser
> > gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
> > This parser does not support specification "null" version "null"
> > java.lang.UnsupportedOperationException: This parser does not support
> > specification "null" version "null"
> > at
> >
> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
> > at
>
> This exception seems very similar to the one we faced and fixed using
> the information here: http://bit.ly/a5zX1e. Could you give it a try ?
>
> Thanks
> Hemanth
>
--
http://www.neko-consulting.com
Ego sum quis ego servo
"Je suis ce que je protège"
"I am what I protect"
Re: Feed hdfs with external data.
Posted by Hemanth Yamijala <yh...@gmail.com>.
Pierre,
> I have a program that generates the data that's supposed to be treated by
> hadoop.
> It's a java program that should write right on hdfs.
> So as a test, I do this:
>
>
>
> Configuration config = new Configuration();
> FileSystem dfs = FileSystem.get(config);
> dfs.mkdirs(new Path("/test"));
>
> Which of course failed, certainly because it's unaware of the position of
> the configuration files.
> How should I do this?
> Thanks.
>
>
> P.S. Just in case, here's my error:
>
> 10/06/23 10:04:06 ERROR conf.Configuration: Failed to set
> setXIncludeAware(true) for parser
> gnu.xml.dom.JAXPFactory@1d6096:java.lang.UnsupportedOperationException:
> This parser does not support specification "null" version "null"
> java.lang.UnsupportedOperationException: This parser does not support
> specification "null" version "null"
> at
> javax.xml.parsers.DocumentBuilderFactory.setXIncludeAware(DocumentBuilderFactory.java:590)
> at
This exception seems very similar to the one we faced and fixed using
the information here: http://bit.ly/a5zX1e. Could you give it a try ?
Thanks
Hemanth