You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Bogdan M. Maryniuk" <bo...@gmail.com> on 2009/10/22 06:23:48 UTC
Random weirdness — anyone explain, please?
Hi!
I have quite odd Hadoop behavior. I wrote a client to my app that
simply is trying to talk to HDFS and do stuff. Version of Hadoop is
20.0. I still suspect CLASSPATH, but would be nice to know details.
So, here is a part of a traceback:
----------------------------------------------------
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
....
----------------------------------------------------
Now, could anybody please feel more light here what FileSystem is
trying to load in the configuration, once configuration was created
like: "... = new Configuration()"? I've looked at the source code, and
it does something this:
----------------------------------------------------
private static FileSystem createFileSystem(URI uri, Configuration conf
) throws IOException {
Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
if (clazz == null) {
throw new IOException("No FileSystem for scheme: " + uri.getScheme());
}
FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
fs.initialize(uri, conf);
return fs;
}
----------------------------------------------------
...but I think I need either long hours to hack or just ask here what
the heck is trying to load and can not. :-) It sounds to me that
CLASSPATH was screwed up, but I've put all the required libraries in
Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
However, when I am trying to write there, my thread dies with the
traceback above.
If anybody at least suggest me where to dig more, I would greatly appreciate it.
--
Kind regards, BM
Things, that are stupid at the beginning, rarely ends up wisely.
Re: Syslog-ng and hadoop logs
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Joe,
We added the following to log4j.properties:
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.facility=local0
log4j.appender.SYSLOG.layout=org.apache.log4j.PatternLayout
log4j.appender.SYSLOG.layout.ConversionPattern=%p %c{2}: %m%n
log4j.appender.SYSLOG.SyslogHost=red
log4j.appender.SYSLOG.threshold=ERROR
log4j.appender.SYSLOG.Header=true
log4j.appender.SYSLOG.FacilityPrinting=true
IIRC, if you don't have have .Header=true, the generated packets are
invalid and syslog chokes on them. Fun!
Then, we add whatever we'd like to go to syslog; i.e.
log4j.logger.org.apache.hadoop.hdfs=INFO,SYSLOGM
or
log4j.rootLogger=DEBUG,SYSLOG,console,DRFA,EventCounter
Brian
On Oct 22, 2009, at 7:01 PM, Joe Hammerman wrote:
> Hi all,
>
> We are working on integrating our hadoop logs which live on the
> datanodes into a central syslog-ng logging server. There seems to be
> a dearth of information available on the internet as how to effect
> this; I was hoping someone here could help out, either with a link
> or by providing examples from their configuration.
>
> Thanks in advance!
>
> Regards,
> Joseph Hammerman
Syslog-ng and hadoop logs
Posted by Joe Hammerman <jh...@videoegg.com>.
Hi all,
We are working on integrating our hadoop logs which live on the datanodes into a central syslog-ng logging server. There seems to be a dearth of information available on the internet as how to effect this; I was hoping someone here could help out, either with a link or by providing examples from their configuration.
Thanks in advance!
Regards,
Joseph Hammerman
Re: Random weirdness — anyone explain, please?
Posted by Philip Zeyliger <ph...@cloudera.com>.
On Wed, Oct 21, 2009 at 9:23 PM, Bogdan M. Maryniuk <
bogdan.maryniuk@gmail.com> wrote:
> Hi!
>
> I have quite odd Hadoop behavior. I wrote a client to my app that
> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
> 20.0. I still suspect CLASSPATH, but would be nice to know details.
> So, here is a part of a traceback:
> ----------------------------------------------------
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
> at
> org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
> ....
> ----------------------------------------------------
>
> Now, could anybody please feel more light here what FileSystem is
> trying to load in the configuration, once configuration was created
> like: "... = new Configuration()"? I've looked at the source code, and
> it does something this:
> ----------------------------------------------------
> private static FileSystem createFileSystem(URI uri, Configuration conf
> ) throws IOException {
> Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
> if (clazz == null) {
> throw new IOException("No FileSystem for scheme: " + uri.getScheme());
> }
> FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
> fs.initialize(uri, conf);
> return fs;
> }
> ----------------------------------------------------
>
Here's what's going on here:
The FileSystem class has a static factory which takes the "scheme" of your
URL ("hdfs" or "s3" or "s3n" or "file"...) and maps that into an
implementation class by looking up a configuration variable that looks like
fs.hdfs.impl (or fs.s3.impl or fs.file.impl, ...). It then loads that
implementation by reflection. (Configuration.getClass is just a short hand
for read a configuration variable, and return the class that it represents.)
One thing you might try to debug is to dump the Configuration (using
configuration.writeXml(System.err) or similar), and making sure everything
there looks reasonable. Typically HDFS is configured from from
core-default.xml, like so:
<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
<description>The FileSystem for hdfs: uris.</description>
</property>
-- Philip
Re: Random weirdness - anyone explain, please?
Posted by BM <bo...@gmail.com>.
On Thu, Oct 22, 2009 at 10:43 PM, #YONG YONG CHENG#
<aa...@pmail.ntu.edu.sg> wrote:
> Maybe you can try putting Hadoop jar files in the Extension classloader directory <domain name>/lib/ext.
Apparently, no. Now it works fine. Sounds like Webspace is just
Sun-style thing: bloated, big, slow, complicated and... wrong. On a
regular GlassFish it works fine.
Thanks! Case closed, was not Hadoop issue. :-)
--
Kind regards, BM
Things, that are stupid at the beginning, rarely ends up wisely.
RE: Random weirdness - anyone explain, please?
Posted by #YONG YONG CHENG# <aa...@pmail.ntu.edu.sg>.
Good Day,
Maybe you can try putting Hadoop jar files in the Extension classloader directory <domain name>/lib/ext.
Thanks.
________________________________
From: BM [mailto:bogdan.maryniuk@gmail.com]
Sent: Thu 10/22/2009 3:59 PM
To: common-user@hadoop.apache.org
Subject: Re: Random weirdness - anyone explain, please?
On Thu, Oct 22, 2009 at 1:33 PM, Huy Phan <da...@gmail.com> wrote:
> Hi Bogdan,
> There's nothing wrong with your code, since you're talking about CLASSPATH,
> I guess you didn't execute your script the right way.
> How did you run your script anyway ? With or without hadoop command ?
Well, that's not a script really. I am trying to integrate the thing
with the GlassFish in order apps talk to Hadoop and put/get media from
there instead. So I wrote a client that is available over JNDI. All
Hadoop-related libs I put into common classpath, so they *are*
available. The only thing is probably that Catalina class loader could
mess up things, but not likely...
Anyway, I am looking to fix this. If somebody would fill more light
here what this method wants? What is interesting, that a test-case
works great, i.e. none of errors like this, file are written in the
HDFS and read perfectly. I suspect that Hadoop does some "fun" with
class loading, but I am not sure, since I did not invested time into
this yet (still hope someone will tell me where could be a problem).
--
Kind regards, BM
Things, that are stupid at the beginning, rarely ends up wisely.
Re: Random weirdness — anyone explain, please?
Posted by BM <bo...@gmail.com>.
On Thu, Oct 22, 2009 at 1:33 PM, Huy Phan <da...@gmail.com> wrote:
> Hi Bogdan,
> There's nothing wrong with your code, since you're talking about CLASSPATH,
> I guess you didn't execute your script the right way.
> How did you run your script anyway ? With or without hadoop command ?
Well, that's not a script really. I am trying to integrate the thing
with the GlassFish in order apps talk to Hadoop and put/get media from
there instead. So I wrote a client that is available over JNDI. All
Hadoop-related libs I put into common classpath, so they *are*
available. The only thing is probably that Catalina class loader could
mess up things, but not likely...
Anyway, I am looking to fix this. If somebody would fill more light
here what this method wants? What is interesting, that a test-case
works great, i.e. none of errors like this, file are written in the
HDFS and read perfectly. I suspect that Hadoop does some "fun" with
class loading, but I am not sure, since I did not invested time into
this yet (still hope someone will tell me where could be a problem).
--
Kind regards, BM
Things, that are stupid at the beginning, rarely ends up wisely.
Re: Random weirdness — anyone explain, please?
Posted by Huy Phan <da...@gmail.com>.
Hi Bogdan,
There's nothing wrong with your code, since you're talking about
CLASSPATH, I guess you didn't execute your script the right way.
How did you run your script anyway ? With or without hadoop command ?
Bogdan M. Maryniuk wrote:
> One more thing. I am creating configuration like this (if it is wrong
> — please tell me why and how is right to do it):
> -------------------------------------------
> this.conf = new Configuration();
> this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
> host, port));
> this.fs = FileSystem.get(this.conf);
> -------------------------------------------
>
> I don't see anything wrong with a code above, unless I just missing
> something undocumented...
> Anyone, please?
>
> Thanks in advance.
>
> --
> Bo
>
>
> On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
> <bo...@gmail.com> wrote:
>
>> Hi!
>>
>> I have quite odd Hadoop behavior. I wrote a client to my app that
>> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
>> 20.0. I still suspect CLASSPATH, but would be nice to know details.
>> So, here is a part of a traceback:
>> ----------------------------------------------------
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:247)
>> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
>> ....
>> ----------------------------------------------------
>>
>> Now, could anybody please feel more light here what FileSystem is
>> trying to load in the configuration, once configuration was created
>> like: "... = new Configuration()"? I've looked at the source code, and
>> it does something this:
>> ----------------------------------------------------
>> private static FileSystem createFileSystem(URI uri, Configuration conf
>> ) throws IOException {
>> Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>> if (clazz == null) {
>> throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>> }
>> FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>> fs.initialize(uri, conf);
>> return fs;
>> }
>> ----------------------------------------------------
>>
>> ...but I think I need either long hours to hack or just ask here what
>> the heck is trying to load and can not. :-) It sounds to me that
>> CLASSPATH was screwed up, but I've put all the required libraries in
>> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
>> However, when I am trying to write there, my thread dies with the
>> traceback above.
>>
>> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>>
>
>
>
>
Re: Random weirdness — anyone explain, please?
Posted by BM <bo...@gmail.com>.
On Thu, Oct 22, 2009 at 11:56 PM, brien colwell <xc...@gmail.com> wrote:
> When I debug Hadoop locally in Eclipse I run jobs with the following
> configurations. I access the HDFS with paths like below . It seems to me
> this should be the same as what you have so I also suspect a classpath
> issue.
>
> baseConf.set("fs.default.name", "file:///");
>
> Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
> FileSystem fs = p.getFileSystem(baseConf) .
Well, not my case: I am accessing a real cluster. That works fine, but
somewhat looks like Catalina class loader dislikes hadoop's forName
tricks and in some cases (depends on a Moon phase and solar
interference) does not works. All become OK after a regular restart.
Which is... weird and wrong. On a regular GlassFish it works fine and
stable, only Webspace freaks out.
I need to dig into it anyway, but sounds like not Hadoop issue anymore.
--
Kind regards, BM
Things, that are stupid at the beginning, rarely ends up wisely.
Re: Random weirdness — anyone explain, please?
Posted by brien colwell <xc...@gmail.com>.
When I debug Hadoop locally in Eclipse I run jobs with the following
configurations. I access the HDFS with paths like below . It seems to me
this should be the same as what you have so I also suspect a classpath
issue.
baseConf.set("fs.default.name", "file:///");
Path p = new Path("hdfs://<namenode>:<port>/path/to/stuff")
FileSystem fs = p.getFileSystem(baseConf) .
Bogdan M. Maryniuk wrote:
> One more thing. I am creating configuration like this (if it is wrong
> — please tell me why and how is right to do it):
> -------------------------------------------
> this.conf = new Configuration();
> this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
> host, port));
> this.fs = FileSystem.get(this.conf);
> -------------------------------------------
>
> I don't see anything wrong with a code above, unless I just missing
> something undocumented...
> Anyone, please?
>
> Thanks in advance.
>
> --
> Bo
>
>
> On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
> <bo...@gmail.com> wrote:
>
>> Hi!
>>
>> I have quite odd Hadoop behavior. I wrote a client to my app that
>> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
>> 20.0. I still suspect CLASSPATH, but would be nice to know details.
>> So, here is a part of a traceback:
>> ----------------------------------------------------
>> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:247)
>> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
>> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
>> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
>> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
>> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
>> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
>> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
>> ....
>> ----------------------------------------------------
>>
>> Now, could anybody please feel more light here what FileSystem is
>> trying to load in the configuration, once configuration was created
>> like: "... = new Configuration()"? I've looked at the source code, and
>> it does something this:
>> ----------------------------------------------------
>> private static FileSystem createFileSystem(URI uri, Configuration conf
>> ) throws IOException {
>> Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
>> if (clazz == null) {
>> throw new IOException("No FileSystem for scheme: " + uri.getScheme());
>> }
>> FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
>> fs.initialize(uri, conf);
>> return fs;
>> }
>> ----------------------------------------------------
>>
>> ...but I think I need either long hours to hack or just ask here what
>> the heck is trying to load and can not. :-) It sounds to me that
>> CLASSPATH was screwed up, but I've put all the required libraries in
>> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
>> However, when I am trying to write there, my thread dies with the
>> traceback above.
>>
>> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>>
>> --
>> Kind regards, BM
>>
>> Things, that are stupid at the beginning, rarely ends up wisely.
>>
>>
>
>
>
>
Re: Random weirdness — anyone explain, please?
Posted by "Bogdan M. Maryniuk" <bo...@gmail.com>.
One more thing. I am creating configuration like this (if it is wrong
— please tell me why and how is right to do it):
-------------------------------------------
this.conf = new Configuration();
this.conf.set("fs.default.name", String.format("hdfs://%s:%s",
host, port));
this.fs = FileSystem.get(this.conf);
-------------------------------------------
I don't see anything wrong with a code above, unless I just missing
something undocumented...
Anyone, please?
Thanks in advance.
--
Bo
On Thu, Oct 22, 2009 at 1:23 PM, Bogdan M. Maryniuk
<bo...@gmail.com> wrote:
> Hi!
>
> I have quite odd Hadoop behavior. I wrote a client to my app that
> simply is trying to talk to HDFS and do stuff. Version of Hadoop is
> 20.0. I still suspect CLASSPATH, but would be nice to know details.
> So, here is a part of a traceback:
> ----------------------------------------------------
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:761)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:806)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1368)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1385)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:191)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:167)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1172)
> ....
> ----------------------------------------------------
>
> Now, could anybody please feel more light here what FileSystem is
> trying to load in the configuration, once configuration was created
> like: "... = new Configuration()"? I've looked at the source code, and
> it does something this:
> ----------------------------------------------------
> private static FileSystem createFileSystem(URI uri, Configuration conf
> ) throws IOException {
> Class<?> clazz = conf.getClass("fs." + uri.getScheme() + ".impl", null);
> if (clazz == null) {
> throw new IOException("No FileSystem for scheme: " + uri.getScheme());
> }
> FileSystem fs = (FileSystem)ReflectionUtils.newInstance(clazz, conf);
> fs.initialize(uri, conf);
> return fs;
> }
> ----------------------------------------------------
>
> ...but I think I need either long hours to hack or just ask here what
> the heck is trying to load and can not. :-) It sounds to me that
> CLASSPATH was screwed up, but I've put all the required libraries in
> Hadoop anyway. And when I am *reading* from the HDFS, everything OK.
> However, when I am trying to write there, my thread dies with the
> traceback above.
>
> If anybody at least suggest me where to dig more, I would greatly appreciate it.
>
> --
> Kind regards, BM
>
> Things, that are stupid at the beginning, rarely ends up wisely.
>
--
Kind regards, BM
Things, that are stupid at the beginning, rarely ends up wisely.