You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bryan Duxbury <br...@rapleaf.com> on 2008/05/13 20:27:54 UTC
Trouble hooking up my app to HDFS
I'm trying to create a java application that writes to HDFS. I have
it set up such that hadoop-0.16.3 is on my machine, and the env
variables HADOOP_HOME and HADOOP_CONF_DIR point to the correct
respective directories. My app lives elsewhere, but generates it's
classpath by looking in those environment variables. Here's what my
generated classpath looks like:
/Users/bryanduxbury/hadoop-0.16.3/conf:/Users/bryanduxbury/
hadoop-0.16.3/hadoop-0.16.3-core.jar:/Users/bryanduxbury/
hadoop-0.16.3/hadoop-0.16.3-test.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/commons-cli-2.0-SNAPSHOT.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/commons-codec-1.3.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/commons-httpclient-3.0.1.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/commons-logging-1.0.4.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/commons-logging-api-1.0.4.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/jets3t-0.5.0.jar:/Users/bryanduxbury/hadoop-0.16.3/
lib/jetty-5.1.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/
commons-el.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/jasper-
compiler.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/jasper-
runtime.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/jsp-
api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/junit-3.8.1.jar:/Users/
bryanduxbury/hadoop-0.16.3/lib/kfs-0.1.jar:/Users/bryanduxbury/
hadoop-0.16.3/lib/log4j-1.2.13.jar:/Users/bryanduxbury/hadoop-0.16.3/
lib/servlet-api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
xmlenc-0.52.jar:/Users/bryanduxbury/projects/hdfs_collector/lib/
jtestr-0.2.jar:/Users/bryanduxbury/projects/hdfs_collector/lib/
jvyaml.jar:/Users/bryanduxbury/projects/hdfs_collector/lib/
libthrift.jar:/Users/bryanduxbury/projects/hdfs_collector/build/
hdfs_collector.jar
The problem I have is that when I go to get a FileSystem object for
my file:/// files (for testing locally), I'm getting errors like this:
[jtestr] java.io.IOException: No FileSystem for scheme: file
[jtestr] org/apache/hadoop/fs/FileSystem.java:1179:in
`createFileSystem'
[jtestr] org/apache/hadoop/fs/FileSystem.java:55:in `access
$300'
[jtestr] org/apache/hadoop/fs/FileSystem.java:1193:in `get'
[jtestr] org/apache/hadoop/fs/FileSystem.java:150:in `get'
[jtestr] org/apache/hadoop/fs/FileSystem.java:124:in
`getNamed'
[jtestr] org/apache/hadoop/fs/FileSystem.java:96:in `get'
I saw an archived message that suggested this was a problem with the
classpath, but there was no resolution that I saw. Does anyone have
any ideas why this might be occurring?
Thanks,
Bryan
Re: Trouble hooking up my app to HDFS
Posted by Bryan Duxbury <br...@rapleaf.com>.
Thanks - this suggestion is good.
I poked around in Configuration trying to see where things get loaded
from and found the "quietmode" flag, which keeps info about where the
config is being sourced from being printed. The log statements here
are info, but they're wrapped with a global hardcoded constant. Maybe
the log statements should be changed to debug and the quietmode
constant removed? It'd make it a lot easier to get access to this
output.
I've found that my problem seems to be that hadoop-default.xml goes
unfound when I'm running my JtestR test suite. For giggles, I created
a quick JUnit test just to see if it could access hadoop, and it
could. I assume this means something about using JRuby messes with
the classloader and interferes with picking up the config files from
the classpath.
-Bryan
On May 15, 2008, at 2:21 AM, Steve Loughran wrote:
> Bryan Duxbury wrote:
>> Nobody has any ideas about this?
>> -Bryan
>> On May 13, 2008, at 11:27 AM, Bryan Duxbury wrote:
>>> I'm trying to create a java application that writes to HDFS. I
>>> have it set up such that hadoop-0.16.3 is on my machine, and the
>>> env variables HADOOP_HOME and HADOOP_CONF_DIR point to the
>>> correct respective directories. My app lives elsewhere, but
>>> generates it's classpath by looking in those environment
>>> variables. Here's what my generated classpath looks like:
>>>
>>> /Users/bryanduxbury/hadoop-0.16.3/conf:/Users/bryanduxbury/
>>> hadoop-0.16.3/hadoop-0.16.3-core.jar:/Users/bryanduxbury/
>>> hadoop-0.16.3/hadoop-0.16.3-test.jar:/Users/bryanduxbury/
>>> hadoop-0.16.3/lib/commons-cli-2.0-SNAPSHOT.jar:/Users/
>>> bryanduxbury/hadoop-0.16.3/lib/commons-codec-1.3.jar:/Users/
>>> bryanduxbury/hadoop-0.16.3/lib/commons-httpclient-3.0.1.jar:/
>>> Users/bryanduxbury/hadoop-0.16.3/lib/commons-logging-1.0.4.jar:/
>>> Users/bryanduxbury/hadoop-0.16.3/lib/commons-logging-
>>> api-1.0.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
>>> jets3t-0.5.0.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
>>> jetty-5.1.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/
>>> commons-el.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/
>>> jasper-compiler.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-
>>> ext/jasper-runtime.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
>>> jetty-ext/jsp-api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
>>> junit-3.8.1.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
>>> kfs-0.1.jar:/Users/bryanduxbury/hadoop-0.16.3/li
> b/log4j-1.2.13.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/servlet-
> api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/xmlenc-0.52.jar:/
> Users/bryanduxbury/projects/hdfs_collector/lib/jtestr-0.2.jar:/
> Users/bryanduxbury/projects/hdfs_collector/lib/jvyaml.jar:/Users/
> bryanduxbury/projects/hdfs_collector/lib/libthrift.jar:/Users/
> bryanduxbury/projects/hdfs_collector/build/hdfs_collector.jar
>>>
>>>
>>> The problem I have is that when I go to get a FileSystem object
>>> for my file:/// files (for testing locally), I'm getting errors
>>> like this:
>>>
>>> [jtestr] java.io.IOException: No FileSystem for scheme: file
>>> [jtestr] org/apache/hadoop/fs/FileSystem.java:1179:in
>>> `createFileSystem'
>>> [jtestr] org/apache/hadoop/fs/FileSystem.java:55:in
>>> `access$300'
>>> [jtestr] org/apache/hadoop/fs/FileSystem.java:1193:in `get'
>>> [jtestr] org/apache/hadoop/fs/FileSystem.java:150:in `get'
>>> [jtestr] org/apache/hadoop/fs/FileSystem.java:124:in
>>> `getNamed'
>>> [jtestr] org/apache/hadoop/fs/FileSystem.java:96:in `get'
>>>
>
> Filesystems get located by looking up your conf file for the
> mapping from url schema to classname. if a filesystem isnt found,
> it means your hadoop-default.xml isnt being picked up (its in the
> root of hadoop-core), or a different version is being picked up,
> which contains invalid data.
>
> this could be g good oppportunity for you to write some extra
> diagnostics for hadoop, something to dump all the job conf
> information on the client, and even validate a conf to look for
> important values.
>
> steve
Re: Trouble hooking up my app to HDFS
Posted by Steve Loughran <st...@apache.org>.
Bryan Duxbury wrote:
> Nobody has any ideas about this?
>
> -Bryan
>
> On May 13, 2008, at 11:27 AM, Bryan Duxbury wrote:
>
>> I'm trying to create a java application that writes to HDFS. I have it
>> set up such that hadoop-0.16.3 is on my machine, and the env variables
>> HADOOP_HOME and HADOOP_CONF_DIR point to the correct respective
>> directories. My app lives elsewhere, but generates it's classpath by
>> looking in those environment variables. Here's what my generated
>> classpath looks like:
>>
>> /Users/bryanduxbury/hadoop-0.16.3/conf:/Users/bryanduxbury/hadoop-0.16.3/hadoop-0.16.3-core.jar:/Users/bryanduxbury/hadoop-0.16.3/hadoop-0.16.3-test.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/commons-cli-2.0-SNAPSHOT.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/commons-codec-1.3.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/commons-httpclient-3.0.1.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/commons-logging-1.0.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/commons-logging-api-1.0.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jets3t-0.5.0.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-5.1.4.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/commons-el.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/jasper-compiler.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/jasper-runtime.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/jetty-ext/jsp-api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/junit-3.8.1.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/kfs-0.1.jar:/Users/bryanduxbury/hadoop-0.16.3/li
b/log4j-1.2.13.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/servlet-api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/xmlenc-0.52.jar:/Users/bryanduxbury/projects/hdfs_collector/lib/jtestr-0.2.jar:/Users/bryanduxbury/projects/hdfs_collector/lib/jvyaml.jar:/Users/bryanduxbury/projects/hdfs_collector/lib/libthrift.jar:/Users/bryanduxbury/projects/hdfs_collector/build/hdfs_collector.jar
>>
>>
>> The problem I have is that when I go to get a FileSystem object for my
>> file:/// files (for testing locally), I'm getting errors like this:
>>
>> [jtestr] java.io.IOException: No FileSystem for scheme: file
>> [jtestr] org/apache/hadoop/fs/FileSystem.java:1179:in
>> `createFileSystem'
>> [jtestr] org/apache/hadoop/fs/FileSystem.java:55:in `access$300'
>> [jtestr] org/apache/hadoop/fs/FileSystem.java:1193:in `get'
>> [jtestr] org/apache/hadoop/fs/FileSystem.java:150:in `get'
>> [jtestr] org/apache/hadoop/fs/FileSystem.java:124:in `getNamed'
>> [jtestr] org/apache/hadoop/fs/FileSystem.java:96:in `get'
>>
Filesystems get located by looking up your conf file for the mapping
from url schema to classname. if a filesystem isnt found, it means your
hadoop-default.xml isnt being picked up (its in the root of
hadoop-core), or a different version is being picked up, which contains
invalid data.
this could be g good oppportunity for you to write some extra
diagnostics for hadoop, something to dump all the job conf information
on the client, and even validate a conf to look for important values.
steve
Re: Trouble hooking up my app to HDFS
Posted by Bryan Duxbury <br...@rapleaf.com>.
Nobody has any ideas about this?
-Bryan
On May 13, 2008, at 11:27 AM, Bryan Duxbury wrote:
> I'm trying to create a java application that writes to HDFS. I have
> it set up such that hadoop-0.16.3 is on my machine, and the env
> variables HADOOP_HOME and HADOOP_CONF_DIR point to the correct
> respective directories. My app lives elsewhere, but generates it's
> classpath by looking in those environment variables. Here's what my
> generated classpath looks like:
>
> /Users/bryanduxbury/hadoop-0.16.3/conf:/Users/bryanduxbury/
> hadoop-0.16.3/hadoop-0.16.3-core.jar:/Users/bryanduxbury/
> hadoop-0.16.3/hadoop-0.16.3-test.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/commons-cli-2.0-SNAPSHOT.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/commons-codec-1.3.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/commons-httpclient-3.0.1.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/commons-logging-1.0.4.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/commons-logging-api-1.0.4.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/jets3t-0.5.0.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/jetty-5.1.4.jar:/Users/bryanduxbury/hadoop-0.16.3/
> lib/jetty-ext/commons-el.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
> jetty-ext/jasper-compiler.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
> jetty-ext/jasper-runtime.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
> jetty-ext/jsp-api.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/
> junit-3.8.1.jar:/Users/bryanduxbury/hadoop-0.16.3/lib/kfs-0.1.jar:/
> Users/bryanduxbury/hadoop-0.16.3/lib/log4j-1.2.13.jar:/Users/
> bryanduxbury/hadoop-0.16.3/lib/servlet-api.jar:/Users/bryanduxbury/
> hadoop-0.16.3/lib/xmlenc-0.52.jar:/Users/bryanduxbury/projects/
> hdfs_collector/lib/jtestr-0.2.jar:/Users/bryanduxbury/projects/
> hdfs_collector/lib/jvyaml.jar:/Users/bryanduxbury/projects/
> hdfs_collector/lib/libthrift.jar:/Users/bryanduxbury/projects/
> hdfs_collector/build/hdfs_collector.jar
>
> The problem I have is that when I go to get a FileSystem object for
> my file:/// files (for testing locally), I'm getting errors like this:
>
> [jtestr] java.io.IOException: No FileSystem for scheme: file
> [jtestr] org/apache/hadoop/fs/FileSystem.java:1179:in
> `createFileSystem'
> [jtestr] org/apache/hadoop/fs/FileSystem.java:55:in `access
> $300'
> [jtestr] org/apache/hadoop/fs/FileSystem.java:1193:in `get'
> [jtestr] org/apache/hadoop/fs/FileSystem.java:150:in `get'
> [jtestr] org/apache/hadoop/fs/FileSystem.java:124:in
> `getNamed'
> [jtestr] org/apache/hadoop/fs/FileSystem.java:96:in `get'
>
> I saw an archived message that suggested this was a problem with
> the classpath, but there was no resolution that I saw. Does anyone
> have any ideas why this might be occurring?
>
> Thanks,
>
> Bryan