Using the HDFS C API

[Phantom <gh...@gmail.com>]

I tried running the hdfs_test from a machine which is not part of the Hadoop
cluster. Could someone please tell me what I am doing wrong (error shown
below)?
I get the following error :

07/06/20 20:16:43 DEBUG conf.Configuration: java.io.IOException: config()
        at org.apache.hadoop.conf.Configuration.<init>(Configuration.java
:93)
        at org.apache.hadoop.dfs.FSConstants.<clinit>(FSConstants.java:120)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(
DFSClient.java:1039)
        at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:318)
        at
org.apache.hadoop.dfs.DistributedFileSystem$RawDistributedFileSystem.create(
DistributedFileSystem.java:143)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.<init>(
ChecksumFileSystem.java:374)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(
ChecksumFileSystem.java:449)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:360)

Exception in thread "main" java.io.IOException: No valid local directories
in property: dfs.client.buffer.dir
        at org.apache.hadoop.conf.Configuration.getFile(Configuration.java
:410)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.newBackupFile(
DFSClient.java:1100)
        at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.<init>(
DFSClient.java:1067)
        at org.apache.hadoop.dfs.DFSClient.create(DFSClient.java:318)
        at
org.apache.hadoop.dfs.DistributedFileSystem$RawDistributedFileSystem.create(
DistributedFileSystem.java:143)
        at org.apache.hadoop.fs.ChecksumFileSystem$FSOutputSummer.<init>(
ChecksumFileSystem.java:374)
        at org.apache.hadoop.fs.ChecksumFileSystem.create(
ChecksumFileSystem.java:449)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:360)
Call to org.apache.hadoop.fs.FileSystem::create((Lorg/apache/hadoop/fs/Path;ZISJ)Lorg/apache/hadoop/fs/FSDataOutputStream;)
failed!
Failed to open /tmp/testfile.txt for writing!

Re: HBase from Map-Reduce update?

[Michael Stack <st...@duboce.net>]

hank williams wrote:
> I have been meaning to ask a similar question, but perhaps a bit more
> broadly about the status and anticipated timeline for hbase. I am
> curious if the effort is purely individual or if there is any
> corporate push (for example from powerset) or if it is just a personal
> project.
Regards the state of HBase, I just made updates to the HBase home page 
news section and edited the tail of the architecture page where it talks 
of current status.  JIRA is -- or should be -- the best source for what 
is currently being worked on.

Broadly, from my point of view:

+ I am able to start up a cluster of 4 nodes and run simple sustained 
loads without the cluster falling over.
+ I'm currently working on loading it with a bunch of web pages with 
columns to hold original content, parsed content, etc. It looks like 
there is sufficient functionality currently implemented for me to do 
this (we'll see).
+ Its slow (See the PerformanceEvaluation page in wiki).
+ Its fragile (E.g HADOOP-1516)

Regards timeline, Powerset where both Jim Kellerman and I work would 
love to have a working HBase as soon as possible (smile). 

Jim and I are both working on HBase near full-time.  Others are also 
making good contributions.

But, IMO, HBase is a big job and its going to take a load of work by a 
bunch of folks making it robust at scale.

> Also, as I understand it from the wiki, hbase works well on a single
> machine but scaling is being worked out. Is there any sense of when an
> anticipated (or desired) target for a scalable version?
Adding nodes to a HBase cluster currently makes it so that the cluster 
can carry more load and handle more requests.  In a sense, with caveats 
-- some of which have been mentioned above while others are yet to be 
discovered -- HBase scales now.

James Kennedy wrote:
> "We need easy interfaces to MapReduce jobs, so they can scan tables. 
> We have been contacted by Vuk Ercegovac <vercego AT SPAMFREE us DOT 
> ibm DOT com> of IBM Almaden Research who expressed an interest in 
> working on an HBase interface to Hadoop map/reduce."
>
> Does anyone know what the status of this is? 
I do not know of any work that has been done on the above.  I made an 
issue for it (HADOOP-1519).

St.Ack


>
> Regards
> Hank
>
> On 6/21/07, James Kennedy <ja...@troove.net> wrote:
>>  From Wiki:
>>
>> "We need easy interfaces to MapReduce jobs, so they can scan tables. We
>> have been contacted by Vuk Ercegovac
>> <vercego AT SPAMFREE us DOT ibm DOT com> of IBM Almaden Research who
>> expressed an interest in working on an HBase interface to Hadoop
>> map/reduce."
>>
>> Does anyone know what the status of this is?
>>

Re: HBase from Map-Reduce update?

[hank williams <ha...@gmail.com>]

I have been meaning to ask a similar question, but perhaps a bit more
broadly about the status and anticipated timeline for hbase. I am
curious if the effort is purely individual or if there is any
corporate push (for example from powerset) or if it is just a personal
project.

Also, as I understand it from the wiki, hbase works well on a single
machine but scaling is being worked out. Is there any sense of when an
anticipated (or desired) target for a scalable version?

Regards
Hank

On 6/21/07, James Kennedy <ja...@troove.net> wrote:
>  From Wiki:
>
> "We need easy interfaces to MapReduce jobs, so they can scan tables. We
> have been contacted by Vuk Ercegovac
> <vercego AT SPAMFREE us DOT ibm DOT com> of IBM Almaden Research who
> expressed an interest in working on an HBase interface to Hadoop
> map/reduce."
>
> Does anyone know what the status of this is?
>

HBase from Map-Reduce update?

[James Kennedy <ja...@troove.net>]

 From Wiki:

"We need easy interfaces to MapReduce jobs, so they can scan tables. We 
have been contacted by Vuk Ercegovac 
<vercego AT SPAMFREE us DOT ibm DOT com> of IBM Almaden Research who 
expressed an interest in working on an HBase interface to Hadoop 
map/reduce."

Does anyone know what the status of this is?