Proper way to access HDFS from C++ pipes

[Keith Wiley <kw...@keithwiley.com>]

I would like to access HDFS directly from C++ pipes, as opposed to merely operating on the input/output keys/values.  I believe this is best achieved through the StreamUtils API, namely FileOutStream.  However, I am unsure how to use it.  For example, if I simply hand an hdfs uri to the FileOutStream, it fails:

CLUSTER: fs.default.name from core-site.xml
PATH: path to a directory I definitely have write permissions in
FILE: a new filename

FileOutStream fos;
fos.open("hdfs://[CLUSTER]/[PATH]/[FILE]", true);

This returns with the failbit set (or the bad bit, I'm not sure which yet).  I have also tried it with the port ":8020" after [CLUSTER].  Made no difference.

I'm not too surprised this didn't work since the implementation of the stream utils functions clearly just calls fopen(), and I'm not aware that fopen() knows how to access files by remote uri, as opposed to local files...so, how do I do this then?

Any ideas?

Thanks.

________________________________________________________________________________
Keith Wiley               kwiley@keithwiley.com               www.keithwiley.com

"I used to be with it, but then they changed what it was.  Now, what I'm with
isn't it, and what's it seems weird and scary to me."
  -- Abe (Grandpa) Simpson
________________________________________________________________________________