Avro-c with libhdfs

[Tibor Benke <bt...@balabit.hu>]

Hi All,

I would like to write Avro files to HDFS using C as the programming 
language. I have a simple queston: Can I use the avro-c API with 
libhdfs? I tried the hdfs-nfs-proxy, but it doesn't support the append 
operation.

Thanks,
Tibor

Re: Avro-c with libhdfs

[Douglas Creager <do...@creagertino.net>]

> I would like to write Avro files to HDFS using C as the programming
> language. I have a simple queston: Can I use the avro-c API with
> libhdfs? I tried the hdfs-nfs-proxy, but it doesn't support the append
> operation.

I don't have any experience with libhdfs, so I'm not sure.  The Avro C
API lets you write Avro data files to a file on the local filesystem,
identified by a `const char *` filename, or a `FILE *` instance that
you've already opened, so my guess is that, out of the box, the only way
it would work would be to create the file locally and then transfer the
existing local file to HDFS.

To get it to work directly with libhdfs, we'd probably need to add a new
API for providing a callback function, which the Avro bindings would
call whenever data needs to be written.  (This is already what happens
under the covers, so it would really just be exposing parts of the
existing functionality in the public API.)

cheers
–doug