You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Matt Frech <ma...@hooklogic.com> on 2012/03/19 17:43:06 UTC

Need a flume-core.jar that will write to S3 and HDFS for CDH Update 3

I'm using running Cloudera's flume(0.9.4-cdh3u3) to write to S3 and HDFS (Hadoop 0.20.2-cdh3u3).  I'm able to get one or the other to work based on the version of hadoop-core.jar I use.

In order to get rid of my "Unexpected response code 404, expected 200" errors when writing to S3,  I'm trying to use emr-hadoop-core-0.20.jar file from an EC2 distribution of hadoop as suggested in this article:
http://www.cloudera.com/blog/2011/02/distributed-flume-setup-with-an-s3-sink/

I used the hadoop-core.jar file linked to by the author here:
http://eric.lubow.org/wp-content/uploads/2011/02/flume-jar.tar.gz

That fixes the S3 writes, but I suspect I need a newer version of that file because when I try to write to hdfs with the updated jar file, I get this error in my hadoop NAMENODE log file:  "WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 10.0.0.118:33499 got version 3 expected version 4".

My question is, am I on the right track to solving this?  And if so, what is the most reliable way to get an updated version of flume-core.jar that will work with my version of flume/hadoop?